osd: skip heartbeat_check for osd which we haven't sent ping to

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: initialize last_pg_stats_ack correctly on first pg_stats message sending

So we don't potentially make an incorrect judgement in the tick_without_osd_lock()
procedure if OSD's first pg_stats message was sent, which as a result shall cause
an unnecessary hunting of new monitor.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: update osd_stat_updated field if we force to report to monitor

On tick_without_osd_lock() waking up, we will force an immediate
report to monitor if a long enough period has elapsed since our
last report was sent.

However, the send_pg_stats() procedure is a noop unless either
osd_stat_updated field is set or pg_stat_queue is empty.
Since the latter condition is non-deterministic, we shall
manually set osd_stat_updated here to kick a pg relevant report
off.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8287 from dillaman/wip-15232

rbd-mirror: initial failover / failback support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7712 from vitek/listbucket_encoding_type_fix

rgw: S3: set EncodingType in ListBucketResult

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8363 from cbodley/wip-cmake-vstart

vstart: fix up cmake paths when VSTART_DEST is given
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8362 from jdurgin/wip-ceph-python-path

ceph.in: fix python libpath for automake as well

Merge pull request #8364 from liewegas/wip-fix-update-op

os/ObjectStore: fix _update_op for split dest_cid

Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: fix up cmake paths when VSTART_DEST is given

mstart.sh uses VSTART_DEST to tell vstart.sh where to create the
cluster. in the cmake case, it needs to get a relative path from there
to the built binaries and libraries - update these paths to use the new
'lib' and 'bin' directories

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/ObjectStore: fix _update_op for split dest_cid

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: add basic failover/fallback test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: use 32bit image creation bid to avoid overflow

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: initial support for primary/non-primary handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: accessor for journal tag data

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: link primary mirror uuid to non-primary journal tag

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: update mirror peer sync state after completion

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track mirror peer replay state

This new state will protect against the case of an rbd-mirror crash
between registering with the remote and starting the image sync.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: allocate local journal tags from mirror remote peer tags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method for retrieving tag struct

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: use consistent "non-primary" verbiage for image mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: image demotion should record new demote journal event

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: blacklist journal error while attempting to acquire lock

The journal policy will return -EPERM if attempting to acquire the
exclusive lock when the image is non-primary.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new demote journal event

This event will be used as a placeholder to ensure that the demotion journal
tag is properly processed by rbd-mirror.  Without an event associated with
the demotion journal tag, the demotion event will be missed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: framework for replay allocating tags in local journal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal reset should re-register the local client

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: cleanup debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: cross-reference local image via global image id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: mirroring directory reverse lookup from global id to image id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: bootstrap requires the global image id

The global image id can be used to cross-reference images that
have been replicated amongst peers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fix state output stream formatter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support creating images with non-primary journals

When rbd-mirror initially creates a replicated image in the
local pool, the journal should be flagged as non-primary to
(1) prevent local ops againsts the image and (2) provide a
clean chain of replicated tags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: don't acquire lock for primary local images

Abort the image replay for any local images that are marked as primary.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: tweaks to support creating mock test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: bootstrap now has initial support for existing images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: properly initialize image copy start/end objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: only delete ImageCtx after open failure

The image is already closed -- it just needs to be destroyed
in a thread context outside of librbd.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: disallow client registration against missing journal

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: added helper to query single client record from disk

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: convert ImageReplayer into templated class

This will facilitate create mock test cases.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper methods to allocate tags associated to remote journals

Allocating tags for the local journal will always be based off the local
commit position.  For remote journals, the local commit position doesn't
reflect actual events replayed by rbd-mirror, so it needs to provide the
remote journal commit position.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: initial integration with librbd lock/journal policies

For the first stage, acquiring the exclusive-lock will no longer
automatically result in the allocation of a journal tag.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: basic policy for journal handling

This will allow rbd-mirror to override the allocation of a new
tag and to restart rbd-mirror's image replayer when a watch error
forces the loss of the exclusive lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: optional 'force' parameter for request lock payload

This will be used when force promoting an image to primary -- when
rbd-mirror receives the forced request, it will immediately abort
playback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: basic policy for controlling the release of the exclusive lock

The rbd-mirror daemon will need to interlock with the request for the lock
from another client with the successful sync of remote journal events.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: journal type traits should be standalone

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8332 from rjfd/wip-15267

rbd-mirror: prevent enabling/disabling an image's mirroring when not in image mode

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: fix python libpath for automake as well

Follow-on to a041e5c9413fd816bc2858509319fa469fa233e5
due to confusion from .pyc files hanging around.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #8320 from dachary/wip-virtualenv-jewel

Makefile-env.am: set a default for CEPH_BUILD_VIRTUALENV (part 2)

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #8360 from jdurgin/wip-ceph-python-path

ceph.in: correct dev python path for automake builds

Tested-by: Sage Weil <sage@redhat.com>

Merge pull request #8361 from liewegas/wip-test-port

test/test_pool_create.sh: fix port

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Merge pull request #8356 from ceph/wip-cython-build-dir

pybind/Makefile.am: Prevent race creating CYTHON_BUILD_DIR

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/test_pool_create.sh: fix port

17108 is used by test/mon/test_pool_quota.sh

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8359 from athanatos/wip-15205

os/ObjectStore: try_move_rename in transaction append and add coverage to store_test

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.in: correct dev python path for automake builds

Accidentally broken by ad2e6f442df0fe302c09fc751fc9e12237789511

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

store_test: apply an appended transaction half the time

Signed-off-by: Samuel Just <sjust@redhat.com>

Transaction: handle OP_TRY_RENAME in dump

Signed-off-by: Samuel Just <sjust@redhat.com>

store_test: add a TryMoveRename test

Signed-off-by: Samuel Just <sjust@redhat.com>

ObjectStore::Transaction::_update_op: handle OP_TRY_RENAME

Fixes: http://tracker.ceph.com/issues/15205
Signed-off-by: Samuel Just <sjust@redhat.com>

rbd: rbd-mirroring: Updated rbd mirroring unit tests to reflect the new conditions to enable/disable image mirroring

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #6317 from cxwshawn/wip-lost-fix

PG: pg down state blocked by osd.x, lost osd.x cannot solve peering stuck

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8292 from vshankar/bluefs-tool-fix

os/bluestore: ceph-bluefs-tool fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8334 from dachary/wip-subman-facts

script: subscription-manager support (part 3)

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #8351 from jcsp/wip-cmake-cython

cmake: Build cython modules and change paths to bin/, lib/
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rbd: rbd-mirroring: Disable image mirroring depends on pool mirroring mode

Signed-off-by: Ricardo Dias <rdias@suse.com>
Fixes: #15267

rbd: rbd-mirroring: Enabling image mirroring depends on pool mirroring mode

Signed-off-by: Ricardo Dias <rdias@suse.com>

script: subscription-manager support (part 3)

Renaming subman.py in subman is cute but does not prevent the generation
of the py[co].

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #8215 from dzafman/wip-14438

Wip 14438

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8261 from rjfd/wip-15217

rbd-mirror: enabling/disabling pool mirroring should update the mirroring directory

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: update for cmake path changes

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: update for cmake build path changes

Cribbed this from Ali's wip-cmake-make-check branch.

Signed-off-by: John Spray <john.spray@redhat.com>

cmake: Cython modules building with cmake

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #8228 from SUSE/wip-15218

doc/dev: add "Deploy a cluster for manual testing" section

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #8256 from dachary/wip-mailmap

mailmap updates

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Adam C. Emerson name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Jenkins name normalization

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Yan Jun affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Zhao Junwang affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Sarthak Munshi affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Rust Shen affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Eric Cook affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Ricardo Dias affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Karol Mroz affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: James Liu affiliation

Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Robert LeBlanc name normalization

Reviewed-by: Robert LeBlanc <robert.leblanc@endurance.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Yankun Li name normalization

Reviewed-by: Yankun Li <liyankun@unitedstack.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

mailmap: Josh Durgin name normalization

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: rbd-mirroring: Added unit tests to test enable/disable image mirroring when pool-level mirroring changes

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: rbd-mirroring: Disable image mirroring when pool-level mirroring is disabled

Signed-off-by: Ricardo Dias <rdias@suse.com>
Fixes: #15217

rbd: rbd-mirroring: Enable image mirroring when pool-level mirroring is enabled

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #8345 from tchaikov/wip-release-notes-indents

doc/release-notes: fix indents

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #8342 from xiexingguo/xxg-wip-doc

doc: fix typo, duplicated content etc. for Jewel release notes

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/release-notes: fix indents

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix typo, duplicated content etc. for Jewel release notes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/Makefile.am: Prevent race creating CYTHON_BUILD_DIR

Fixes http://tracker.ceph.com/issues/15276

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #8321 from linuxbox2/jewel-nfs-fixes-2

rgw_file: set owner uid, gid, and Unix mode on new objects

Merge pull request #8106 from linuxbox2/wip-cmake-ssl

cmake: add FindOpenSSL.cmake

Merge pull request #8312 from SUSE/wip-submittingpatches

doc: amend Fixes instructions in SubmittingPatches

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw_file tests: allow override of owner_uid and owner_gid

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>