global: don't link lttng into libglobal

Rely on dynamic initialization instead. Linking lttng in this way had
the unfortunate side effect of causing radosgw to segfault (when
daemonized) during sigterm processing (ie. during lttng_ust_exit()).

This was originally removed in 638738f, but accidentally re-added via
5f61d36.

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #9180 from dillaman/wip-15746-jewel

jewel: doc: update mirroring guide to include pool/image status commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: update mirroring guide to include pool/image status commands

Fixes: http://tracker.ceph.com/issues/15746
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 4381e341c435c80793b847115eed616b8359c4e6)

doc: fixup: "rbd-mirror daemon" instead of "rbd-daemon"

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit ddd6345f8f7edcff638ed435aef316ba3e0e40dd)

Merge pull request #8815 from ktdreyer/wip-15646-jewel-system-targets

jewel: debian: install systemd target files

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: dh_systemd_start is in the dh-systemd package

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 89154d1e01dbc58dc92f37741031f7ddb0448d2e)

debian: install systemd target files

* enable it using dh_systemd_enable
* start the target using dh_systemd_start
* move the dh_installinit, dh_systemd_enable, dh_systemd_start calls
  down, so they can identify the service files if they care about them.

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f950a0a7f5b35e13668379bc69dcb5d98a2dfc1f)

10.2.1

Merge pull request #8853 from ceph/wip-jewel-backports

jewel: several backports

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9041 from vshankar/jewel-15721-backport

jewel: rbd: helpful error message on map failure

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #9081 from yehudasa/wip-rgw-period-commit-jewel

rgw: period commit fix

Merge pull request #8969 from Abhishekvrshny/wip-15731-jewel

jewel: MDSAuthCap parse no longer fails on paths with hyphens

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8970 from Abhishekvrshny/wip-15732-jewel

jewel: MDS incarnation no longer gets lost after remove filesystem

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8971 from Abhishekvrshny/wip-15738-jewel

jewel: handle standby-replay nodes properly in upgrades

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9054 from yehudasa/wip-rgw-admin-output-jewel

jewel: rgw admin output

Merge pull request #9053 from yehudasa/wip-15745-jewel

jewel: rgw: handle stripe transition when flushing final pending_data_bl

Merge pull request #9047 from ceph/wip-rgw-period-delete-jewel

jewel: rgw: period delete fixes

Merge pull request #9076 from dillaman/wip-cmake-test-rbd-mirror-jewel

jewel: cmake: fix rbd compile errors

cmake: fix rbd compile errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8d4c00d1bec431f8c721195c81684c4095079c1d)

cmake: add library cls_journal for target unittest_librbd

otherwise if we want to generate target unittest_librbd we need to
(re)generate cls_journal manually

Signed-off-by: runsisi <runsisi@zte.com.cn>
(cherry picked from commit 7efb29416b789fcb7e68aab554d6033e6b5ebb43)

Merge pull request #9063 from dachary/wip-15838-jewel

jewel: tests: make check fails on ext4

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: enable make check on ext4 (part 2)

Followup of 475cc08c330e639b226f2ab57dedbe4495f63022

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b0b3a068d4fcc7b739c5dd3d68297e2d517c378e)

qa/workunits/ceph-helpers.sh: make ceph-osd behave on ext4

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 475cc08c330e639b226f2ab57dedbe4495f63022)

radosgw-admin: 'period commit' sends to new master zone by default

if no --remote or --url is given, set --remote = master_zone_id to
override the rest_master_conn (which refers to the old master zone)

Fixes: http://tracker.ceph.com/issues/15828
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fbe1a05355549dfe0c05750683eeb61e275950c2)

radosgw-admin: allow --remote to specify zone or zonegroup id

--remote was only looking for zonegroup connections. extended to check
for zone connections as well

updated the usage to clarify that it expects zone/zonegroup ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9fcff89ee1e60009fa367e33c915a90354c9a9cc)

radosgw-admin: 'zonegroup add' prints updated zonegroup

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d355208023fe08a7e16f1078c872286048c4bcc6)

radosgw-admin: 'zonegroup modify' prints updated zonegroup

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a033f40caf60083111886aa804fcdbf3ee60aef8)

radosgw-admin: 'zone modify' prints updated zone

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8a639014a6f2dd950062aa405bbee1762ca23e09)

rgw: handle stripe transition when flushing final pending_data_bl

Fixes: http://tracker.ceph.com/issues/15745
When complete_writing_data() is called, if pending_data_bl is not empty
we still need to handle stripe transition correctly. If pending_data_bl
has more data that we can allow in current stripe, move to the next one.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b7a7d41839558b1a8786a66e27d9d80e606f61c7)

Merge pull request #9052 from dillaman/wip-cmake-rbd-mirror-replay-status-jewel

jewel: cmake: add missing source file to rbd_mirror/image_replayer

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: add missing source file to rbd_mirror/image_replayer

fixes an undefined reference when linking librbd_mirror_internal.a

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2f3f56170ab47fc10d4232101ae2e85398a4c299)

Merge pull request #9051 from athanatos/wip-15815-jewel

Wip 15815 jewel

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_keyvaluedb: clean up test dir

Just like ceph_test_objectstore

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2b075aa52d4475338c0f3cd3360e1ee4de8befa7)

ceph_test_keyvaluedb: fixup unit test for keyvaluedb

Fixes: http://tracker.ceph.com/issues/15435
Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>
(cherry picked from commit 85c14de20936ff9ee224f9b6569a73f9bae75f19)

Merge pull request #9050 from dillaman/wip-15821

jewel: rbd-mirror: image sync needs to handle snapshot size and protection status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: add missing header dependency for MockJournaler

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fa91919c856ac8186545bc8f8da31a8f438a9b5d)

rbd-mirror: image sync object copy was not properly mapping snapshots

When the snapshot sequence is out-of-sync between remote and local
clusters (expected), the objects would not be written with the
correct snapshot context.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dd8f08039bf5354a0413c9da4263d1075243e9c9)

rbd-mirror: clean up image sync debug messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit acca475fe5c9ff024c7f76397e7f49e4c4558b09)

librbd: readahead should not read past end of image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit adb14303ccf8a26ed5536bb692ae63d92ac40a9f)

test: randomize rbd-mirror test case IO

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0d75f2ce61cee164da4e5b7ffb4e69c286e6036a)

rbd-mirror: don't set object map to nonexistent during image sync

The object maps are initialized to nonexistent at snapshot creation,
so this operation is unnecessary.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fe5d4a811f9d9d1160ff7104fe9243cc17ef1bef)

librbd: improve object map debug messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c657752a5067e11405dca59461ce9cbdf062602)

test: snapshot stress test rbd-mirror image sync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 468287bd182fa0cd02f7551ad6d729ff67ab1a22)

rbd-mirror: allocate empty object map during snapshot create

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 57a7fe0f44c5542a72120d0beed8b38e9ccd66db)

librbd: optionally skip object map creation upon snapshot

This feature is used by rbd-mirror during image bootstrap since
it will directly create an empty object map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit db16240c79be7e28d65edac2088352b9b146da5f)

rbd-mirror: refactor snapshot create into its own state machine

Initial implementation will resize the image (if needed) and create
the snapshot.  It is stubbed out for future state machines for handling
object map and clones.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8adc47fbf8efa3991bd247be43b786676fe5fff9)

rbd-mirror: sync snapshot protection status during bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6fba2b2306c8f04ce088fb15ef70e1dac282ff3)

test: stub test cases for rbd-mirror image bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d0f732877213ba9eec9d3158b839c5a4c796e7ab)

test: share journal::MockJournaler between librbd and rbd-mirror

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 04db7ddec661ef195077c3fab06f248705b17220)

Conflicts:
src/test/librbd/test_mock_Journal.cc: MockJournaler moved to new file

Merge pull request #9040 from dillaman/wip-15818

jewel: librbd: potential concurrent event processing during journal replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9038 from dillaman/wip-15817

jewel: journal: incorrectly computed object offset within set

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9039 from dillaman/wip-15820

jewel: librbd: client-side handling for incompatible object map sizes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: guard object map against incompatible image sizes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 42d3d3ac2d33360aae071330cfa34b87c87eadc5)

librbd: properly handle object map open returning error code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 570de56bfc7fd2edd5cf1cf5b7b92084cd3352a3)

librbd: add client-side memory constraint when loading object map

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2dd34b2c7b8c376817aafe5cfdda7aa9cfb60e1a)

Merge pull request #9048 from dillaman/wip-15819

jewel: librbd: constrain size of AioWriteEvent journal entries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: split large AioWrite journal events

Ensure that AioWrite journal events fit within a single
journal object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 70b9fbf2745c795fe345f0298a2f30af80a12985)

journal: prohibit journal entries larger than the soft object size

This avoids complications like a librbd write event of 128MB trying
to fit within a 24MB journal object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d5f307dec10429a1ffc042919b71ac25466a1b72)

Merge pull request #9036 from dillaman/wip-15814

jewel: librbd: fix get/list mirror image status API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fixup to get/list mirror image status API

- embed name and rbd_mirror_image_info_t into rbd_mirror_image_status_t;
- index image status list by image id to make 'start' work correctly;
- provide rbd_mirror_image_status_list_cleanup function.

Fixes: #15771
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit e6c709b3a8be489a900178b8b0d962705d5687e9)

Merge pull request #9003 from dillaman/wip-15743

jewel: librbd: reduce log level for image format 1 warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9010 from dillaman/wip-15736

jewel: rbd-mirror: admin socket commands to start/stop/restart mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: admin socket commands to start/stop/restart mirroring

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit a22cf518f9cc0872f72a8a850ea01558094ebd00)

qa/workunits/rbd: fix rbd-mirror log file name

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 4c28fc2aa378c10ce6e78b1a2f98d5e14ac211e9)

rbd-mirror: make image replayer asok commands available when not started

Initially the asok commands were registered only after the image
replayer start (and unregistered on stop) because their names were
built using remote pool and image names, which became known only after
start.

Now, the asok commands are registered on the image replayer construction
using the temporary name "remote_pool_name/global_image_id". They are
re-registered using "remote_pool_name/remote_image_name" when the
image replayer is started.  Also the commands are not unregistered on
the image replayer stop.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 21790484dc66e97e7625b5b2afabb7efe3a92b08)

qa/workunits/rbd: add env vars to adapt rbd_mirror.sh to teuthology

With these set, this script can work in teuthology, where we want to
control the how daemons run and their log locations, valgrind
settings, etc. The workunit task always sets CEPH_ID, and allows
other env vars to be configured optionally.

The RBD_MIRROR_USE_EXISTING_CLUSTER option can also speed up local testing.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit dcd11a1f262e0c56bb1b37d54283cda06abbf4c7)

qa: rbd_mirror.sh: change parameters to cluster rather than daemon name

Daemon name is only useful to differentiate asok and pid filenames, so
just inline its use there. Everywhere else replace the 'daemon'
parameter with cluster or local_cluster as appropriate.

Remove the CLUSTER{1,2}_DAEMON vars as well, since they are no longer
used.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 5fe64fa806f3bbdbdb950a5643238cc2508814b0)

Merge pull request #9005 from dillaman/wip-15716

jewel: rbd-nbd: fix rbd-nbd aio callback error handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

jewel: librbd: unlock image if journal error encountered during lock

jewel: librbd: unlock image if journal error encountered during lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9009 from dillaman/wip-15713

jewel: librbd: possible race condition leads to use-after-free

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible race condition leads to use-after-free

The invoke async operation state machine can complete before the
owner lock is released.  Use a stack reference to prevent
use-after-free.

Fixes: http://tracker.ceph.com/issues/15690
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d9421a25b582e41550526714b71fb55f05abb1de)

Merge pull request #9004 from dillaman/wip-15708

jewel: rbd: help message distinction between commands and aliases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8972 from Abhishekvrshny/wip-15697-jewel

jewel: python: clone operation will fail if config overridden with "rbd default format = 1"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9008 from dillaman/wip-15698

jewel: rbd-mirror: lockdep error during bootstrap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix for duplicates in list_periods()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit abed30befd86c68922759a66db589285fe01e54b)

rgw: 'period delete' cleans up all period objects

Fixes: http://tracker.ceph.com/issues/15469
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a286b32224d4a9757058415f98a2feff683ac520)

librbd: disable automatic refresh of image upon lock message

There is no need to refresh the image upon request/release/acquire
of the exclusive lock.  The next IO or maintenance op will kick off
the refresh.  This is interfering with the refresh state machine
unit test case (since two concurrent refreshes shouldn't be possible).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ee617a0ca56d752daf62bfffa59219b2c540a428)

librbd: update_features should release lock if acquired

There is a small window where requests are unblocked and before
the image is shut down where new requests can be accepted.  These
are correctly canceled during the shut down, but it can lead to
warning messages.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 608e09851396a9eeb7cf6dc1c7f687ef7a033601)

librbd: avoid applying refreshed image config within librados callback

There is a potential that a synchronous API call could deadlock a
image refresh.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ce5c701bc47b0959f8453b6b92dee4804d3b1d75)

rbd-mirror: don't use temporary string for snap operation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 06faf83bbfab000c8200a9cbe781adb0082b3f7b)

librbd: avoid recursive locking within operation state machine

Fixes: http://tracker.ceph.com/issues/15664
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 91a4890ee78c25391c1548fdacb2b51c46a47415)

Merge pull request #8869 from Abhishekvrshny/wip-15669-jewel

jewel: Errors encountered disabling object-map while flatten is in-progress

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8870 from Abhishekvrshny/wip-15668-jewel

jewel: rbd disk-usage CLI command should support calculating full image usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8868 from Abhishekvrshny/wip-15666-jewel

jewel: rbd CLI to retrieve rbd mirror state for a pool / specific image

Conflicts:
src/tools/rbd_mirror/ImageReplayer.cc: trivial resolution

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9046 from dillaman/wip-fix-snapc-validation-jewel

jewel: librbd: put the validation of image snap context earlier

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: put the validation of image snap context earlier

we'd better validate the snap context right after we got the mutable
metadata before we could go any further

Signed-off-by: runsisi <runsisi@zte.com.cn>
(cherry picked from commit ca8ae66ba42e7a598ad23ed911938589dc288456)

Merge pull request #8867 from Abhishekvrshny/wip-15661-jewel

jewel: Parent image is closed twice if error encountered while opening

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8866 from Abhishekvrshny/wip-15660-jewel

jewel: rbd-mirror can crash if start up is interrupted

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9044 from dillaman/wip-librbd-bad-header-jewel

jewel: librbd: does not crash if image header is too short

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: does not crash if image header is too short

if something goes wrong with the object, and returns a chunk shorter
than expected, don't panic

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7b52183addda83f98c8b1f37d3de20ca0fc4687b)

Merge pull request #8803 from SUSE/wip-15605-jewel

jewel: Potential double free of SetSnapRequest instance

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8865 from Abhishekvrshny/wip-15649-jewel

jewel: possible double-free of object map invalidation request upon error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: helpful error message on map failure

Failure to map an rbd image provides pretty much nothing to
debug (or rectify) the nature of failure. This change tries
to improve the same by providing helpful error messages (by
pointing to dmesg, etc..).

Fixes: http://http://tracker.ceph.com/issues/15816
Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd: assertion to ensure no concurrent processing of replay events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 44827a3e8e02529be84c829a76ef01f70e0d20ee)

journal: suppress notifications if client still in try_pop_front loop

One such example is popping the last entry from an object.  The next
object will be automatically prefetched.  When that object is received,
we do not want to alert the user that entries are available since
try_pop_front already indicated more records were available.

Fixes: http://tracker.ceph.com/issues/15755
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3b8d21ac13865791d8fc4c519f47c1f7d6096e76)

librbd: delay processing of next journal entry until flush in-progress

When replaying a journal flush event, do not start processing the next
journal entry until after the flush is in progress to ensure the barrier
is correctly guarding against future writes.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5d205ef33cb8b5517f141d46a62947dd583500e0)

Merge pull request #8930 from theanalyst/wip-15707-jewel

jewel: multisite: Issues with Deleting Buckets

Merge pull request #9029 from theanalyst/wip-15680

jewel: rgw: radosgw-admin zone set cuts pool names short if name starts with a period

Merge pull request #9022 from yehudasa/wip-rgw-leak-3-jewel

jewel: rgw: leak fixes

Merge pull request #9021 from yehudasa/wip-rgw-xstorageurl-fix-jewel

jewel: rgw: fix printing wrong X-Storage-Url in Swift's TempAuth.