librbd: metadata retrieval added to open image state machine

Fixes: http://tracker.ceph.com/issues/15928
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b64cb31f8e7796175b4709929c017b3236649462)

cls_rbd: async version of metadata_list helper method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 985cb38211c51c95d84479df231c4f53847cb2ec)

Merge pull request #9232 from jdurgin/wip-mirror-workunit-jewel

jewel: qa/workunits/rbd: fixed rbd_mirror teuthology runtime errors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: fixed rbd_mirror teuthology runtime errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8ef09c4d8d3c7c6fbc9b6f7ef17cca0179e73468)

Merge pull request #9226 from dillaman/wip-15950

jewel: rbd-mirror: potential crash during image status update

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9069 from dillaman/fixup-mirror_image_status_list-jewel

jewel: cls::rbd: mirror_image_status_list returned max 64 items

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

journal: reset watch step after pruning expired tag

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 48c2f4e079000021b6ebe807bb1dba0dc1a6a11d)

rbd-mirror: additional debug messages during image replayer start/stop

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit afc891f7fcb000d106b1d722386f6d81ddeddab6)

rbd-mirror: ensure proper handling of status updates during shutdown

Previously, several shutdown race conditions could occur due to the
use of the async work queue for scheduling updates.

Fixes: http://tracker.ceph.com/issues/15909
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9ae4edc05660665c3a135536a7c51d4070d28a40)

rbd-mirror: track bootstrap state within image status

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e5dd97b4bf4a993911febf34dce90470a2a91d59)

rbd-mirror: combine ImageReplayer stopped and uninitialized states

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a8f6dde4dbae8039380f31e026030a22a89fb4bd)

rbd-mirror: lock ordering issue in status update callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5c04c1361074ca9e9df1bc1c2dd82dfafc2a1304)

rbd-mirror: don't unregister asok commands if image replayer start failed

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 2fd6fdd3dc9ecb915362f5e3aa8a224210273540)

rbd-mirror: avoid potential deadlock

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 46bf727d43af4975beeb514630a99d59cfb0a406)

Merge pull request #9217 from dillaman/wip-15945

jewel: journal: live replay might skip entries from previous object set

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: rbd-mirror daemon stress test

This test repeatedly runs rbd bench-write, kills the process
randomly to create an unclean journal shutdown, and verifies
that the image content replicates correctly.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e2ab3128a37f4f36c9a1de6ddf2c0fe2e48aa7e2)

journal: helper method to detect newer tags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0a8a6126ea35344e85af7eb64ffc490938edba51)

journal: skip partially complete tag entries during playback

If a journal client does not fully write out its buffered entries
before quiting, replay should skip over all remaining out-of-
sequence entries for the tag.

Fixes: http://tracker.ceph.com/issues/15864
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9454f7e4c62437b1c288f371009feba1fd374584)

journal: close, advance, and open object set ordering

Flush in-flight appends to open objects before advancing the
active object set.  Additionally, don't start recording to the
new objects until after advancing the active set.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit de830057d0f7286914e019540c6263423cb60428)

journal: new ObjectRecorder closed callback

The callback will be invoked if there were in-flight appends
when the close was requested.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit bba91437dbe3b7a9b6da8a61ccc4c597858c8efc)

journal: do not flag append as full if already known to be full

Once an object has overflowed or a close is in-progress, re-attempting
to close the object is not needed since the async process is already
underway.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 45620dc312ca2aeaf084638144aaa4783b622c4a)

journal: delay object overflow event until in-flight appends settled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ee8d311a1fcdf7a85cdc168e1740fbaa252e87d3)

journal: ignore flush on closed/overflowed object

The journal would be in-progress on transitioning to a new
object recorder in a newer object set.  Once the records
re-attach to the new object player they will automatically
flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a13cb821f71b6b1b3378464bcc2d8318eb0da48a)

journal: implicitly detach future's flush handler on append

If the future is already in-flight, there is no purpose served
by requesting the future be flushed to disk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 1cb9b1a1b25d0f4c2d9a9e22661636e195fc70a0)

journal: async callback for advancing the active object set

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ca89abfd9e6f0e8edc3234afea387757ef694826)

journal: re-fetch active object before advancing set during replay

During a live replay, it's possible that an append and and overflow
into the next object could race with the live playback of the same
object.  Re-fetch an "empty" object at least once before advancing
to next set to ensure all records have been read.

Fixes: http://tracker.ceph.com/issues/15665
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6056f8c45c99bd37cb18933a37cc238c7e9a7c7d)

Merge pull request #9210 from yehudasa/wip-15926-jewel

radosgw-admin: fix 'period push' handling of --url

radosgw-admin: fix 'period push' handling of --url

was calling send_to_remote_gateway(), but passing 'url' instead of
'remote'. now uses send_to_remote_or_url() to accept either

Fixes: http://tracker.ceph.com/issues/15926
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9194 from ceph/wip-jewel-no-lttng-global-link

Do not link lttng into libglobal

Reviewed-by: Boris Ranto <branto@redhat.com>

global: don't link lttng into libglobal

Rely on dynamic initialization instead. Linking lttng in this way had
the unfortunate side effect of causing radosgw to segfault (when
daemonized) during sigterm processing (ie. during lttng_ust_exit()).

This was originally removed in 638738f, but accidentally re-added via
5f61d36.

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #9180 from dillaman/wip-15746-jewel

jewel: doc: update mirroring guide to include pool/image status commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: update mirroring guide to include pool/image status commands

Fixes: http://tracker.ceph.com/issues/15746
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 4381e341c435c80793b847115eed616b8359c4e6)

doc: fixup: "rbd-mirror daemon" instead of "rbd-daemon"

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit ddd6345f8f7edcff638ed435aef316ba3e0e40dd)

Merge pull request #8815 from ktdreyer/wip-15646-jewel-system-targets

jewel: debian: install systemd target files

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: dh_systemd_start is in the dh-systemd package

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 89154d1e01dbc58dc92f37741031f7ddb0448d2e)

debian: install systemd target files

* enable it using dh_systemd_enable
* start the target using dh_systemd_start
* move the dh_installinit, dh_systemd_enable, dh_systemd_start calls
  down, so they can identify the service files if they care about them.

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f950a0a7f5b35e13668379bc69dcb5d98a2dfc1f)

10.2.1

Merge pull request #8853 from ceph/wip-jewel-backports

jewel: several backports

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9041 from vshankar/jewel-15721-backport

jewel: rbd: helpful error message on map failure

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #9081 from yehudasa/wip-rgw-period-commit-jewel

rgw: period commit fix

Merge pull request #8969 from Abhishekvrshny/wip-15731-jewel

jewel: MDSAuthCap parse no longer fails on paths with hyphens

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8970 from Abhishekvrshny/wip-15732-jewel

jewel: MDS incarnation no longer gets lost after remove filesystem

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8971 from Abhishekvrshny/wip-15738-jewel

jewel: handle standby-replay nodes properly in upgrades

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9054 from yehudasa/wip-rgw-admin-output-jewel

jewel: rgw admin output

Merge pull request #9053 from yehudasa/wip-15745-jewel

jewel: rgw: handle stripe transition when flushing final pending_data_bl

Merge pull request #9047 from ceph/wip-rgw-period-delete-jewel

jewel: rgw: period delete fixes

Merge pull request #9076 from dillaman/wip-cmake-test-rbd-mirror-jewel

jewel: cmake: fix rbd compile errors

cmake: fix rbd compile errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8d4c00d1bec431f8c721195c81684c4095079c1d)

cmake: add library cls_journal for target unittest_librbd

otherwise if we want to generate target unittest_librbd we need to
(re)generate cls_journal manually

Signed-off-by: runsisi <runsisi@zte.com.cn>
(cherry picked from commit 7efb29416b789fcb7e68aab554d6033e6b5ebb43)

cls::rbd: mirror_image_status_list returned max 64 items

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit e925ce86243c91e6d64ff8d7b45166e4e3f2650b)

Merge pull request #9063 from dachary/wip-15838-jewel

jewel: tests: make check fails on ext4

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: enable make check on ext4 (part 2)

Followup of 475cc08c330e639b226f2ab57dedbe4495f63022

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b0b3a068d4fcc7b739c5dd3d68297e2d517c378e)

qa/workunits/ceph-helpers.sh: make ceph-osd behave on ext4

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 475cc08c330e639b226f2ab57dedbe4495f63022)

radosgw-admin: 'period commit' sends to new master zone by default

if no --remote or --url is given, set --remote = master_zone_id to
override the rest_master_conn (which refers to the old master zone)

Fixes: http://tracker.ceph.com/issues/15828
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fbe1a05355549dfe0c05750683eeb61e275950c2)

radosgw-admin: allow --remote to specify zone or zonegroup id

--remote was only looking for zonegroup connections. extended to check
for zone connections as well

updated the usage to clarify that it expects zone/zonegroup ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9fcff89ee1e60009fa367e33c915a90354c9a9cc)

radosgw-admin: 'zonegroup add' prints updated zonegroup

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d355208023fe08a7e16f1078c872286048c4bcc6)

radosgw-admin: 'zonegroup modify' prints updated zonegroup

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a033f40caf60083111886aa804fcdbf3ee60aef8)

radosgw-admin: 'zone modify' prints updated zone

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8a639014a6f2dd950062aa405bbee1762ca23e09)

rgw: handle stripe transition when flushing final pending_data_bl

Fixes: http://tracker.ceph.com/issues/15745
When complete_writing_data() is called, if pending_data_bl is not empty
we still need to handle stripe transition correctly. If pending_data_bl
has more data that we can allow in current stripe, move to the next one.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b7a7d41839558b1a8786a66e27d9d80e606f61c7)

Merge pull request #9052 from dillaman/wip-cmake-rbd-mirror-replay-status-jewel

jewel: cmake: add missing source file to rbd_mirror/image_replayer

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: add missing source file to rbd_mirror/image_replayer

fixes an undefined reference when linking librbd_mirror_internal.a

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2f3f56170ab47fc10d4232101ae2e85398a4c299)

Merge pull request #9051 from athanatos/wip-15815-jewel

Wip 15815 jewel

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_keyvaluedb: clean up test dir

Just like ceph_test_objectstore

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2b075aa52d4475338c0f3cd3360e1ee4de8befa7)

ceph_test_keyvaluedb: fixup unit test for keyvaluedb

Fixes: http://tracker.ceph.com/issues/15435
Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>
(cherry picked from commit 85c14de20936ff9ee224f9b6569a73f9bae75f19)

Merge pull request #9050 from dillaman/wip-15821

jewel: rbd-mirror: image sync needs to handle snapshot size and protection status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: add missing header dependency for MockJournaler

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fa91919c856ac8186545bc8f8da31a8f438a9b5d)

rbd-mirror: image sync object copy was not properly mapping snapshots

When the snapshot sequence is out-of-sync between remote and local
clusters (expected), the objects would not be written with the
correct snapshot context.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dd8f08039bf5354a0413c9da4263d1075243e9c9)

rbd-mirror: clean up image sync debug messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit acca475fe5c9ff024c7f76397e7f49e4c4558b09)

librbd: readahead should not read past end of image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit adb14303ccf8a26ed5536bb692ae63d92ac40a9f)

test: randomize rbd-mirror test case IO

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0d75f2ce61cee164da4e5b7ffb4e69c286e6036a)

rbd-mirror: don't set object map to nonexistent during image sync

The object maps are initialized to nonexistent at snapshot creation,
so this operation is unnecessary.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fe5d4a811f9d9d1160ff7104fe9243cc17ef1bef)

librbd: improve object map debug messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c657752a5067e11405dca59461ce9cbdf062602)

test: snapshot stress test rbd-mirror image sync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 468287bd182fa0cd02f7551ad6d729ff67ab1a22)

rbd-mirror: allocate empty object map during snapshot create

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 57a7fe0f44c5542a72120d0beed8b38e9ccd66db)

librbd: optionally skip object map creation upon snapshot

This feature is used by rbd-mirror during image bootstrap since
it will directly create an empty object map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit db16240c79be7e28d65edac2088352b9b146da5f)

rbd-mirror: refactor snapshot create into its own state machine

Initial implementation will resize the image (if needed) and create
the snapshot.  It is stubbed out for future state machines for handling
object map and clones.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8adc47fbf8efa3991bd247be43b786676fe5fff9)

rbd-mirror: sync snapshot protection status during bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6fba2b2306c8f04ce088fb15ef70e1dac282ff3)

test: stub test cases for rbd-mirror image bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d0f732877213ba9eec9d3158b839c5a4c796e7ab)

test: share journal::MockJournaler between librbd and rbd-mirror

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 04db7ddec661ef195077c3fab06f248705b17220)

Conflicts:
src/test/librbd/test_mock_Journal.cc: MockJournaler moved to new file

Merge pull request #9040 from dillaman/wip-15818

jewel: librbd: potential concurrent event processing during journal replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9038 from dillaman/wip-15817

jewel: journal: incorrectly computed object offset within set

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9039 from dillaman/wip-15820

jewel: librbd: client-side handling for incompatible object map sizes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: guard object map against incompatible image sizes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 42d3d3ac2d33360aae071330cfa34b87c87eadc5)

librbd: properly handle object map open returning error code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 570de56bfc7fd2edd5cf1cf5b7b92084cd3352a3)

librbd: add client-side memory constraint when loading object map

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2dd34b2c7b8c376817aafe5cfdda7aa9cfb60e1a)

Merge pull request #9048 from dillaman/wip-15819

jewel: librbd: constrain size of AioWriteEvent journal entries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: split large AioWrite journal events

Ensure that AioWrite journal events fit within a single
journal object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 70b9fbf2745c795fe345f0298a2f30af80a12985)

journal: prohibit journal entries larger than the soft object size

This avoids complications like a librbd write event of 128MB trying
to fit within a 24MB journal object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d5f307dec10429a1ffc042919b71ac25466a1b72)

Merge pull request #9036 from dillaman/wip-15814

jewel: librbd: fix get/list mirror image status API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fixup to get/list mirror image status API

- embed name and rbd_mirror_image_info_t into rbd_mirror_image_status_t;
- index image status list by image id to make 'start' work correctly;
- provide rbd_mirror_image_status_list_cleanup function.

Fixes: #15771
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit e6c709b3a8be489a900178b8b0d962705d5687e9)

Merge pull request #9003 from dillaman/wip-15743

jewel: librbd: reduce log level for image format 1 warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9010 from dillaman/wip-15736

jewel: rbd-mirror: admin socket commands to start/stop/restart mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: admin socket commands to start/stop/restart mirroring

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit a22cf518f9cc0872f72a8a850ea01558094ebd00)

qa/workunits/rbd: fix rbd-mirror log file name

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 4c28fc2aa378c10ce6e78b1a2f98d5e14ac211e9)

rbd-mirror: make image replayer asok commands available when not started

Initially the asok commands were registered only after the image
replayer start (and unregistered on stop) because their names were
built using remote pool and image names, which became known only after
start.

Now, the asok commands are registered on the image replayer construction
using the temporary name "remote_pool_name/global_image_id". They are
re-registered using "remote_pool_name/remote_image_name" when the
image replayer is started.  Also the commands are not unregistered on
the image replayer stop.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 21790484dc66e97e7625b5b2afabb7efe3a92b08)

qa/workunits/rbd: add env vars to adapt rbd_mirror.sh to teuthology

With these set, this script can work in teuthology, where we want to
control the how daemons run and their log locations, valgrind
settings, etc. The workunit task always sets CEPH_ID, and allows
other env vars to be configured optionally.

The RBD_MIRROR_USE_EXISTING_CLUSTER option can also speed up local testing.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit dcd11a1f262e0c56bb1b37d54283cda06abbf4c7)

qa: rbd_mirror.sh: change parameters to cluster rather than daemon name

Daemon name is only useful to differentiate asok and pid filenames, so
just inline its use there. Everywhere else replace the 'daemon'
parameter with cluster or local_cluster as appropriate.

Remove the CLUSTER{1,2}_DAEMON vars as well, since they are no longer
used.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 5fe64fa806f3bbdbdb950a5643238cc2508814b0)

Merge pull request #9005 from dillaman/wip-15716

jewel: rbd-nbd: fix rbd-nbd aio callback error handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

jewel: librbd: unlock image if journal error encountered during lock

jewel: librbd: unlock image if journal error encountered during lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9009 from dillaman/wip-15713

jewel: librbd: possible race condition leads to use-after-free

Reviewed-by: Jason Dillaman <dillaman@redhat.com>