cmake: fix rbd compile errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8d4c00d1bec431f8c721195c81684c4095079c1d)

cmake: add library cls_journal for target unittest_librbd

otherwise if we want to generate target unittest_librbd we need to
(re)generate cls_journal manually

Signed-off-by: runsisi <runsisi@zte.com.cn>
(cherry picked from commit 7efb29416b789fcb7e68aab554d6033e6b5ebb43)

Merge pull request #9063 from dachary/wip-15838-jewel

jewel: tests: make check fails on ext4

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: enable make check on ext4 (part 2)

Followup of 475cc08c330e639b226f2ab57dedbe4495f63022

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b0b3a068d4fcc7b739c5dd3d68297e2d517c378e)

qa/workunits/ceph-helpers.sh: make ceph-osd behave on ext4

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 475cc08c330e639b226f2ab57dedbe4495f63022)

Merge pull request #9052 from dillaman/wip-cmake-rbd-mirror-replay-status-jewel

jewel: cmake: add missing source file to rbd_mirror/image_replayer

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: add missing source file to rbd_mirror/image_replayer

fixes an undefined reference when linking librbd_mirror_internal.a

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2f3f56170ab47fc10d4232101ae2e85398a4c299)

Merge pull request #9051 from athanatos/wip-15815-jewel

Wip 15815 jewel

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_keyvaluedb: clean up test dir

Just like ceph_test_objectstore

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2b075aa52d4475338c0f3cd3360e1ee4de8befa7)

ceph_test_keyvaluedb: fixup unit test for keyvaluedb

Fixes: http://tracker.ceph.com/issues/15435
Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>
(cherry picked from commit 85c14de20936ff9ee224f9b6569a73f9bae75f19)

Merge pull request #9050 from dillaman/wip-15821

jewel: rbd-mirror: image sync needs to handle snapshot size and protection status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: add missing header dependency for MockJournaler

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fa91919c856ac8186545bc8f8da31a8f438a9b5d)

rbd-mirror: image sync object copy was not properly mapping snapshots

When the snapshot sequence is out-of-sync between remote and local
clusters (expected), the objects would not be written with the
correct snapshot context.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dd8f08039bf5354a0413c9da4263d1075243e9c9)

rbd-mirror: clean up image sync debug messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit acca475fe5c9ff024c7f76397e7f49e4c4558b09)

librbd: readahead should not read past end of image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit adb14303ccf8a26ed5536bb692ae63d92ac40a9f)

test: randomize rbd-mirror test case IO

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0d75f2ce61cee164da4e5b7ffb4e69c286e6036a)

rbd-mirror: don't set object map to nonexistent during image sync

The object maps are initialized to nonexistent at snapshot creation,
so this operation is unnecessary.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fe5d4a811f9d9d1160ff7104fe9243cc17ef1bef)

librbd: improve object map debug messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c657752a5067e11405dca59461ce9cbdf062602)

test: snapshot stress test rbd-mirror image sync

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 468287bd182fa0cd02f7551ad6d729ff67ab1a22)

rbd-mirror: allocate empty object map during snapshot create

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 57a7fe0f44c5542a72120d0beed8b38e9ccd66db)

librbd: optionally skip object map creation upon snapshot

This feature is used by rbd-mirror during image bootstrap since
it will directly create an empty object map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit db16240c79be7e28d65edac2088352b9b146da5f)

rbd-mirror: refactor snapshot create into its own state machine

Initial implementation will resize the image (if needed) and create
the snapshot.  It is stubbed out for future state machines for handling
object map and clones.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8adc47fbf8efa3991bd247be43b786676fe5fff9)

rbd-mirror: sync snapshot protection status during bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6fba2b2306c8f04ce088fb15ef70e1dac282ff3)

test: stub test cases for rbd-mirror image bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d0f732877213ba9eec9d3158b839c5a4c796e7ab)

test: share journal::MockJournaler between librbd and rbd-mirror

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 04db7ddec661ef195077c3fab06f248705b17220)

Conflicts:
src/test/librbd/test_mock_Journal.cc: MockJournaler moved to new file

Merge pull request #9040 from dillaman/wip-15818

jewel: librbd: potential concurrent event processing during journal replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9038 from dillaman/wip-15817

jewel: journal: incorrectly computed object offset within set

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9039 from dillaman/wip-15820

jewel: librbd: client-side handling for incompatible object map sizes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: guard object map against incompatible image sizes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 42d3d3ac2d33360aae071330cfa34b87c87eadc5)

librbd: properly handle object map open returning error code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 570de56bfc7fd2edd5cf1cf5b7b92084cd3352a3)

librbd: add client-side memory constraint when loading object map

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2dd34b2c7b8c376817aafe5cfdda7aa9cfb60e1a)

Merge pull request #9048 from dillaman/wip-15819

jewel: librbd: constrain size of AioWriteEvent journal entries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: split large AioWrite journal events

Ensure that AioWrite journal events fit within a single
journal object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 70b9fbf2745c795fe345f0298a2f30af80a12985)

journal: prohibit journal entries larger than the soft object size

This avoids complications like a librbd write event of 128MB trying
to fit within a 24MB journal object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d5f307dec10429a1ffc042919b71ac25466a1b72)

Merge pull request #9036 from dillaman/wip-15814

jewel: librbd: fix get/list mirror image status API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fixup to get/list mirror image status API

- embed name and rbd_mirror_image_info_t into rbd_mirror_image_status_t;
- index image status list by image id to make 'start' work correctly;
- provide rbd_mirror_image_status_list_cleanup function.

Fixes: #15771
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit e6c709b3a8be489a900178b8b0d962705d5687e9)

Merge pull request #9003 from dillaman/wip-15743

jewel: librbd: reduce log level for image format 1 warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9010 from dillaman/wip-15736

jewel: rbd-mirror: admin socket commands to start/stop/restart mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: admin socket commands to start/stop/restart mirroring

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit a22cf518f9cc0872f72a8a850ea01558094ebd00)

qa/workunits/rbd: fix rbd-mirror log file name

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 4c28fc2aa378c10ce6e78b1a2f98d5e14ac211e9)

rbd-mirror: make image replayer asok commands available when not started

Initially the asok commands were registered only after the image
replayer start (and unregistered on stop) because their names were
built using remote pool and image names, which became known only after
start.

Now, the asok commands are registered on the image replayer construction
using the temporary name "remote_pool_name/global_image_id". They are
re-registered using "remote_pool_name/remote_image_name" when the
image replayer is started.  Also the commands are not unregistered on
the image replayer stop.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 21790484dc66e97e7625b5b2afabb7efe3a92b08)

qa/workunits/rbd: add env vars to adapt rbd_mirror.sh to teuthology

With these set, this script can work in teuthology, where we want to
control the how daemons run and their log locations, valgrind
settings, etc. The workunit task always sets CEPH_ID, and allows
other env vars to be configured optionally.

The RBD_MIRROR_USE_EXISTING_CLUSTER option can also speed up local testing.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit dcd11a1f262e0c56bb1b37d54283cda06abbf4c7)

qa: rbd_mirror.sh: change parameters to cluster rather than daemon name

Daemon name is only useful to differentiate asok and pid filenames, so
just inline its use there. Everywhere else replace the 'daemon'
parameter with cluster or local_cluster as appropriate.

Remove the CLUSTER{1,2}_DAEMON vars as well, since they are no longer
used.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 5fe64fa806f3bbdbdb950a5643238cc2508814b0)

Merge pull request #9005 from dillaman/wip-15716

jewel: rbd-nbd: fix rbd-nbd aio callback error handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

jewel: librbd: unlock image if journal error encountered during lock

jewel: librbd: unlock image if journal error encountered during lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9009 from dillaman/wip-15713

jewel: librbd: possible race condition leads to use-after-free

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: possible race condition leads to use-after-free

The invoke async operation state machine can complete before the
owner lock is released.  Use a stack reference to prevent
use-after-free.

Fixes: http://tracker.ceph.com/issues/15690
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d9421a25b582e41550526714b71fb55f05abb1de)

Merge pull request #9004 from dillaman/wip-15708

jewel: rbd: help message distinction between commands and aliases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8972 from Abhishekvrshny/wip-15697-jewel

jewel: python: clone operation will fail if config overridden with "rbd default format = 1"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9008 from dillaman/wip-15698

jewel: rbd-mirror: lockdep error during bootstrap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: disable automatic refresh of image upon lock message

There is no need to refresh the image upon request/release/acquire
of the exclusive lock.  The next IO or maintenance op will kick off
the refresh.  This is interfering with the refresh state machine
unit test case (since two concurrent refreshes shouldn't be possible).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ee617a0ca56d752daf62bfffa59219b2c540a428)

librbd: update_features should release lock if acquired

There is a small window where requests are unblocked and before
the image is shut down where new requests can be accepted.  These
are correctly canceled during the shut down, but it can lead to
warning messages.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 608e09851396a9eeb7cf6dc1c7f687ef7a033601)

librbd: avoid applying refreshed image config within librados callback

There is a potential that a synchronous API call could deadlock a
image refresh.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ce5c701bc47b0959f8453b6b92dee4804d3b1d75)

rbd-mirror: don't use temporary string for snap operation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 06faf83bbfab000c8200a9cbe781adb0082b3f7b)

librbd: avoid recursive locking within operation state machine

Fixes: http://tracker.ceph.com/issues/15664
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 91a4890ee78c25391c1548fdacb2b51c46a47415)

Merge pull request #8869 from Abhishekvrshny/wip-15669-jewel

jewel: Errors encountered disabling object-map while flatten is in-progress

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8870 from Abhishekvrshny/wip-15668-jewel

jewel: rbd disk-usage CLI command should support calculating full image usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8868 from Abhishekvrshny/wip-15666-jewel

jewel: rbd CLI to retrieve rbd mirror state for a pool / specific image

Conflicts:
src/tools/rbd_mirror/ImageReplayer.cc: trivial resolution

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9046 from dillaman/wip-fix-snapc-validation-jewel

jewel: librbd: put the validation of image snap context earlier

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: put the validation of image snap context earlier

we'd better validate the snap context right after we got the mutable
metadata before we could go any further

Signed-off-by: runsisi <runsisi@zte.com.cn>
(cherry picked from commit ca8ae66ba42e7a598ad23ed911938589dc288456)

Merge pull request #8867 from Abhishekvrshny/wip-15661-jewel

jewel: Parent image is closed twice if error encountered while opening

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8866 from Abhishekvrshny/wip-15660-jewel

jewel: rbd-mirror can crash if start up is interrupted

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9044 from dillaman/wip-librbd-bad-header-jewel

jewel: librbd: does not crash if image header is too short

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: does not crash if image header is too short

if something goes wrong with the object, and returns a chunk shorter
than expected, don't panic

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7b52183addda83f98c8b1f37d3de20ca0fc4687b)

Merge pull request #8803 from SUSE/wip-15605-jewel

jewel: Potential double free of SetSnapRequest instance

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8865 from Abhishekvrshny/wip-15649-jewel

jewel: possible double-free of object map invalidation request upon error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: assertion to ensure no concurrent processing of replay events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 44827a3e8e02529be84c829a76ef01f70e0d20ee)

journal: suppress notifications if client still in try_pop_front loop

One such example is popping the last entry from an object.  The next
object will be automatically prefetched.  When that object is received,
we do not want to alert the user that entries are available since
try_pop_front already indicated more records were available.

Fixes: http://tracker.ceph.com/issues/15755
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3b8d21ac13865791d8fc4c519f47c1f7d6096e76)

librbd: delay processing of next journal entry until flush in-progress

When replaying a journal flush event, do not start processing the next
journal entry until after the flush is in progress to ensure the barrier
is correctly guarding against future writes.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5d205ef33cb8b5517f141d46a62947dd583500e0)

Merge pull request #8930 from theanalyst/wip-15707-jewel

jewel: multisite: Issues with Deleting Buckets

Merge pull request #9029 from theanalyst/wip-15680

jewel: rgw: radosgw-admin zone set cuts pool names short if name starts with a period

Merge pull request #9022 from yehudasa/wip-rgw-leak-3-jewel

jewel: rgw: leak fixes

Merge pull request #9021 from yehudasa/wip-rgw-xstorageurl-fix-jewel

jewel: rgw: fix printing wrong X-Storage-Url in Swift's TempAuth.

Merge pull request #9020 from yehudasa/wip-15597-jewel

jewel: rgw: fix issue #15597

Merge pull request #9019 from yehudasa/wip-rgw-admin-zonegroup-jewel

jewel: radosgw-admin: add missing --zonegroup-id to usage

Merge pull request #9018 from yehudasa/wip-15626-jewel

rgw: add AWS4 completion support for RGW_OP_SET_BUCKET_WEBSITE

Merge pull request #9017 from yehudasa/wip-15625-jewel

jewel: rgw: segfault at RGWAsyncGetSystemObj

Merge pull request #9016 from yehudasa/wip-rgw-admin-modify-jewel

jewel: radosgw-admin: update usage for zone[group] modify

Merge pull request #9015 from yehudasa/wip-rgw-zonegroup-remove-jewel

jewel: radosgw-admin: zonegroup remove command

journal: incorrectly computed object offset within set

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f3372a1bcf81fcfd75e7ef5901dbcdc1daa062a3)

test: update rbd integration cram test to remove format 1 warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 098038fad47b181a9ddd11767397082b05405be5)

rgw: fix suffix in RGWZoneParams::fix_pool_names()

Fixes: http://tracker.ceph.com/issues/15598
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6f6b0c925994b0369eda390a6b3fd20005e70c9b)

rgw: code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d53ffbfe65611425b08af71e0d24bfe2a0076f1e)

rgw: upgrade default zonegroup and set correct zone[group] id

Fixes: http://tracker.ceph.com/issues/15597
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 27806c71922eef2f4cde360d7ba40dee9d5bd4bd)

rgw: RGWHTTPClient requests can unregister themselves early

No need to wait for req_state to complete anymore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 91f61d68bf5fc39152d75fbc633f088e17d53d9e)

Conflicts:
src/rgw/rgw_http_client.cc

rgw: move around sync_env.init()

call sync_env.init() even if we're initalized in RGWRemoteDataLog::init()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 3d8442effc1c175e0b7c5064ed41bc642f6cdcb0)

rgw: rados crs, explicit cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1c876eb3873692042c0af3644f8ee3ad15ffdbf5)

rgw: RGWHTTPManager, avoid referring to req_data->client when completing

this is past calling req_data->finish(), client can be destroyed by now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7ada4df124a71eae413e7dec0e25264a8deffcad)

rgw: civetweb_callback() refer to store inside lock

pe->store might change when reconfiguring

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 2ca38dab071c7d20a224a4e1dacfd68548fe52dd)

rgw: RGWPeriodPusher, stop http manager explicitly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 98433e589f2a8b519726ed23b1565c5eb331c68e)

rgw: RGWHTTPManager, can call stop() more than once

destructor calls stop(), but it can be also called explicitly, if caller
wants to ensure requests are not being handled anymore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit bdce81a2ae00f154420386e107a104cbf100d96d)

rgw: RGWReadRESTResourceCR, fix refcounting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d1ec31def71505dc95139fbdd3b49c92fec1d296)

rgw: RGWReadRemoteDataLogShardCR fix destructor

was missing destructor, constructor was broken

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 95162a4c1bf61f7329e363b7efd5686f7e1025f0)

rgw: RGWDataSyncStatusManager, cleanup if failing init

init can be called more than once, ended up with leaking some resources

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 14cd44e0847fd914068f48dd0208ce7c1fe4eecb)

rgw: rest crs, explicitly call cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit a3a3f4f5883f3fc2bf47a7ebfa538206b7ab51d9)

rgw: more leaks fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit cb797d4fd754b4a33eb34c249da3aabc65a7aba3)

rgw: drop a reference to http op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4442d1c19acfc28c2dd4a011365ebebe08750963)

rgw: fix printing wrong X-Storage-Url in Swift's TempAuth.

If the option "rgw_swift_account_in_url" is being set to true,
both user and subuser IDs will be present in the X-Storage-Url
header generated by the implementation of Swift API's TempAuth.
This patch rectifies this behavior and makes that only the user
identifier will be placed there.

Fixes: http://tracker.ceph.com/issues/15667
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit 88f4c67d822388d2038454a25ead7e9a11f404b5)

radosgw-admin: add missing --zonegroup-id to usage

also clarified that --rgw-zone takes the zone by name

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Reported-by: John Wilkins <jowilkin@redhat.com>
(cherry picked from commit 97fad754565b262529073e1b9f8d1799bfdfe0de)

rgw: don't allow any concurrent sync requests on the same key

We used to allow concurrent requests, as long as these had different op,
which wasn't correct.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit edea6d58dd25995bcc1ed4fc5be6f72ce4a6835a)