rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 06a333b116383de4d39c9e098e6e5fa195ceb370)

qa/workunits/rbd: override rbd-mirror integration test poll frequency

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 574be7486ad737892422aed0322f80e5750a75a0)

rbd-mirror: do not cancel maintenance ops with missing finish events

librbd will replay these ops when opening an image, so rbd-mirror
should also ensure these ops are replayed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 862e581553fff510286b58135a1fd69705c06096)

rbd-mirror: potential memory leak when attempting to cancel image sync

The cancel request could race with the actual scheduling of the image
sync operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6cdf955bad500561ddada2791641eba5fb27762)

rbd-mirror: fix issues detected when attempting clean shut down

Fixed lockdep issue from status update callback and fixed the
potential for a stuck status state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0275c7ca23b27dc5250cd33f317e2273470a9fe8)

rbd-mirror: shut down image replayers in parallel

When multiple pools are being replicated, start the shut down
process concurrently across all pool replayers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 73cdd08007c27d2c3c41fe644601e7a144f21c82)

rbd-mirror: configuration options to control replay throttling

Fixes: http://tracker.ceph.com/issues/16223
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 24883e0605907d1f9bcd1206c8a95c3bde30d5dc)

librbd: new configuration option to restrict journal payload size

Ensure that, by default, IO journal events are broken up into manageable
sizes when factoring in that an rbd-mirror daemon might be replaying
events from thousands of images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 11d7500b9bcda7b7c1d8756ade3373f404257f32)

librbd: wait for journal commit op event to be safely recorded

Operation request op finish events should not be fire and forget.
Instead, ensure the event is committed to the journal before
completing the op. This will avoid several possible split-brain
events during mirroring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 47e0fbf231e52d00069c97b72c57c3158445bcf0)

Conflicts:
src/test/librbd/operation/test_mock_ResizeRequest.cc: no shrink restriction

journal: optimize speed of live replay journal pruning

When streaming playback, avoid the unnecessary watch delay when
one or more entries have been pruned.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 08a8ee98c03b5dfb30341c8d209f0c231b2c5d27)

journal: possible deadlock during flush of journal entries

If a future flush is requested at the exact same moment that an
overflow is detected, the two threads will deadlock since locks
are not taken in a consistent order.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c65471de4b0f54b8ed722f5deaf51ba62632e37)

journal: improve debug log messages

rbd-mirror debugging involved potentially thousands of journals
concurrently running. The instance address will correlate log
messages between journals.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 11475f4fe740cccdfea459ebeabdca8cb94dc911)

journal: support streaming entry playback

Now that it's possible for the ObjectPlayer to only read a
partial subset of available entries, the JournalPlayer needs
to detect that more entries might be available.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 28d5ca16cbcb445f985469413b2a8a3048ab66b7)

journal: replay should only read from a single object set

Previously it was prefetching up to 2 object sets worth of journal
data objects which consumed too much memory.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2666d366645b22a5db2a2bcbfce466726bf0b3c0)

journal: optionally restrict maximum entry payload size

Journal playback will need to read at least a full entry which was
currently limited to the maximum object size. In memory constrained
environment, this new optional limit will set a fix upper bound on
memory usage.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8c1877b82fee0db1dba76252b32ff348226d41a7)

journal: optionally fetch entries in small chunks during replay

Support fetching the full object or incremental chunks (with a
minimum of at least a single decoded entry if available).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f7362e9a57e484fffd840ca0eef01778dcacb65b)

journal: helper class for organizing optional settings

Additional runtime configuration settings will be needed. The
new class will avoid the need to expand the constructor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dad8328f2d502d18923c35f7b86a0cc2ccec133a)

rbd-mirror: preprocess journal events prior to applying

Fixes: http://tracker.ceph.com/issues/16622
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4df913d10b2dd0444db806fccb2812547efa1b56)

rbd-mirror: event preprocessor to handle snap rename operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fdfca557370c9d86acb81d50edb6aafc42044747)

librbd: improve journaling debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 270cb74bc276bfb1f828a6a6933fa827f6cdce42)

Conflicts:
src/librbd/journal/Replay.cc: no snap limit restriction

librbd: separate journal event decoding and processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 57cd75e8058b84b5dce38f3d8f4b7b4138ac6c9a)

librbd: record original snap name in snap rename journal event

Remote peers need a key to map snapshot ids between clusters.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f70b90c48d9520bbb4bb29058375e8205cf63771)

librbd: simple duplicate op checks for all maintenance operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 77699bfe749bc7a898024638fb8347c53fe12123)

Conflicts:
src/test/librbd/mock/MockOperations.h: no shrink restriction

qa/workunits/rbd: exercise snapshot renames within rbd-mirror test

Snapshot rename operations utilize the (cluster) unique snapshot
sequence to prevent attempts at replays. When mirroring to a
different cluster, these sequences will not align.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2f4cb26d8bbd1457bc261547103b56ad40b3c464)

librbd: re-register watch on old format image rename

The watching object name is changed when renaming an old format image,
so unregister the watcher before the rename, and register back after,
to avoid "Transport endpoint is not connected" error.

Fixes: http://tracker.ceph.com/issues/16321
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 1a3973c8814200dc449a557a4fc8212911633985)

rbd-mirror: gracefully restart pool replayer when blacklisted

Fixes: http://tracker.ceph.com/issues/16349
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2f55aa5e33b2fe242ebb9702ba9ff6f8d5cef96c)

rbd-mirror: do not clear image replayer error code after shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 1fc27545c2092c178293ed636b6aadb11bc8cbd3)

rbd-mirror: image deleter should use provided librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 03c2aec4c613f11fcfed5bcd47855005f42abd79)

rbd-mirror: each pool replayer should use its own librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 48f301decbf1f27937bb77a3b47e54933f272d7d)

rbd-mirror: clean up ImageReplayer before stopping state machine

Fixes: http://tracker.ceph.com/issues/16489
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c97f7245a015dbdda25584134840d023fd65cdd1)

Merge pull request #10678 from dillaman/wip-16701-jewel

jewel: rbd-mirror: include local pool id in resync throttle unique key

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10614 from SUSE/wip-16959-jewel

jewel: rpm: OBS needs ExclusiveArch

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10519 from Abhishekvrshny/wip-16312-jewel

jewel: selinux denials in RGW

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10364 from SUSE/wip-16750-jewel

jewel: ceph-osd-prestart.sh contains Upstart-specific code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10008 from dachary/wip-16099-jewel

jewel: ceph-base requires parted

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9917 from SUSE/wip-16461-jewel

jewel: ceph Resource Agent does not work with systemd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10303 from ukernel/jewel-16655

jewel: ceph-fuse: link to libtcmalloc or jemalloc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10420 from SUSE/wip-16798-jewel

jewel: ceph command line tool chokes on ceph –w (the dash is unicode 'en dash' &ndash, copy-paste to reproduce)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10520 from Abhishekvrshny/wip-16319-jewel

jewel: radosgw-admin: inconsistency in uid/email handling

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10073 from SUSE/wip-16565-jewel

jewel: rgw: data sync stops after getting error in all data log sync shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10524 from Abhishekvrshny/wip-16731-jewel

jewel: failed to create bucket after upgrade from hammer to jewel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10523 from Abhishekvrshny/wip-16700-jewel

jewel: rgw: segmentation fault on error_repo in data sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10518 from Abhishekvrshny/wip-16272-jewel

jewel: rgw ldap: fix ldap bindpw parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10144 from dillaman/wip-fix-missing-return-jewel

jewel: librbd: fix missing return statement if failed to get mirror image state

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10552 from ceph/wip-16507-jewel

jewel: expose buffer const_iterator symbols

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10497 from Abhishekvrshny/wip-16586-jewel

jewel: partprobe intermittent issues during ceph-disk prepare

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10716 from ceph/wip-16975

qa: add rados test script for upgrades

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10732 from dillaman/wip-rbd-default-format-jewel

doc: format 2 now is the default image format

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: format 2 now is the default image format

Signed-off-by: Chengwei Yang <yangchengwei@qiyi.com>
(cherry picked from commit c8c92f8a2c2106d11b18c70c4c183c3724c2f3c2)

Merge pull request #10217 from SUSE/wip-16636-jewel

jewel: rgw: document multi tenancy

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9453 from wido/jewel-issue-15348

jewel: rgw: Set Access-Control-Allow-Origin to a Asterisk if allowed in a rule
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10693 from dreamhost/wip-15975-jewel

rgw: Fallback to Host header for bucket name.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa: remove tmap_migrate tests from upgrade testing

tmap_upgrade only exists in jewel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: add rados test script for upgrades

Disable master tests that are not meant to work on jewel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rgw: Fallback to Host header for bucket name.

RGW should fallback to using the Host header as the bucket name if valid &
possible even when it is NOT a suffix match against the DNS names, or a match
against the CNAME rule.

This mirrors AWS S3 behavior for these cases (The AWS S3 does not do any DNS
lookups regardless).

Backport: jewel
Fixes: http://tracker.ceph.com/issues/15975
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 46aae19eeb91bf3ac78a94c9d4812a788d9042a8)

rbd-mirror: include local pool id in resync throttle unique key

Fixes: http://tracker.ceph.com/issues/16536
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8ad36cab7cbf3492bfa972b43e4a5f75a110bfe6)

test: ensure unique template specializations per test case

With optimizations disabled, there can be cross translation unit
symbol leaking that is not present when inlining is enabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 995a16ebcbe79fd14e6de3b25087373744238294)

test: missing template specialization for unittest_rbd_mirror

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 187e2c87b50d64e93736a42fcf449c55e8550125)

qa/workunits/rbd: increase writes in large image count test

This will help to test edge cases where the remote image does
or does not own the exclusive lock when the sync starts.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d16698f96c65b9627cc1f64ef80b7f1e39d69d45)

rbd-mirror: prune sync points referencing missing snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 378914f6b8a8a8c5517fee8fa186268078fbcbfb)

rbd-mirror: ignore empty snapshot sequence mapping

This invalid condition will be gracefully detected and handled
when the snapshot mappings are computed.

Fixes: http://tracker.ceph.com/issues/16329
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 58ed8a18e91401333bc3f3f957ce5d715b6687b6)

qa/workunits/rbd: remove temporary image exports

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 45498d04990861a034e2aad69a93658e018c991a)

rbd-mirror: fix potential image replayer state transition race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3e224c6c03a6a5de0e179bd788387cfa3ff49e9d)

rbd-mirror: cancel image deletion callback on shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c3f1cb3a34e1b4d258877d519e683e25bf65c65a)

rbd-mirror: fixed potential leaking image deletion context callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9d1cd90c220f95d23d2694b4f1fb6b2fed073ced)

rbd-mirror: ImageReplayer doesn't need image deleted callback

The Replayer will detect that the ImageReplayer is stopped and will
schedule the "wait for deleted" callback before attempting to
restart the ImageReplayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7e5afc71dfd3ec3e7b0ddaca96f92ac6e0414006)

rbd-mirror: use async callback when deletion not in-progress

Fixes: http://tracker.ceph.com/issues/16491
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c136f443366fe804057a9c1f8c01c3f0a33a4d63)

rbd-mirror: tests: ImageSyncThrottler unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 8ca9a84f36efa73ad17fd27f545dff3716cd798c)

rbd-mirror: Usage of image-sync throttler in BootstrapRequest

Fixes: http://tracker.ceph.com/issues/15239
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 6a91146255d84229688bd8b378732be5975a778b)

rbd-mirror: Implementation of image-sync throttler

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c4f926d6980d1efd95771885a11d8cc4ebd2e4c3)

rbd-mirror: tests: Support for inflight image sync point update

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit cee543bd96bc99f219024aedbfafc5dcd99abb5a)

rbd-mirror: image-sync: Periodically update sync point object number

Fixes: http://tracker.ceph.com/issues/15108
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c2eedf4d27b73d7eaf3fda54f9b0ee74e455bc1d)

rbd-mirror: image-replayer: Fix bug in resync listener remotion

Fixes: http://tracker.ceph.com/issues/16488
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 11a5851e90bca2c1813156bf12dda0192965db37)

rbd-mirror: resync: Added unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 8953825956f2a560d17c7b651d1724a95f2f7d1d)

rbd-mirror: image-replayer: Implementation of resync operation

Fixes: http://tracker.ceph.com/issues/15670
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit f2b114238103315197a9eefc8490786c484c8520)

rbd: journal: Support for listening updates on client metadata

Currently we only support listening for image resync requests.

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 0dd85739e9034912b86250ced2834dd7fb3d92fd)

journal: Support for registering metadata listeners in the Journaler

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c535eb0399cd649371db233ee1ed01cf5441a4b3)

Merge pull request #9790 from SUSE/wip-16381-jewel

jewel: comparing return code to ERR_NOT_MODIFIED in rgw_rest_s3.cc (needs minus sign)
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10026 from SUSE/wip-16392-jewel

jewel: master: build failures with boost > 1.58

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10625 from dachary/wip-16969-jewel

jewel: src/script/subman fails with KeyError: 'nband'

Reviewed-by: Kefu Chai <kchai@redhat.com>

subman: use replace instead of format

Otherwise all {} are assumed to be substituted. Add a test.

Fixes: http://tracker.ceph.com/issues/16961
Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b006c136761746e12704e349a672325b8b6e51f5)

cmake: script that sets env vars for unit tests

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 15a669435aaa78562746f6728bb185b1e99a4274)

Merge pull request #10054 from dachary/wip-16484-jewel

jewel: ExclusiveLock object leaked when switching to snapshot

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10041 from dachary/wip-16315-jewel

jewel: When journaling is enabled, a flush request shouldn't flush the cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10055 from dachary/wip-16485-jewel

jewel: Whitelist EBUSY error from snap unprotect for journal replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10053 from dachary/wip-16483-jewel

jewel: Close journal and object map before flagging exclusive lock as released

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10052 from dachary/wip-16482-jewel

jewel: Timeout sending mirroring notification shouldn't result in failure

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10051 from dachary/wip-16460-jewel

jewel: Crash when utilizing advisory locking API functions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10050 from dachary/wip-16459-jewel

jewel: rbd-mirror should disable proxied maintenance ops for non-primary image

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10047 from dachary/wip-16426-jewel

jewel: Possible race condition during journal transition from replay to ready

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10046 from dachary/wip-16425-jewel

jewel: rbd-mirror: potential race condition accessing local image journal

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10045 from dachary/wip-16424-jewel

jewel: Journal needs to handle duplicate maintenance op tids

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10044 from dachary/wip-16423-jewel

jewel: Journal duplicate op detection can cause lockdep error

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10043 from dachary/wip-16371-jewel

jewel: rbd-mirror: ensure replay status formatter has completed before stopping replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10042 from dachary/wip-16372-jewel

jewel: Unable to disable journaling feature if in unexpected mirror state

Merge pull request #10010 from dachary/wip-16486-jewel

jewel: Object map/fast-diff invalidated if journal replays the same snap remove event

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10009 from dachary/wip-16514-jewel

jewel: Image removal doesn't necessarily clean up all rbd_mirroring entries

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ExclusiveArch for suse_version

for SLES supports only x86_64 and aarch64 targets
for openSUSE (Tumbleweed and Leap) add ppc64/ppc64le targets.

fixes: http://tracker.ceph.com/issues/16936
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
(cherry picked from commit a8db0f4d96d71223083bf6c2275acd892666a13e)

Merge pull request #9952 from dillaman/wip-fix-use-after-free-jewel

jewel: librbd: potential use after free on refresh error

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9752 from dillaman/wip-fix-task-finisher-jewel

jewel: librbd: cancel all tasks should wait until finisher is done

Reviewed-by: Loic Dachary <ldachary@redhat.com>