librbd: convert AioImageRequest derived classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a945c2c235d9259b1d4a8a579d7e6efc2ed35916)

librbd: removed namespace indentation from legacy classes

Better follows the Ceph C++ style guide

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f1e391982b43ddfb363ff913260460368a6d5834)

librbd: do not record journal events if append is disabled by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 405142c615352613e8bacee46e92484eb0c08f26)

librbd: remove unused journal replay canceled callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f453554edcb16ec72714b2456c08bab9e339b1eb)

librbd: optionally support disabling journal appends via policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f7eeacd02f8f80c5d1d5d9cfed35bbd23f64b018)

librbd: optimize header file dependency & modify related file.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 2185a6275c89d9462611e965da407ea0d504987d)

librbd: optimize operation header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 88afa408683238b803606f3d63de326ead8cefee)

librbd: optimize journal header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 59397e27a04fb0f2b990a86dcaf3c418457d1b72)

librbd: optimize image_watcher header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit aee1559f312e9493d6580e5ffcb606fe21ee3c59)

librbd: optimize image header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 461958ebc7e8ccef76b3f08a979482762267c10b)

librbd: optimize exclusive_lock  header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 01282eb5cdeda7bbbb77438c7d1953d023ea54b6)

librbd: optimize object-map header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 1a2276927d5956918f7a4830b4a44048ac090328)

test: fix librbd tests for rbd_skip_partial_discard

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 9717417b7b8aa1b24a82a0bfddbfc23748188641)

Conflicts:
src/rocksdb this was a mistake in the original commit

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Fixes: http://tracker.ceph.com/issues/16386
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit dc41731fbfd73d9fbb63d3ff360d4c5dd62deaf1)

librbd: object_may_exist always return true when you write an empty object

if you write an empty object, object map is updated firstly

Signed-off-by: xinxin shu <shuxinxin@chinac.com>
(cherry picked from commit a54073808146d205d54d6a932a6e0fd750f1dc38)

librbd: ensure that AIO ops are started prior to request

Fixes: http://tracker.ceph.com/issues/16708
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6064f2346de0a8bf2878bf5bfac9a992cda7c4ca)

librbd: helper method for creating and starting AioCompletions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3df7213c0a9f0186e3a37c9e4a10f1c8bc84446e)

Merge pull request #10649 from dachary/wip-16867-jewel

jewel: mkfs.xfs slow performance with discards and object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10684 from dillaman/wip-16904-jewel

jewel: rbd-mirror: reduce memory footprint during journal replay

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 06a333b116383de4d39c9e098e6e5fa195ceb370)

qa/workunits/rbd: override rbd-mirror integration test poll frequency

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 574be7486ad737892422aed0322f80e5750a75a0)

rbd-mirror: do not cancel maintenance ops with missing finish events

librbd will replay these ops when opening an image, so rbd-mirror
should also ensure these ops are replayed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 862e581553fff510286b58135a1fd69705c06096)

rbd-mirror: potential memory leak when attempting to cancel image sync

The cancel request could race with the actual scheduling of the image
sync operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6cdf955bad500561ddada2791641eba5fb27762)

rbd-mirror: fix issues detected when attempting clean shut down

Fixed lockdep issue from status update callback and fixed the
potential for a stuck status state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0275c7ca23b27dc5250cd33f317e2273470a9fe8)

rbd-mirror: shut down image replayers in parallel

When multiple pools are being replicated, start the shut down
process concurrently across all pool replayers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 73cdd08007c27d2c3c41fe644601e7a144f21c82)

rbd-mirror: configuration options to control replay throttling

Fixes: http://tracker.ceph.com/issues/16223
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 24883e0605907d1f9bcd1206c8a95c3bde30d5dc)

librbd: new configuration option to restrict journal payload size

Ensure that, by default, IO journal events are broken up into manageable
sizes when factoring in that an rbd-mirror daemon might be replaying
events from thousands of images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 11d7500b9bcda7b7c1d8756ade3373f404257f32)

librbd: wait for journal commit op event to be safely recorded

Operation request op finish events should not be fire and forget.
Instead, ensure the event is committed to the journal before
completing the op. This will avoid several possible split-brain
events during mirroring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 47e0fbf231e52d00069c97b72c57c3158445bcf0)

Conflicts:
src/test/librbd/operation/test_mock_ResizeRequest.cc: no shrink restriction

journal: optimize speed of live replay journal pruning

When streaming playback, avoid the unnecessary watch delay when
one or more entries have been pruned.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 08a8ee98c03b5dfb30341c8d209f0c231b2c5d27)

journal: possible deadlock during flush of journal entries

If a future flush is requested at the exact same moment that an
overflow is detected, the two threads will deadlock since locks
are not taken in a consistent order.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c65471de4b0f54b8ed722f5deaf51ba62632e37)

journal: improve debug log messages

rbd-mirror debugging involved potentially thousands of journals
concurrently running. The instance address will correlate log
messages between journals.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 11475f4fe740cccdfea459ebeabdca8cb94dc911)

journal: support streaming entry playback

Now that it's possible for the ObjectPlayer to only read a
partial subset of available entries, the JournalPlayer needs
to detect that more entries might be available.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 28d5ca16cbcb445f985469413b2a8a3048ab66b7)

journal: replay should only read from a single object set

Previously it was prefetching up to 2 object sets worth of journal
data objects which consumed too much memory.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2666d366645b22a5db2a2bcbfce466726bf0b3c0)

journal: optionally restrict maximum entry payload size

Journal playback will need to read at least a full entry which was
currently limited to the maximum object size. In memory constrained
environment, this new optional limit will set a fix upper bound on
memory usage.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8c1877b82fee0db1dba76252b32ff348226d41a7)

journal: optionally fetch entries in small chunks during replay

Support fetching the full object or incremental chunks (with a
minimum of at least a single decoded entry if available).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f7362e9a57e484fffd840ca0eef01778dcacb65b)

journal: helper class for organizing optional settings

Additional runtime configuration settings will be needed. The
new class will avoid the need to expand the constructor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dad8328f2d502d18923c35f7b86a0cc2ccec133a)

rbd-mirror: preprocess journal events prior to applying

Fixes: http://tracker.ceph.com/issues/16622
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4df913d10b2dd0444db806fccb2812547efa1b56)

rbd-mirror: event preprocessor to handle snap rename operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fdfca557370c9d86acb81d50edb6aafc42044747)

librbd: improve journaling debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 270cb74bc276bfb1f828a6a6933fa827f6cdce42)

Conflicts:
src/librbd/journal/Replay.cc: no snap limit restriction

librbd: separate journal event decoding and processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 57cd75e8058b84b5dce38f3d8f4b7b4138ac6c9a)

librbd: record original snap name in snap rename journal event

Remote peers need a key to map snapshot ids between clusters.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f70b90c48d9520bbb4bb29058375e8205cf63771)

librbd: simple duplicate op checks for all maintenance operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 77699bfe749bc7a898024638fb8347c53fe12123)

Conflicts:
src/test/librbd/mock/MockOperations.h: no shrink restriction

qa/workunits/rbd: exercise snapshot renames within rbd-mirror test

Snapshot rename operations utilize the (cluster) unique snapshot
sequence to prevent attempts at replays. When mirroring to a
different cluster, these sequences will not align.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2f4cb26d8bbd1457bc261547103b56ad40b3c464)

librbd: re-register watch on old format image rename

The watching object name is changed when renaming an old format image,
so unregister the watcher before the rename, and register back after,
to avoid "Transport endpoint is not connected" error.

Fixes: http://tracker.ceph.com/issues/16321
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 1a3973c8814200dc449a557a4fc8212911633985)

rbd-mirror: gracefully restart pool replayer when blacklisted

Fixes: http://tracker.ceph.com/issues/16349
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2f55aa5e33b2fe242ebb9702ba9ff6f8d5cef96c)

rbd-mirror: do not clear image replayer error code after shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 1fc27545c2092c178293ed636b6aadb11bc8cbd3)

rbd-mirror: image deleter should use provided librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 03c2aec4c613f11fcfed5bcd47855005f42abd79)

rbd-mirror: each pool replayer should use its own librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 48f301decbf1f27937bb77a3b47e54933f272d7d)

rbd-mirror: clean up ImageReplayer before stopping state machine

Fixes: http://tracker.ceph.com/issues/16489
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c97f7245a015dbdda25584134840d023fd65cdd1)

Merge pull request #10679 from dillaman/wip-16735-jewel

jewel: rbd-nbd does not properly handle resize notifications

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10148 from SUSE/wip-16599-jewel

jewel: rgw: Swift API returns double space usage and objects of account metadata

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #10678 from dillaman/wip-16701-jewel

jewel: rbd-mirror: include local pool id in resync throttle unique key

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10614 from SUSE/wip-16959-jewel

jewel: rpm: OBS needs ExclusiveArch

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10519 from Abhishekvrshny/wip-16312-jewel

jewel: selinux denials in RGW

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10364 from SUSE/wip-16750-jewel

jewel: ceph-osd-prestart.sh contains Upstart-specific code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10008 from dachary/wip-16099-jewel

jewel: ceph-base requires parted

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9917 from SUSE/wip-16461-jewel

jewel: ceph Resource Agent does not work with systemd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10303 from ukernel/jewel-16655

jewel: ceph-fuse: link to libtcmalloc or jemalloc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10420 from SUSE/wip-16798-jewel

jewel: ceph command line tool chokes on ceph –w (the dash is unicode 'en dash' &ndash, copy-paste to reproduce)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10520 from Abhishekvrshny/wip-16319-jewel

jewel: radosgw-admin: inconsistency in uid/email handling

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10073 from SUSE/wip-16565-jewel

jewel: rgw: data sync stops after getting error in all data log sync shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10524 from Abhishekvrshny/wip-16731-jewel

jewel: failed to create bucket after upgrade from hammer to jewel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10523 from Abhishekvrshny/wip-16700-jewel

jewel: rgw: segmentation fault on error_repo in data sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10518 from Abhishekvrshny/wip-16272-jewel

jewel: rgw ldap: fix ldap bindpw parsing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10144 from dillaman/wip-fix-missing-return-jewel

jewel: librbd: fix missing return statement if failed to get mirror image state

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10552 from ceph/wip-16507-jewel

jewel: expose buffer const_iterator symbols

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10497 from Abhishekvrshny/wip-16586-jewel

jewel: partprobe intermittent issues during ceph-disk prepare

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10716 from ceph/wip-16975

qa: add rados test script for upgrades

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10732 from dillaman/wip-rbd-default-format-jewel

doc: format 2 now is the default image format

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: format 2 now is the default image format

Signed-off-by: Chengwei Yang <yangchengwei@qiyi.com>
(cherry picked from commit c8c92f8a2c2106d11b18c70c4c183c3724c2f3c2)

Merge pull request #10217 from SUSE/wip-16636-jewel

jewel: rgw: document multi tenancy

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9453 from wido/jewel-issue-15348

jewel: rgw: Set Access-Control-Allow-Origin to a Asterisk if allowed in a rule
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10693 from dreamhost/wip-15975-jewel

rgw: Fallback to Host header for bucket name.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa: remove tmap_migrate tests from upgrade testing

tmap_upgrade only exists in jewel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: add rados test script for upgrades

Disable master tests that are not meant to work on jewel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rgw: Fallback to Host header for bucket name.

RGW should fallback to using the Host header as the bucket name if valid &
possible even when it is NOT a suffix match against the DNS names, or a match
against the CNAME rule.

This mirrors AWS S3 behavior for these cases (The AWS S3 does not do any DNS
lookups regardless).

Backport: jewel
Fixes: http://tracker.ceph.com/issues/15975
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 46aae19eeb91bf3ac78a94c9d4812a788d9042a8)

rbd-mirror: include local pool id in resync throttle unique key

Fixes: http://tracker.ceph.com/issues/16536
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8ad36cab7cbf3492bfa972b43e4a5f75a110bfe6)

test: ensure unique template specializations per test case

With optimizations disabled, there can be cross translation unit
symbol leaking that is not present when inlining is enabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 995a16ebcbe79fd14e6de3b25087373744238294)

test: missing template specialization for unittest_rbd_mirror

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 187e2c87b50d64e93736a42fcf449c55e8550125)

qa/workunits/rbd: increase writes in large image count test

This will help to test edge cases where the remote image does
or does not own the exclusive lock when the sync starts.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d16698f96c65b9627cc1f64ef80b7f1e39d69d45)

rbd-mirror: prune sync points referencing missing snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 378914f6b8a8a8c5517fee8fa186268078fbcbfb)

rbd-mirror: ignore empty snapshot sequence mapping

This invalid condition will be gracefully detected and handled
when the snapshot mappings are computed.

Fixes: http://tracker.ceph.com/issues/16329
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 58ed8a18e91401333bc3f3f957ce5d715b6687b6)

qa/workunits/rbd: remove temporary image exports

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 45498d04990861a034e2aad69a93658e018c991a)

rbd-mirror: fix potential image replayer state transition race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3e224c6c03a6a5de0e179bd788387cfa3ff49e9d)

rbd-mirror: cancel image deletion callback on shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c3f1cb3a34e1b4d258877d519e683e25bf65c65a)

rbd-mirror: fixed potential leaking image deletion context callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9d1cd90c220f95d23d2694b4f1fb6b2fed073ced)

rbd-mirror: ImageReplayer doesn't need image deleted callback

The Replayer will detect that the ImageReplayer is stopped and will
schedule the "wait for deleted" callback before attempting to
restart the ImageReplayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7e5afc71dfd3ec3e7b0ddaca96f92ac6e0414006)

rbd-mirror: use async callback when deletion not in-progress

Fixes: http://tracker.ceph.com/issues/16491
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c136f443366fe804057a9c1f8c01c3f0a33a4d63)

rbd-mirror: tests: ImageSyncThrottler unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 8ca9a84f36efa73ad17fd27f545dff3716cd798c)

rbd-mirror: Usage of image-sync throttler in BootstrapRequest

Fixes: http://tracker.ceph.com/issues/15239
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 6a91146255d84229688bd8b378732be5975a778b)

rbd-mirror: Implementation of image-sync throttler

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c4f926d6980d1efd95771885a11d8cc4ebd2e4c3)

rbd-mirror: tests: Support for inflight image sync point update

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit cee543bd96bc99f219024aedbfafc5dcd99abb5a)

rbd-mirror: image-sync: Periodically update sync point object number

Fixes: http://tracker.ceph.com/issues/15108
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c2eedf4d27b73d7eaf3fda54f9b0ee74e455bc1d)

rbd-mirror: image-replayer: Fix bug in resync listener remotion

Fixes: http://tracker.ceph.com/issues/16488
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 11a5851e90bca2c1813156bf12dda0192965db37)

rbd-mirror: resync: Added unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 8953825956f2a560d17c7b651d1724a95f2f7d1d)

rbd-mirror: image-replayer: Implementation of resync operation

Fixes: http://tracker.ceph.com/issues/15670
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit f2b114238103315197a9eefc8490786c484c8520)

rbd: journal: Support for listening updates on client metadata

Currently we only support listening for image resync requests.

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 0dd85739e9034912b86250ced2834dd7fb3d92fd)

journal: Support for registering metadata listeners in the Journaler

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c535eb0399cd649371db233ee1ed01cf5441a4b3)

test: fsx: fix rbd-nbd daemon logging

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 09710c46d3811e602250f858fe7292925c1e3c06)

qa/workunits/rbd: image resize test for rbd-nbd

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit fa58acbb6fcc79e5a9bfd4b86250a0e2b9c8efa1)