rgw: adjust manifest head object

adjust the manifest head object:
 - when reading manifest, set the head object to the object
   we read the manifest from (and its size). Some manifests are
   broken and point at a different object
 - when copying multipart object, set the manifest head object to
   point at the new head object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 419264586fc46337322f0d60b0ffcdcce3bb5c5a)

rgw: adjust objs when copying obj with explicit_objs set

If the head object contains data, need to set it in the list of
objs (in addition to being pointed at by head_obj).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1e012112b99004745952accc6fb11ada5e8e0045)

rgw: patch manifest to handle explicit objs copy issue

Fixes: http://tracker.ceph.com/issues/16435
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b51476466e5bb03bbaa6e428bb6bb189a259b9fd)

Merge pull request #9405 from SUSE/wip-16083-jewel

jewel: mds: wrongly treat symlink inode as normal file/dir when symlink inode is stale on kcephfs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10654 from dachary/wip-9577-jewel

jewel: mon: "mon metadata" fails when only one monitor exists

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10791 from dachary/wip-17004-jewel

jewel: rbd-mirror: FAILED assert(m_state == STATE_STOPPING)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10827 from dillaman/wip-16950-jewel

jewel: librbd: delay acquiring lock if image watch has failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10653 from dachary/wip-10495-jewel

jewel : 60-ceph-partuuid-workaround-rules still needed by debian jessie (udev 215-17)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10357 from SUSE/wip-16748-jewel

jewel: mount.ceph: move from ceph-base to ceph-common and add symlink in /sbin for SUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10790 from dachary/wip-17005-jewel

jewel: ImageReplayer::is_replaying does not include flush state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10792 from dachary/wip-16978-jewel

jewel: rbd-mirror: FAILED assert(m_on_update_status_finish == nullptr)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10646 from dachary/wip-16576-jewel

jewel: rbd-mirror: FAILED assert(m_local_image_ctx->object_map != nullptr)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10647 from dachary/wip-16593-jewel

jewel: FAILED assert(object_no < m_object_map.size())

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10786 from dachary/wip-17061-jewel

jewel: bashism in src/rbdmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10421 from SUSE/wip-16659-jewel

jewel: ReplicatedBackend doesn't increment stats on pull, only push

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10501 from Abhishekvrshny/wip-16621-jewel

jewel: mds:  tell command blocks forever with async messenger (TestVolumeClient.test_evict_client failure)

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10816 from SUSE/wip-17092-jewel

jewel: build/ops: need rocksdb commit 7ca731b12ce for ppc64le build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: delay acquiring exclusive lock if watch has failed

Fixes: http://tracker.ceph.com/issues/16923
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dfe9f3eac9cca3b83962e0e1b7eac38e6e76d7a5)

librbd: convert ImageWatcher class to template

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 814c305ce8c35b5ce01d7e29a912d5ef3978754b)

 Conflicts:
src/librbd/ImageWatcher.cc: no shrink guard
src/librbd/Operations.cc: no shrink guard

Merge pull request #10103 from xiaoxichen/wip-16037-jewel

jewel: MDSMonitor::check_subs() is very buggy

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10105 from xiaoxichen/wip-16515-jewel

jewel: Session::check_access() is buggy

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10106 from xiaoxichen/wip-16215-jewel

jewel: client: crash in unmount when fuse_use_invalidate_cb is enabled

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10108 from xiaoxichen/wip-16320-jewel

jewel: fuse mounted file systems fails SAMBA CTDB ping_pong rw test with v9.0.2

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10199 from SUSE/wip-16625-jewel

jewel: Failing file operations on kernel based cephfs mount point leaves unaccessible file behind on hammer 0.94.7

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10499 from Abhishekvrshny/wip-16299-jewel

jewel: mds: fix SnapRealm::have_past_parents_open()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10500 from Abhishekvrshny/wip-16620-jewel

jewel: Fix shutting down mds timed-out due to deadlock

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10502 from Abhishekvrshny/wip-16797-jewel

jewel: MDS Deadlock on shutdown active rank while busy with metadata IO

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10104 from xiaoxichen/wip-16560-jewel

jewel: mds: enforce a dirfrag limit on entries

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10074 from ceph/jewel-16002

jewel: ObjectCacher split BufferHead read fix

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

build/ops: bump rocksdb submodule

Fixes a FTBFS on ppc64le

Fixes: http://tracker.ceph.com/issues/17092
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10537 from theanalyst/wip-16778-jewel

jewel: rgw multisite: preserve zone's extra pool

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10655 from dachary/wip-16163-jewel

jewel: rgw: can set negative max_buckets on RGWUserInfo
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10580 from dreamhost/wip-16928-jewel

jewel: rgw: Fix civetweb IPv6
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10216 from SUSE/wip-16637-jewel

jewel: add socket backlog setting for via ceph.conf
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10167 from jmunhoz/aws4-streaming-backport-jewel

jewel: rgw: aws4: add STREAMING-AWS4-HMAC-SHA256-PAYLOAD support
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9544 from Abhishekvrshny/wip-16085-jewel

jewel: A query on a static large object fails with 404 error
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10525 from Abhishekvrshny/wip-16732-jewel

jewel: Bucket index shards orphaned after bucket delete
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10188 from dreamhost/wip-16618-jewel

jewel: rgw: fix multi-delete query param parsing.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9266 from Abhishekvrshny/wip-15964-jewel

jewel: rgw: realm pull fails when using apache frontend
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10710 from rzarzynski/wip-16393-jewel

jewel: rgw: improve support for Swift's object versioning.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10797 from trociny/wip-17080-jewel

jewel: the option 'rbd_cache_writethrough_until_flush=true' dosn't work

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: cache was not switching to writeback after first flush

Fixes: http://tracker.ceph.com/issues/16654
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2d9840af39555ce00246b50e4a5c186798bd88ff)

test: unit test cases for disabling librbd journal by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4a256f5044ac54a8a0a9598ee7fd151b8ea08f84)

Conflicts:
src/test/librbd/CMakeLists.txt (test_ConsistencyGroups.cc)

librbd: utilize factory methods to create AioObjectRequest objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2e5076eb19172919deeb0f4d11f2b2b6cc03fab3)

librbd: convert AioObjectRequest/AioObjectRead classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 840a473e7fb94124ca7c571fc3dd2c787e0d265a)

librbd: move read callback helpers to narrowest scope

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 65b336f68596a430629692a8682c5cfe883c31f1)

librbd: convert AioImageRequest derived classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a945c2c235d9259b1d4a8a579d7e6efc2ed35916)

librbd: removed namespace indentation from legacy classes

Better follows the Ceph C++ style guide

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f1e391982b43ddfb363ff913260460368a6d5834)

librbd: do not record journal events if append is disabled by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 405142c615352613e8bacee46e92484eb0c08f26)

librbd: remove unused journal replay canceled callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f453554edcb16ec72714b2456c08bab9e339b1eb)

librbd: optionally support disabling journal appends via policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f7eeacd02f8f80c5d1d5d9cfed35bbd23f64b018)

librbd: optimize header file dependency & modify related file.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 2185a6275c89d9462611e965da407ea0d504987d)

librbd: optimize operation header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 88afa408683238b803606f3d63de326ead8cefee)

librbd: optimize journal header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 59397e27a04fb0f2b990a86dcaf3c418457d1b72)

librbd: optimize image_watcher header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit aee1559f312e9493d6580e5ffcb606fe21ee3c59)

librbd: optimize image header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 461958ebc7e8ccef76b3f08a979482762267c10b)

librbd: optimize exclusive_lock  header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 01282eb5cdeda7bbbb77438c7d1953d023ea54b6)

librbd: optimize object-map header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 1a2276927d5956918f7a4830b4a44048ac090328)

test: fix librbd tests for rbd_skip_partial_discard

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 9717417b7b8aa1b24a82a0bfddbfc23748188641)

Conflicts:
src/rocksdb this was a mistake in the original commit

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Fixes: http://tracker.ceph.com/issues/16386
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit dc41731fbfd73d9fbb63d3ff360d4c5dd62deaf1)

librbd: object_may_exist always return true when you write an empty object

if you write an empty object, object map is updated firstly

Signed-off-by: xinxin shu <shuxinxin@chinac.com>
(cherry picked from commit a54073808146d205d54d6a932a6e0fd750f1dc38)

librbd: ensure that AIO ops are started prior to request

Fixes: http://tracker.ceph.com/issues/16708
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6064f2346de0a8bf2878bf5bfac9a992cda7c4ca)

librbd: helper method for creating and starting AioCompletions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3df7213c0a9f0186e3a37c9e4a10f1c8bc84446e)

Merge pull request #10649 from dachary/wip-16867-jewel

jewel: mkfs.xfs slow performance with discards and object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: potential assertion failure during error-induced shutdown

Fixes: http://tracker.ceph.com/issues/16956
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6a465d9dad417e8b52909c5478f7e3e433748948)

rbd-mirror: potential race condition during failure shutdown

Fixes: http://tracker.ceph.com/issues/16980
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 74ec7c91f17630c77647cfc9813090d688b3410d)

rbd-mirror: replaying state should include flush action

Fixes: http://tracker.ceph.com/issues/16970
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7246f1b771b4d1f336ac11b9e881c9eb32bcd4e1)

doc: added rbdmap man page to RBD restructured index

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4e05cbf7e6eda797a9b08e0081aead19523d5762)

rbdmap: specify bash shell interpreter

Fixes: http://tracker.ceph.com/issues/16608
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 51237c33d1ec4034c5c07f5d63d63838a76bce88)

Merge pull request #10684 from dillaman/wip-16904-jewel

jewel: rbd-mirror: reduce memory footprint during journal replay

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 06a333b116383de4d39c9e098e6e5fa195ceb370)

qa/workunits/rbd: override rbd-mirror integration test poll frequency

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 574be7486ad737892422aed0322f80e5750a75a0)

rbd-mirror: do not cancel maintenance ops with missing finish events

librbd will replay these ops when opening an image, so rbd-mirror
should also ensure these ops are replayed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 862e581553fff510286b58135a1fd69705c06096)

rbd-mirror: potential memory leak when attempting to cancel image sync

The cancel request could race with the actual scheduling of the image
sync operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6cdf955bad500561ddada2791641eba5fb27762)

rbd-mirror: fix issues detected when attempting clean shut down

Fixed lockdep issue from status update callback and fixed the
potential for a stuck status state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0275c7ca23b27dc5250cd33f317e2273470a9fe8)

rbd-mirror: shut down image replayers in parallel

When multiple pools are being replicated, start the shut down
process concurrently across all pool replayers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 73cdd08007c27d2c3c41fe644601e7a144f21c82)

rbd-mirror: configuration options to control replay throttling

Fixes: http://tracker.ceph.com/issues/16223
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 24883e0605907d1f9bcd1206c8a95c3bde30d5dc)

librbd: new configuration option to restrict journal payload size

Ensure that, by default, IO journal events are broken up into manageable
sizes when factoring in that an rbd-mirror daemon might be replaying
events from thousands of images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 11d7500b9bcda7b7c1d8756ade3373f404257f32)

librbd: wait for journal commit op event to be safely recorded

Operation request op finish events should not be fire and forget.
Instead, ensure the event is committed to the journal before
completing the op. This will avoid several possible split-brain
events during mirroring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 47e0fbf231e52d00069c97b72c57c3158445bcf0)

Conflicts:
src/test/librbd/operation/test_mock_ResizeRequest.cc: no shrink restriction

journal: optimize speed of live replay journal pruning

When streaming playback, avoid the unnecessary watch delay when
one or more entries have been pruned.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 08a8ee98c03b5dfb30341c8d209f0c231b2c5d27)

journal: possible deadlock during flush of journal entries

If a future flush is requested at the exact same moment that an
overflow is detected, the two threads will deadlock since locks
are not taken in a consistent order.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c65471de4b0f54b8ed722f5deaf51ba62632e37)

journal: improve debug log messages

rbd-mirror debugging involved potentially thousands of journals
concurrently running. The instance address will correlate log
messages between journals.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 11475f4fe740cccdfea459ebeabdca8cb94dc911)

journal: support streaming entry playback

Now that it's possible for the ObjectPlayer to only read a
partial subset of available entries, the JournalPlayer needs
to detect that more entries might be available.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 28d5ca16cbcb445f985469413b2a8a3048ab66b7)

journal: replay should only read from a single object set

Previously it was prefetching up to 2 object sets worth of journal
data objects which consumed too much memory.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2666d366645b22a5db2a2bcbfce466726bf0b3c0)

journal: optionally restrict maximum entry payload size

Journal playback will need to read at least a full entry which was
currently limited to the maximum object size. In memory constrained
environment, this new optional limit will set a fix upper bound on
memory usage.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8c1877b82fee0db1dba76252b32ff348226d41a7)

journal: optionally fetch entries in small chunks during replay

Support fetching the full object or incremental chunks (with a
minimum of at least a single decoded entry if available).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f7362e9a57e484fffd840ca0eef01778dcacb65b)

journal: helper class for organizing optional settings

Additional runtime configuration settings will be needed. The
new class will avoid the need to expand the constructor.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dad8328f2d502d18923c35f7b86a0cc2ccec133a)

rbd-mirror: preprocess journal events prior to applying

Fixes: http://tracker.ceph.com/issues/16622
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4df913d10b2dd0444db806fccb2812547efa1b56)

rbd-mirror: event preprocessor to handle snap rename operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fdfca557370c9d86acb81d50edb6aafc42044747)

librbd: improve journaling debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 270cb74bc276bfb1f828a6a6933fa827f6cdce42)

Conflicts:
src/librbd/journal/Replay.cc: no snap limit restriction

librbd: separate journal event decoding and processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 57cd75e8058b84b5dce38f3d8f4b7b4138ac6c9a)

librbd: record original snap name in snap rename journal event

Remote peers need a key to map snapshot ids between clusters.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f70b90c48d9520bbb4bb29058375e8205cf63771)

librbd: simple duplicate op checks for all maintenance operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 77699bfe749bc7a898024638fb8347c53fe12123)

Conflicts:
src/test/librbd/mock/MockOperations.h: no shrink restriction

qa/workunits/rbd: exercise snapshot renames within rbd-mirror test

Snapshot rename operations utilize the (cluster) unique snapshot
sequence to prevent attempts at replays. When mirroring to a
different cluster, these sequences will not align.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2f4cb26d8bbd1457bc261547103b56ad40b3c464)

librbd: re-register watch on old format image rename

The watching object name is changed when renaming an old format image,
so unregister the watcher before the rename, and register back after,
to avoid "Transport endpoint is not connected" error.

Fixes: http://tracker.ceph.com/issues/16321
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 1a3973c8814200dc449a557a4fc8212911633985)

rbd-mirror: gracefully restart pool replayer when blacklisted

Fixes: http://tracker.ceph.com/issues/16349
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2f55aa5e33b2fe242ebb9702ba9ff6f8d5cef96c)

rbd-mirror: do not clear image replayer error code after shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 1fc27545c2092c178293ed636b6aadb11bc8cbd3)

rbd-mirror: image deleter should use provided librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 03c2aec4c613f11fcfed5bcd47855005f42abd79)

rbd-mirror: each pool replayer should use its own librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 48f301decbf1f27937bb77a3b47e54933f272d7d)

rbd-mirror: clean up ImageReplayer before stopping state machine

Fixes: http://tracker.ceph.com/issues/16489
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c97f7245a015dbdda25584134840d023fd65cdd1)