Merge pull request #10760 from dachary/wip-16865-jewel

jewel: saw valgrind issues in ReplicatedPG::new_repop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10662 from dachary/wip-16901-jewel

jewel: segfault in RGWOp_MDLog_Notify

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #10763 from dachary/wip-17034-jewel

jewel: rgw: object expirer's hints might be trimmed without processing in some circumstances
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10658 from dachary/wip-16862-jewel

jewel: default zone and zonegroup cannot be added to a realm
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10659 from dachary/wip-16863-jewel

jewel: use zone endpoints instead of zonegroup endpoints
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10660 from dachary/wip-16864-jewel

jewel: multisite segfault on ~RGWRealmWatcher if realm was deleted
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10847 from gregsfortytwo/jewel-fix-mds

mds: fix double-unlock on shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10663 from dachary/wip-16934-jewel

jewel: Add zone rename to radosgw_admin

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10664 from dachary/wip-16958-jewel

jewel: Bug when using port 443s in rgw.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10764 from dachary/wip-17032-jewel

jewel: multisite: RGWPeriodPuller tries to pull from itself

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10765 from dachary/wip-17006-jewel

jewel: Increase log level for messages occuring while running rgw admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10845 from cbodley/wip-16702

jewel: rgw: add tenant support to multisite sync

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9405 from SUSE/wip-16083-jewel

jewel: mds: wrongly treat symlink inode as normal file/dir when symlink inode is stale on kcephfs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10654 from dachary/wip-9577-jewel

jewel: mon: "mon metadata" fails when only one monitor exists

Reviewed-by: John Spray <john.spray@redhat.com>

mds: fix double-unlock on shutdown

We did a bad backport or something and accidentally ended up with two Unlock()
calls on mds_lock. Don't.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

rgw: collect skips a specific coroutine stack

Fixes: http://tracker.ceph.com/issues/16665
Instead of drain_all_but() that specifies number of stacks to leave behind,
added drain_all_but_stack() that has a specific stack specified. This is needed
so that we don't call wakeup() through lease_cr->go_down() on a cr stack that
was already collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5a2e8f0526db92a290c711f82627fc5042c290ea)

Merge pull request #10791 from dachary/wip-17004-jewel

jewel: rbd-mirror: FAILED assert(m_state == STATE_STOPPING)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10827 from dillaman/wip-16950-jewel

jewel: librbd: delay acquiring lock if image watch has failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix compilation

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 874de58024e12d893ae050cd421ef67c9521c753)

rgw: remove datalog keys from error repo on ENOENT

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 16976eedb70292e821193e39bb577a68df1bc95a)

test/rgw: add optional --tenant flag to test_multi.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6f65d192cd3917cdcf7d9deb5396750d2c641842)

rgw: add tenant id to GetBucketInstanceInfo

use the rgw_bucket overload of get_bucket_instance_info() so it can
supply the tenant id

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3cf715c6828d7d9732bf6a6fd9b46bbd8c08de5f)

rgw: carry tenant id with data sync

use rgw_bucket_shard to track buckets, which includes tenant id

Fixes: http://tracker.ceph.com/issues/16469
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d945e2099525e39588b572e357b115df98c8cdca)

rgw: add tenant to url for RGWRESTStreamWriteRequest

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2a1e1a776c58f62fbcb57877d56b92eb1e8933e6)

rgw: add tenant to rgw_bucket json format

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fc0df4802b48fa095587b93557c82da5446d9ca3)

rgw: add rgw_bucket_parse_bucket_key()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d5ac140040ea2f17e33d2846870cb033c7ae6ffa)

rgw: add tenant name to datalog entries

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c8aa668fc3e34a7651ce6e2dda62902ced808541)

rgw: convert bucket instance listings back to metadata key format

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3eae201c8a1513bae23233700c754ec076360664)

rgw: use tenant/ for bucket instance metadata keys

to work around the ambiguity of parsing tenant: and :shard in the same
bucket instance metadata key, use tenant/ instead

to preserve backward compatibility with existing objects, new helper
function rgw_bucket_instance_key_to_oid() converts this / back to a :
before being used as an object name

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a0befa1e7be3aeb37209bac2e2a7de0edf5d5a95)

rgw: add get_key() methods to format rgw_buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 51ff8ef7c6bb7c9f55f3ca8ad059b029a4088cdf)

rgw: data sync debug logging

a high level debug logging of the data sync process.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 97ef16c0cc0ad8345b5a897108717d83ab0aa9ab)

rgw: modifying multi-site log messages.

Increasing the log level and removing 'ERROR' from messages that appear during normal multi-site operation.
Fixes: http://tracker.ceph.com/issues/16121
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit e1737ddaa035f23757d021dd34b0c87d24ce372c)

Merge pull request #10653 from dachary/wip-10495-jewel

jewel : 60-ceph-partuuid-workaround-rules still needed by debian jessie (udev 215-17)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10357 from SUSE/wip-16748-jewel

jewel: mount.ceph: move from ceph-base to ceph-common and add symlink in /sbin for SUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10790 from dachary/wip-17005-jewel

jewel: ImageReplayer::is_replaying does not include flush state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10792 from dachary/wip-16978-jewel

jewel: rbd-mirror: FAILED assert(m_on_update_status_finish == nullptr)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10646 from dachary/wip-16576-jewel

jewel: rbd-mirror: FAILED assert(m_local_image_ctx->object_map != nullptr)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10647 from dachary/wip-16593-jewel

jewel: FAILED assert(object_no < m_object_map.size())

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10786 from dachary/wip-17061-jewel

jewel: bashism in src/rbdmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10421 from SUSE/wip-16659-jewel

jewel: ReplicatedBackend doesn't increment stats on pull, only push

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10501 from Abhishekvrshny/wip-16621-jewel

jewel: mds:  tell command blocks forever with async messenger (TestVolumeClient.test_evict_client failure)

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10816 from SUSE/wip-17092-jewel

jewel: build/ops: need rocksdb commit 7ca731b12ce for ppc64le build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: delay acquiring exclusive lock if watch has failed

Fixes: http://tracker.ceph.com/issues/16923
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dfe9f3eac9cca3b83962e0e1b7eac38e6e76d7a5)

librbd: convert ImageWatcher class to template

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 814c305ce8c35b5ce01d7e29a912d5ef3978754b)

 Conflicts:
src/librbd/ImageWatcher.cc: no shrink guard
src/librbd/Operations.cc: no shrink guard

Merge pull request #10103 from xiaoxichen/wip-16037-jewel

jewel: MDSMonitor::check_subs() is very buggy

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10105 from xiaoxichen/wip-16515-jewel

jewel: Session::check_access() is buggy

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10106 from xiaoxichen/wip-16215-jewel

jewel: client: crash in unmount when fuse_use_invalidate_cb is enabled

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10108 from xiaoxichen/wip-16320-jewel

jewel: fuse mounted file systems fails SAMBA CTDB ping_pong rw test with v9.0.2

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10199 from SUSE/wip-16625-jewel

jewel: Failing file operations on kernel based cephfs mount point leaves unaccessible file behind on hammer 0.94.7

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10499 from Abhishekvrshny/wip-16299-jewel

jewel: mds: fix SnapRealm::have_past_parents_open()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10500 from Abhishekvrshny/wip-16620-jewel

jewel: Fix shutting down mds timed-out due to deadlock

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10502 from Abhishekvrshny/wip-16797-jewel

jewel: MDS Deadlock on shutdown active rank while busy with metadata IO

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10104 from xiaoxichen/wip-16560-jewel

jewel: mds: enforce a dirfrag limit on entries

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10074 from ceph/jewel-16002

jewel: ObjectCacher split BufferHead read fix

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

build/ops: bump rocksdb submodule

Fixes a FTBFS on ppc64le

Fixes: http://tracker.ceph.com/issues/17092
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10537 from theanalyst/wip-16778-jewel

jewel: rgw multisite: preserve zone's extra pool

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10655 from dachary/wip-16163-jewel

jewel: rgw: can set negative max_buckets on RGWUserInfo
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10580 from dreamhost/wip-16928-jewel

jewel: rgw: Fix civetweb IPv6
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10216 from SUSE/wip-16637-jewel

jewel: add socket backlog setting for via ceph.conf
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10167 from jmunhoz/aws4-streaming-backport-jewel

jewel: rgw: aws4: add STREAMING-AWS4-HMAC-SHA256-PAYLOAD support
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9544 from Abhishekvrshny/wip-16085-jewel

jewel: A query on a static large object fails with 404 error
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10525 from Abhishekvrshny/wip-16732-jewel

jewel: Bucket index shards orphaned after bucket delete
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10188 from dreamhost/wip-16618-jewel

jewel: rgw: fix multi-delete query param parsing.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9266 from Abhishekvrshny/wip-15964-jewel

jewel: rgw: realm pull fails when using apache frontend
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10710 from rzarzynski/wip-16393-jewel

jewel: rgw: improve support for Swift's object versioning.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10797 from trociny/wip-17080-jewel

jewel: the option 'rbd_cache_writethrough_until_flush=true' dosn't work

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: cache was not switching to writeback after first flush

Fixes: http://tracker.ceph.com/issues/16654
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2d9840af39555ce00246b50e4a5c186798bd88ff)

test: unit test cases for disabling librbd journal by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4a256f5044ac54a8a0a9598ee7fd151b8ea08f84)

Conflicts:
src/test/librbd/CMakeLists.txt (test_ConsistencyGroups.cc)

librbd: utilize factory methods to create AioObjectRequest objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2e5076eb19172919deeb0f4d11f2b2b6cc03fab3)

librbd: convert AioObjectRequest/AioObjectRead classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 840a473e7fb94124ca7c571fc3dd2c787e0d265a)

librbd: move read callback helpers to narrowest scope

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 65b336f68596a430629692a8682c5cfe883c31f1)

librbd: convert AioImageRequest derived classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a945c2c235d9259b1d4a8a579d7e6efc2ed35916)

librbd: removed namespace indentation from legacy classes

Better follows the Ceph C++ style guide

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f1e391982b43ddfb363ff913260460368a6d5834)

librbd: do not record journal events if append is disabled by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 405142c615352613e8bacee46e92484eb0c08f26)

librbd: remove unused journal replay canceled callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f453554edcb16ec72714b2456c08bab9e339b1eb)

librbd: optionally support disabling journal appends via policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f7eeacd02f8f80c5d1d5d9cfed35bbd23f64b018)

librbd: optimize header file dependency & modify related file.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 2185a6275c89d9462611e965da407ea0d504987d)

librbd: optimize operation header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 88afa408683238b803606f3d63de326ead8cefee)

librbd: optimize journal header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 59397e27a04fb0f2b990a86dcaf3c418457d1b72)

librbd: optimize image_watcher header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit aee1559f312e9493d6580e5ffcb606fe21ee3c59)

librbd: optimize image header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 461958ebc7e8ccef76b3f08a979482762267c10b)

librbd: optimize exclusive_lock  header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 01282eb5cdeda7bbbb77438c7d1953d023ea54b6)

librbd: optimize object-map header file dependency

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 1a2276927d5956918f7a4830b4a44048ac090328)

test: fix librbd tests for rbd_skip_partial_discard

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 9717417b7b8aa1b24a82a0bfddbfc23748188641)

Conflicts:
src/rocksdb this was a mistake in the original commit

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Fixes: http://tracker.ceph.com/issues/16386
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit dc41731fbfd73d9fbb63d3ff360d4c5dd62deaf1)

librbd: object_may_exist always return true when you write an empty object

if you write an empty object, object map is updated firstly

Signed-off-by: xinxin shu <shuxinxin@chinac.com>
(cherry picked from commit a54073808146d205d54d6a932a6e0fd750f1dc38)

librbd: ensure that AIO ops are started prior to request

Fixes: http://tracker.ceph.com/issues/16708
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6064f2346de0a8bf2878bf5bfac9a992cda7c4ca)

librbd: helper method for creating and starting AioCompletions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3df7213c0a9f0186e3a37c9e4a10f1c8bc84446e)

Merge pull request #10649 from dachary/wip-16867-jewel

jewel: mkfs.xfs slow performance with discards and object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: potential assertion failure during error-induced shutdown

Fixes: http://tracker.ceph.com/issues/16956
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6a465d9dad417e8b52909c5478f7e3e433748948)

rbd-mirror: potential race condition during failure shutdown

Fixes: http://tracker.ceph.com/issues/16980
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 74ec7c91f17630c77647cfc9813090d688b3410d)

rbd-mirror: replaying state should include flush action

Fixes: http://tracker.ceph.com/issues/16970
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7246f1b771b4d1f336ac11b9e881c9eb32bcd4e1)

doc: added rbdmap man page to RBD restructured index

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4e05cbf7e6eda797a9b08e0081aead19523d5762)

rbdmap: specify bash shell interpreter

Fixes: http://tracker.ceph.com/issues/16608
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 51237c33d1ec4034c5c07f5d63d63838a76bce88)

Merge pull request #10684 from dillaman/wip-16904-jewel

jewel: rbd-mirror: reduce memory footprint during journal replay

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 06a333b116383de4d39c9e098e6e5fa195ceb370)

qa/workunits/rbd: override rbd-mirror integration test poll frequency

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 574be7486ad737892422aed0322f80e5750a75a0)

rbd-mirror: do not cancel maintenance ops with missing finish events

librbd will replay these ops when opening an image, so rbd-mirror
should also ensure these ops are replayed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 862e581553fff510286b58135a1fd69705c06096)

rbd-mirror: potential memory leak when attempting to cancel image sync

The cancel request could race with the actual scheduling of the image
sync operation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6cdf955bad500561ddada2791641eba5fb27762)

rbd-mirror: fix issues detected when attempting clean shut down

Fixed lockdep issue from status update callback and fixed the
potential for a stuck status state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0275c7ca23b27dc5250cd33f317e2273470a9fe8)