rgw: forward input data when forwarding set_bucket_version to master

Fixes: http://tracker.ceph.com/issues/16494
Needed to keep input data around to be forwarded correctly. Also, master
does not send any data back, so don't try to parse anything.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1eec00bef1e5540bf3e31d1f8fb1645eb64b1e62)

rgw: set args when rebuilding req_info

Was missing args, so it didn't sign requests with subresources
correctly when forwarding.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit dd1068c54e50671d8904b99189823418a5a5bc07)

Merge pull request #10656 from dachary/wip-16564-jewel

jewel: cors auto memleak

Merge pull request #11252 from ceph/wip-offline-split-jewel

ceph-objectstore-tool: add a way to split filestore directories offline

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10866 from dachary/wip-17122-jewel

jewel: COPY broke multipart files uploaded under dumpling

Merge pull request #10890 from dachary/wip-17140-jewel

jewel: rgw: period commit loses zonegroup changes: region_map converted repeatedly

Merge pull request #11021 from cbodley/wip-16794

jewel: rgw: back off bucket sync on failures, don't store marker

Merge pull request #10657 from dachary/wip-16792-jewel

jewel: metadata sync can skip markers for failed/incomplete entries

ceph-objectstore-tool: add a way to split filestore directories offline

Use the usual split code, and split each dir that meets the
usual split criteria.

This can be run with lower than usual split settings, to avoid costly
online splits. To make sure the directories are not merged again, use
a load merge threshold (e.g. filestore merge threshold = 1), and
adjust the split multiplier accordingly.

Fixes: http://tracker.ceph.com/issues/17220
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit e7b0428e0e8d8f5459311dc698d94a3ac4f04684)

Conflicts:
src/os/filestore/FileStore.cc
src/tools/ceph_objectstore_tool.cc
(trivial resolution)

Merge pull request #10888 from dachary/wip-17062-jewel

jewel: tests: fix TestClsRbd.mirror_image failure in upgrade:jewel-x-master-distro-basic-vps

10.2.3

rgw: fix collection of object sync errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit aacc31f72cef6f151459e4a0543850edeeaf1938)

rgw: fix marker tracker completion handling

Was not tracking high markers correctly. Could only work if there was a single
hole in the completion range. Just keep a map of all the complete entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f5801c726efdc2f3067a071e6bb5ac83fd0cd147)

rgw: collect() stops if error encoutered

and returns true if needed to be called again

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5528932e4c327a0fb0ce34a676eb59707a575325)

rgw: back off bucket sync on failures, don't store marker

Fixes: http://tracker.ceph.com/issues/16742
If we fail on any single entry in bucket, skip updating the marker tracker
so that next time we'll go over that entry, and back off. This will trigger
a report to the data sync error repo and eventually a retry on the failing
object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1f3fec807043fd313ef7c66ff48f18b82c8bfa66)

Merge pull request #10804 from dachary/wip-17089-jewel

jewel: OSD failed to subscribe skipped osdmaps after ceph osd pause

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10760 from dachary/wip-16865-jewel

jewel: saw valgrind issues in ReplicatedPG::new_repop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10662 from dachary/wip-16901-jewel

jewel: segfault in RGWOp_MDLog_Notify

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #10763 from dachary/wip-17034-jewel

jewel: rgw: object expirer's hints might be trimmed without processing in some circumstances
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10658 from dachary/wip-16862-jewel

jewel: default zone and zonegroup cannot be added to a realm
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10659 from dachary/wip-16863-jewel

jewel: use zone endpoints instead of zonegroup endpoints
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10660 from dachary/wip-16864-jewel

jewel: multisite segfault on ~RGWRealmWatcher if realm was deleted
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: delete region map after upgrade to zonegroup map

convert_regionmap() reads the region map and uses it to initialize the
zonegroup map.  but it doesn't remove the region_map afterwards, so
radosgw (and some radosgw-admin commands) will keep doing this on
startup, overwriting any changes made to the period/zonegroup map

Fixes: http://tracker.ceph.com/issues/17051
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 215cd6f6ff7b8cb74df6dc3b94a5928f56ab9540)

qa/workunits: support filtering cls_rbd unit test cases

Fixes: http://tracker.ceph.com/issues/16529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0891ede57cd254aa75c66301fde4a2eb6957a3d6)

Merge pull request #10847 from gregsfortytwo/jewel-fix-mds

mds: fix double-unlock on shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10663 from dachary/wip-16934-jewel

jewel: Add zone rename to radosgw_admin

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10664 from dachary/wip-16958-jewel

jewel: Bug when using port 443s in rgw.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10764 from dachary/wip-17032-jewel

jewel: multisite: RGWPeriodPuller tries to pull from itself

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10765 from dachary/wip-17006-jewel

jewel: Increase log level for messages occuring while running rgw admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10845 from cbodley/wip-16702

jewel: rgw: add tenant support to multisite sync

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: adjust manifest head object

adjust the manifest head object:
 - when reading manifest, set the head object to the object
   we read the manifest from (and its size). Some manifests are
   broken and point at a different object
 - when copying multipart object, set the manifest head object to
   point at the new head object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 419264586fc46337322f0d60b0ffcdcce3bb5c5a)

rgw: adjust objs when copying obj with explicit_objs set

If the head object contains data, need to set it in the list of
objs (in addition to being pointed at by head_obj).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1e012112b99004745952accc6fb11ada5e8e0045)

rgw: patch manifest to handle explicit objs copy issue

Fixes: http://tracker.ceph.com/issues/16435
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b51476466e5bb03bbaa6e428bb6bb189a259b9fd)

Merge pull request #9405 from SUSE/wip-16083-jewel

jewel: mds: wrongly treat symlink inode as normal file/dir when symlink inode is stale on kcephfs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10654 from dachary/wip-9577-jewel

jewel: mon: "mon metadata" fails when only one monitor exists

Reviewed-by: John Spray <john.spray@redhat.com>

mds: fix double-unlock on shutdown

We did a bad backport or something and accidentally ended up with two Unlock()
calls on mds_lock. Don't.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

rgw: collect skips a specific coroutine stack

Fixes: http://tracker.ceph.com/issues/16665
Instead of drain_all_but() that specifies number of stacks to leave behind,
added drain_all_but_stack() that has a specific stack specified. This is needed
so that we don't call wakeup() through lease_cr->go_down() on a cr stack that
was already collected.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5a2e8f0526db92a290c711f82627fc5042c290ea)

Merge pull request #10791 from dachary/wip-17004-jewel

jewel: rbd-mirror: FAILED assert(m_state == STATE_STOPPING)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10827 from dillaman/wip-16950-jewel

jewel: librbd: delay acquiring lock if image watch has failed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix compilation

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 874de58024e12d893ae050cd421ef67c9521c753)

rgw: remove datalog keys from error repo on ENOENT

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 16976eedb70292e821193e39bb577a68df1bc95a)

test/rgw: add optional --tenant flag to test_multi.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6f65d192cd3917cdcf7d9deb5396750d2c641842)

rgw: add tenant id to GetBucketInstanceInfo

use the rgw_bucket overload of get_bucket_instance_info() so it can
supply the tenant id

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3cf715c6828d7d9732bf6a6fd9b46bbd8c08de5f)

rgw: carry tenant id with data sync

use rgw_bucket_shard to track buckets, which includes tenant id

Fixes: http://tracker.ceph.com/issues/16469
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d945e2099525e39588b572e357b115df98c8cdca)

rgw: add tenant to url for RGWRESTStreamWriteRequest

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2a1e1a776c58f62fbcb57877d56b92eb1e8933e6)

rgw: add tenant to rgw_bucket json format

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fc0df4802b48fa095587b93557c82da5446d9ca3)

rgw: add rgw_bucket_parse_bucket_key()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d5ac140040ea2f17e33d2846870cb033c7ae6ffa)

rgw: add tenant name to datalog entries

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c8aa668fc3e34a7651ce6e2dda62902ced808541)

rgw: convert bucket instance listings back to metadata key format

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3eae201c8a1513bae23233700c754ec076360664)

rgw: use tenant/ for bucket instance metadata keys

to work around the ambiguity of parsing tenant: and :shard in the same
bucket instance metadata key, use tenant/ instead

to preserve backward compatibility with existing objects, new helper
function rgw_bucket_instance_key_to_oid() converts this / back to a :
before being used as an object name

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a0befa1e7be3aeb37209bac2e2a7de0edf5d5a95)

rgw: add get_key() methods to format rgw_buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 51ff8ef7c6bb7c9f55f3ca8ad059b029a4088cdf)

rgw: data sync debug logging

a high level debug logging of the data sync process.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 97ef16c0cc0ad8345b5a897108717d83ab0aa9ab)

rgw: modifying multi-site log messages.

Increasing the log level and removing 'ERROR' from messages that appear during normal multi-site operation.
Fixes: http://tracker.ceph.com/issues/16121
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit e1737ddaa035f23757d021dd34b0c87d24ce372c)

Merge pull request #10653 from dachary/wip-10495-jewel

jewel : 60-ceph-partuuid-workaround-rules still needed by debian jessie (udev 215-17)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10357 from SUSE/wip-16748-jewel

jewel: mount.ceph: move from ceph-base to ceph-common and add symlink in /sbin for SUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10790 from dachary/wip-17005-jewel

jewel: ImageReplayer::is_replaying does not include flush state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10792 from dachary/wip-16978-jewel

jewel: rbd-mirror: FAILED assert(m_on_update_status_finish == nullptr)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10646 from dachary/wip-16576-jewel

jewel: rbd-mirror: FAILED assert(m_local_image_ctx->object_map != nullptr)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10647 from dachary/wip-16593-jewel

jewel: FAILED assert(object_no < m_object_map.size())

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10786 from dachary/wip-17061-jewel

jewel: bashism in src/rbdmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10421 from SUSE/wip-16659-jewel

jewel: ReplicatedBackend doesn't increment stats on pull, only push

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10501 from Abhishekvrshny/wip-16621-jewel

jewel: mds:  tell command blocks forever with async messenger (TestVolumeClient.test_evict_client failure)

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10816 from SUSE/wip-17092-jewel

jewel: build/ops: need rocksdb commit 7ca731b12ce for ppc64le build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: delay acquiring exclusive lock if watch has failed

Fixes: http://tracker.ceph.com/issues/16923
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dfe9f3eac9cca3b83962e0e1b7eac38e6e76d7a5)

librbd: convert ImageWatcher class to template

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 814c305ce8c35b5ce01d7e29a912d5ef3978754b)

 Conflicts:
src/librbd/ImageWatcher.cc: no shrink guard
src/librbd/Operations.cc: no shrink guard

Merge pull request #10103 from xiaoxichen/wip-16037-jewel

jewel: MDSMonitor::check_subs() is very buggy

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10105 from xiaoxichen/wip-16515-jewel

jewel: Session::check_access() is buggy

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10106 from xiaoxichen/wip-16215-jewel

jewel: client: crash in unmount when fuse_use_invalidate_cb is enabled

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10108 from xiaoxichen/wip-16320-jewel

jewel: fuse mounted file systems fails SAMBA CTDB ping_pong rw test with v9.0.2

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10199 from SUSE/wip-16625-jewel

jewel: Failing file operations on kernel based cephfs mount point leaves unaccessible file behind on hammer 0.94.7

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10499 from Abhishekvrshny/wip-16299-jewel

jewel: mds: fix SnapRealm::have_past_parents_open()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10500 from Abhishekvrshny/wip-16620-jewel

jewel: Fix shutting down mds timed-out due to deadlock

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10502 from Abhishekvrshny/wip-16797-jewel

jewel: MDS Deadlock on shutdown active rank while busy with metadata IO

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10104 from xiaoxichen/wip-16560-jewel

jewel: mds: enforce a dirfrag limit on entries

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10074 from ceph/jewel-16002

jewel: ObjectCacher split BufferHead read fix

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

build/ops: bump rocksdb submodule

Fixes a FTBFS on ppc64le

Fixes: http://tracker.ceph.com/issues/17092
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10537 from theanalyst/wip-16778-jewel

jewel: rgw multisite: preserve zone's extra pool

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10655 from dachary/wip-16163-jewel

jewel: rgw: can set negative max_buckets on RGWUserInfo
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10580 from dreamhost/wip-16928-jewel

jewel: rgw: Fix civetweb IPv6
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10216 from SUSE/wip-16637-jewel

jewel: add socket backlog setting for via ceph.conf
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10167 from jmunhoz/aws4-streaming-backport-jewel

jewel: rgw: aws4: add STREAMING-AWS4-HMAC-SHA256-PAYLOAD support
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9544 from Abhishekvrshny/wip-16085-jewel

jewel: A query on a static large object fails with 404 error
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10525 from Abhishekvrshny/wip-16732-jewel

jewel: Bucket index shards orphaned after bucket delete
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10188 from dreamhost/wip-16618-jewel

jewel: rgw: fix multi-delete query param parsing.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9266 from Abhishekvrshny/wip-15964-jewel

jewel: rgw: realm pull fails when using apache frontend
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10710 from rzarzynski/wip-16393-jewel

jewel: rgw: improve support for Swift's object versioning.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

osd: set objecter.osdmap when starts

if any OSD is full or the pause flag is set in cluster, objecter will
always try to subscribe the next osdmap using the mon client shared with
OSD. but if the OSD's osdmap is very far behind on osd maps due to some
reasons, `MonClient::sub_want_increment()` will reject the subscribe
request from OSD, because it's asking for older maps while the mon client
is about to send the request from objecter for the *next* osdmap.

so we need to update objecter with the latest local osdmap, so it is
able to skip the new osdmaps if OSD needs to catch up with the cluster
first.

Fixes: http://tracker.ceph.com/issues/17023
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 1515e08e7d425f01d2e9d02c34b2ae40cdf2611a)

mon/MonClient: should update sub_sent with sub_new

std::map::insert() does not overwrite existing items with the same key.
so we need to do this in a different way. and we will check sub_sent
aftewards, so we need to keep an updated sub_sent around.

Fixes: http://tracker.ceph.com/issues/17023
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 550814900ebca9dd89e088b9abe24f66eb5e3e6d)

Merge pull request #10797 from trociny/wip-17080-jewel

jewel: the option 'rbd_cache_writethrough_until_flush=true' dosn't work

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: cache was not switching to writeback after first flush

Fixes: http://tracker.ceph.com/issues/16654
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2d9840af39555ce00246b50e4a5c186798bd88ff)

test: unit test cases for disabling librbd journal by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4a256f5044ac54a8a0a9598ee7fd151b8ea08f84)

Conflicts:
src/test/librbd/CMakeLists.txt (test_ConsistencyGroups.cc)

librbd: utilize factory methods to create AioObjectRequest objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2e5076eb19172919deeb0f4d11f2b2b6cc03fab3)

librbd: convert AioObjectRequest/AioObjectRead classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 840a473e7fb94124ca7c571fc3dd2c787e0d265a)

librbd: move read callback helpers to narrowest scope

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 65b336f68596a430629692a8682c5cfe883c31f1)

librbd: convert AioImageRequest derived classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a945c2c235d9259b1d4a8a579d7e6efc2ed35916)

librbd: removed namespace indentation from legacy classes

Better follows the Ceph C++ style guide

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f1e391982b43ddfb363ff913260460368a6d5834)

librbd: do not record journal events if append is disabled by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 405142c615352613e8bacee46e92484eb0c08f26)

librbd: remove unused journal replay canceled callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f453554edcb16ec72714b2456c08bab9e339b1eb)

librbd: optionally support disabling journal appends via policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f7eeacd02f8f80c5d1d5d9cfed35bbd23f64b018)

librbd: optimize header file dependency & modify related file.

Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>
(cherry picked from commit 2185a6275c89d9462611e965da407ea0d504987d)