Merge pull request #29267 from smithfarm/wip-40128-mimic

mimic: rgw: Swift interface: server side copy fails if object name contains '?'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29268 from smithfarm/wip-40136-mimic

mimic: rgw: the Multi-Object Delete operation of S3 API wrongly handles the Code response element

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29270 from smithfarm/wip-40141-mimic

mimic: rgw: multisite: 'radosgw-admin bucket sync status' should call syncs_from(source.name) instead of id

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29273 from smithfarm/wip-40144-mimic

mimic: rgw: Multisite sync corruption for large multipart obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29277 from smithfarm/wip-40225-mimic

mimic: rgw_file: include tenant when hashing bucket names

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #29279 from smithfarm/wip-40351-mimic

mimic: rgw/OutputDataSocket: append_output(buffer::list&) says it will (but does not) discard output at data_max_backlog

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29284 from smithfarm/wip-40354-mimic

mimic: rgw: multisite: RGWListBucketIndexesCR for data full sync needs pagination

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29285 from smithfarm/wip-40357-mimic

mimic: rgw: provide admin-friendly reshard status output

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29288 from smithfarm/wip-40360-mimic

mimic: rgw: set null version object issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29463 from jan--f/wip-41056-mimic

mimic: ceph-volume does not recognize wal/db partitions created by ceph-disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: fall back to PARTTYPE if PARTLABEL is empty

In some cases ceph-disk does not populate PARTLABEL for wal and db
partitions. This commit adds the assumption that the empty string is a
valid label and falls back to identifying those disks by PARTTYPE.

Fixes: https://tracker.ceph.com/issues/40917
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 82d2ae7ffdf62ca93c7848fb62e506db4cda893c)

ceph-volume: adjust tests for empty PARTLABEL fields

It seems like ceph-disk sometimes creates OSDs where the wal and db
partitions have no PARTLABEL set. This commit alters tests so that the
empty string is considered a valid label.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bb29110c3c6e081568231131c70522dee6d12bb5)

ceph-volume: refactor ceph-disk_member unittests

This adds a parameterized fixture as input to the tests. Should make
writing new tests easier since only fixture parameters need to be added.
In the future we should be able to reuse the fixtures to build other
fixtures and reuse the parametrization.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f42287c3416ad09c112e64756f5051c5c13aa344)

Merge pull request #29029 from liewegas/wip-mgr-crash-py3-mimic

mimic: mgr/crash: fix python3 invalid syntax problems

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29150 from pdvian/wip-40269-mimic

mimic: build/ops: make "patch" build dependency explicit

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29197 from smithfarm/wip-38963-mimic

mimic: mgr: DaemonServer::handle_conf_change - broken locking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29201 from smithfarm/wip-40268-mimic

mimic: common: data race in OutputDataSocket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29217 from smithfarm/wip-39692-mimic

mimic: bluestore: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29243 from jan--f/mimic-c-v-extend-batch

mimic: c-v extend batch

ceph-volume: create two NVME drives for testing

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: bz#1650306
(cherry picked from commit 893b61b256bcff142dbd90b777b2c9df9e032e9d)

ceph-volume: set block_db_size to 9GB in centos7 mixed type test

This is to enable testing of the --block-db-size flag

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit eab962c6ef68c588674bdac6fc4a86c84714f300)

ceph-volume: use two NVME drives in centos7 mixed_type tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 934c32acb318aac9aff3c800cdbf99bbdee04d06)

Merge pull request #29350 from jan--f/wip-40977-mimic

mimic: ceph-volume: missing string substitution when reporting mounts

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume:util: Use proper param substition

Fixes: https://tracker.ceph.com/issues/25030
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
(cherry picked from commit 654660cd89fe9d93087d390805651c860bf4e862)

Merge pull request #26991 from ceph/backport-mimic-26957

mimic: ceph-volume: look for rotational data in lsblk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: look for rotational data in lsblk

Also ask lsblk if a device is rotational if no information
is found in /sys/block, default to True if nothing is found.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit de748f9aef894b247622ee5dc49c1fbc31097ad0)

Merge pull request #28918 from smithfarm/wip-40222-mimic

mimic: mds: reset heartbeat during long-running loops in recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #29187 from xiaoxichen/wip-40875-mimic

mimic: cephfs: client: bump ll_ref from int32 to uint64_t

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29200 from smithfarm/wip-39685-mimic

mimic: cephfs: ceph-fuse: client hang because its bad session PipeConnection to mds

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #29208 from smithfarm/wip-40317-mimic

mimic: tests: test_volume_client: print python version correctly

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29210 from smithfarm/wip-38099-mimic

mimic: mds: remove cache drop admin socket command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29211 from smithfarm/wip-38687-mimic

mimic: kcephfs TestClientLimits.test_client_pin fails with client caps fell below min

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #27028 from mrandybu/mimic-alt

mimic: ceph_detect_init: Add support for ALT Linux

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28210 from votdev/fix_rgw_issue

mimic: mgr/dashboard: Fix various RGW issues

Reviewed-by:  Tiago Melo <tmelo@suse.com>

Merge pull request #28911 from bk201/wip-40660-mimic

mimic: mgr/dashboard: cephfs multimds graphs stack together

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29024 from ifed01/wip-ifed-fix-len0-mimic

mimic: os/bluestore: avoid length overflow in extents returned by Stupid Allocator

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #29174 from neha-ojha/wip-40583-mimic

mimic: common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29189 from badone/wip-mimic-lazy-omap-stats-backport-tracker-38552

mimic: core: lazy omap stat collection

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29196 from smithfarm/wip-37749-mimic

mimic: tools: Add clear-data-digest command to objectstore tool

Reviewed-by: David Zafman <dzafman@redhat.com>

test/rgw: test truncated reads in RGWHTTPManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c669cf293f02b3c68a4c539d7accf7ddf35cb1ee)

Conflicts:
    src/test/rgw/test_http_manager.cc
- RGWHTTP::process() only takes one argument in mimic

rgw: set null version object acl issues

1.set null version object acl will create empty index
RGWRados::set_attrs did not clear instance, so index prepare, complete got instance=null,
which lead to empty index 1000_<obj>_i_null.
there is no harm to create empty index, but listomapkeys to find that key.

2.if object is exist with versioned key, we can set none exists null version object
order:
1) enable bucket version
2) put obj
3) disable bucket version
4) set versoned_id=null acl will succeed which should not

Fixes: http://tracker.ceph.com/issues/36763
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit ca6e5c89e8d6da498da15f0eea4bcd4d271b44e9)

rgw: provide admin friendly reshard status output

Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit ba4167c2f30fbb868482d48aaada6388f58de920)

rgw: update the "radosgw-admin reshard status"
command documentation with expected output examples

Fixes: http://tracker.ceph.com/issues/37615
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit 8565ec0db69223ae71766ba000eecc0d2f7d14c2)

rgw/multisite:RGWListBucketIndexesCR for data full sync pagination

RGWListBucketIndexesCR uses the 'marker' from remote zone's RGWOp_Metadata_List
admin api to list all keys for data sync.

Fixes http://tracker.ceph.com/issues/39551

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 708ba6dcc90525c8c4316760e01ee9b1e6da21b7)

Conflicts:
src/rgw/rgw_data_sync.cc
- trivial

Merge pull request #29016 from ivancich/mimic-abort-multipart-fix

mimic: rgw: abort multipart fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29021 from smithfarm/wip-39074-mimic

mimic: rgw: multisite: data sync loops back to the start of the datalog after reaching the end

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29139 from ivancich/mimic-fix-resharding-race

mimic: rgw: fix race b/w bucket reshard and ops waiting on reshard completion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29017 from smithfarm/wip-38080-mimic

mimic: multisite: overwrites in versioning-suspended buckets fail to sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/OutputDataSocket: actually discard data on full buffer

A dout message in OutputDataSocket::append_output() states that
data will be dropped when appending would cause data_max_backlog
to be exceeded--but the method appends it anyway.

Log output discards at level 0, as messages will be lost.  Suppress
repeated warnings mod 100.  Switch to vector.

Fixes: http://tracker.ceph.com/issues/40178
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit c806b825dae649829de8847d36cb21ffd2bbee8e)

Conflicts:
src/common/OutputDataSocket.cc
src/common/OutputDataSocket.h

rgw_file: pretty-print fh_key

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 84a2c22108a3d15b836680894ba750b637e49cd2)

rgw_file: include tenant in hashes of object

Because bucket names are taken as object names in the top
of an export.  Make hashing by tenant general to avoid disjoint
hashing of bucket.

Fixes: http://tracker.ceph.com/issues/40118
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 8e0fd5fbfa7c770f6b668e79b772179946027bce)

rgw_file: include tenant when hashing bucket names

Prevent identical paths from distinct tenants from colliding in
RGW NFS handle cache.

Fixes: http://tracker.ceph.com/issues/40118
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit b800a9de83dff23a150ed7d236cb61c8b7d971ae)

Conflicts:
src/rgw/rgw_file.cc
- ceph_assert

rgw: fix miss handle curl error return

if meet tcp packet loss, curl return will be result=18(CURLE_PARTIAL_FILE)
and http_status=200, so sync will continue and cause content miss match.

Fixes: https://tracker.ceph.com/issues/39992
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit 9b7025857026c64cf076e82d762556ac2bcd2625)

ceph-volume: add explicit mixed-type functional tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bea35c63c83fc2ab87fc983774ed8181cbe74e5b)

rgw: fetch_remote_obj() compares expected object size

if the size of object data received doesn't match the value returned in
the Rgwx-Object-Size header, fail with -EIO

Fixes: https://tracker.ceph.com/issues/39992
Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c6c4c5639ea34c49329719c2725886e740acd0b3)

radosgw-admin: fix syncs_from in 'bucket sync status'

If a zone is set up with sync_from_all=false, the bucket sync status
command will incorrectly report "not in sync_from" because it's calling
'zone.syncs_from(source.id)', where syncs_from() expects a zone name
instead of id

Fixes: http://tracker.ceph.com/issues/40022
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9da888686c3a8880976cd604bdafc4361068c985)

Conflicts:
src/rgw/rgw_zone.h
- in mimic, this code is in src/rgw/rgw_rados.h - made the change there instead

rgw: dump s3_code as the Code response element in RGWDeleteMultiObj_ObjStore_S3.

Fixes: http://tracker.ceph.com/issues/18241
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit 8406083fb14042cb461a3aaa84b82fd78df5168f)

ceph-volume: add tests for Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 01185fb32479f271beeb14dcf94e32b1401ad2d9)

ceph-volume: adds a lvm_size property to util.Device

This will give us the size lvm should report because it takes into
account the 1GB physical extent size we set.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5197c16b44fcf344862201549777dc2935fc19a3)

rgw: parse_copy_location defers url-decode

don't url-decode until after we search for the ?, or we'll truncate
object names that contain a url-encoded ?

Fixes: http://tracker.ceph.com/issues/27217
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 804bb853f1c7b5ce546c100d61c6fc526838c134)

ceph-volume: functional tests, add libvirt customization

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c330a4758755e3815b30e4f00a4a00e8319db399)

os/bluestore: be verbose about objects that existing on rmcoll

This is always a bug (OSD doesn't try to remove a collection unless it
thinks it is empty), and not seeing it at default debug levels makes it
hard to track down.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e387cbc45037db78db5dfbb01483b8d76e1c276f)

Conflicts:
src/os/bluestore/BlueStore.cc
- mimic does not have 87e8231d3fa61bc95ea0cda4d731e30f3d4de9d5

osd/PrimaryLogPG: disallow ops on objects with an empty name

These may conflict with pgmeta objects.  And are going to cause other
problems later (e.g., https://tracker.ceph.com/issues/38724).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf6e8a4a4db8b7d7101ca80b844ee953aa4580b2)

osd/PG: fix cleanup of pgmeta-like objects on PG deletion

If an object has an empty 'name' field, it "looks" like a pgmeta object,
and the PG cleanup code was skipping it.  However, we were letting these
objects get created.

Fix by only skipping *our* pgmeta object.  If there are other pgmeta-like
objects in the PG collection, clean them up.

Fixes: https://tracker.ceph.com/issues/38724
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a0f48db5cbfc48761eeac0daa259fecbaa573969)

# Conflicts:
# src/osd/PG.cc
- location of this code is do_delete_work in master, _delete_some in n

common: OutputDataSocket retakes mutex on error path

the list of buffers is protected by mutex m_lock. when dump_data() fails
and puts unwritten buffers back into the list, it needs to reaquire the
lock

Fixes: http://tracker.ceph.com/issues/40188
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8645e24dd4fb3ecc9f02b34a1be916620b5f71b1)

Conflicts:
    src/common/OutputDataSocket.cc
- use Mutex::Locker instead of std::scoped_lock

mgr/DaemonServer: handle_conf_change - fix broken locking

The is_locked_by_me() is only meant to be safe if you know
you hold the lock. It's meant to be used in an assertion.

Fixes: http://tracker.ceph.com/issues/38899
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a929a5a01508e661fc7f3cea52822320d39983d2)

Conflicts:
    src/mgr/DaemonServer.cc
- use Mutex::Locker instead of std::lock_guard

tools: Add clear-data-digest command to objectstore tool.

There may be a situation where data digest in object info is
inconsistent with that computed from object data, then deep-scrub
will fail even though all three repicas have the same object data.

Fixes: https://tracker.ceph.com/issues/37935
Signed-off-by: Li Yichao <liyichao.good@gmail.com>
(cherry picked from commit da5832b2b49be4ab23f7ffc5ba1ba630168b886f)

Conflicts:
    src/tools/ceph_objectstore_tool.cc
- change cbegin() to begin() to get it to build

qa/cephfs: relax min_caps_per_client check

new kernel client proactively release caps. caps count can go below
mds_min_caps_per_client

Fixes: http://tracker.ceph.com/issues/38270
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 8e81bd74c59259a06f587bb425d58bf2b2176341)

Conflicts:
qa/tasks/cephfs/test_client_limits.py
- mimic TestClientLimits does not have expected_caps method at all

mds: remove cache drop asok command

`cache drop` is a long running command that will block the asok interface
(while the tell version does not). Attempting to abort the command with ^C or
equivalents will simply cause the `ceph` command to exit but won't stop the
asok command handler from waiting for the cache drop operation to complete.

Instead, just allow the tell version.

Fixes: http://tracker.ceph.com/issues/38020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7fa1e3c37f8c7fb709ae9e070a3154a7084c0584)

Conflicts:
PendingReleaseNotes
qa/tasks/cephfs/test_misc.py
src/mds/MDSDaemon.cc

qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Fixes: http://tracker.ceph.com/issues/40184
Signed-off-by: Lianne <liyan.wang@xtaotech.com>
(cherry picked from commit 7c7c7870d38902a0df83a0fdecaa56baad556d82)

client: fix fuse client hang because its bad session PipeConnection
to mds.

There still has a risk client will hang all the time and which
happened in my enviroment a few days ago. Well konw, the start
of mds will go through four stages at least(replay, reconnect,
rejoin, active). According to current logic, we recived
replay mdsmap and marked down pipe will return directly and if
we miss the next reconnect mdsmap the client may hang all the
time becuase we lost the opportunity due to old_inc equal to
new_inc. So we shuold judge if we have lost MDSMap::reconnect
by comparing newstate and oldstate.

Fixes: http://tracker.ceph.com/issues/39305
Signed-off-by: Guan yunfei <yunfei.guan@xtaotech.com>
(cherry picked from commit 51c9f5cf9af61fe317259ac79a503ef8a7139622)

Conflicts:
src/client/Client.cc
- mimic has "kick_maxsize_requests(session); wake_inode_waiters(session);"
  where master has just "wake_up_session_caps(session, true)

mimic: osd: Modify lazy omap stats to only show one copy

Only show stats from the primary to get more in line with other pg stats
representations.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 490829c8eaf14f41de978ac0f93ecda3df8bcdef)

mimic: tests: Add test for lazy omap stat collection

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 88e9ca58a06ba572ff4c6f6125a8ba7f8e0a7795)

mimic: osd: Implement lazy omap usage statistics

Opportunistic gathering of omap statistics during deep scrub.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 9e21ef0e40ff89b10dfdf857c703336496b3171f)

 Conflicts:
doc/dev/placement-group.rst: Trivial
src/osd/osd_types.h: Trivial

Client: bump ll_ref from int32 to uint64_t

ll_ref can surge up if application repeatly lookup same dentry
and finally cause overflow.

https://tracker.ceph.com/issues/40775 is an example.

in kernel fuse driver and libfuse, uint64_t is used, update
ceph-fuse side to match.

Fixes: https://tracker.ceph.com/issues/40775
Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>
(cherry picked from commit bb46de885336a8cd2adc9c0507e543c6832346e7)

 Conflicts:
src/client/Client.h
        use new code, skip all non-toched changes.
src/client/Inode.h
        use new code.

common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Warning when there are already 2M omap keys does not help, reduce this
threshold so that preemptive measures can be taken based on this.

Fixes: https://tracker.ceph.com/issues/40583
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 1ec948409ab547effbd5c656af8f0c9565f1717a)

rgw: add RGWRados::repair_olh() to reconstruct olh attrs

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1309e2d50ee8118afe1138cd6677d84438edb55a)

Conflicts:
    src/rgw/rgw_rados.cc
- no rgw_bl_str() in mimic
- mimic does not have a4d3977cfacc5eb0de16618f2c957bb417505597

rgw: add RGWRados::bi_get_olh() to read/decode olh entry

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b8d7f0ad1f882b999e0fd7f3e45f8579599714cf)

Conflicts:
    src/rgw/rgw_rados.cc
- bufferlist::cbegin()

rgw: bi_get() takes bucket_info

and calls the bucket_info overload of BucketShard::init() to avoid
reading the bucket info again

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a15e8736340c916120fef1f3a5ff1b982cf7c837)

rgw: add BucketShard::init overload for bucket_info,obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c6c445f6074df113a28adcae89248b909d031f8e)

test/rgw: add multisite test_version_suspended_incremental_sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 81fa354c1dadf4f957592f20fe9e02acd70671e0)

Conflicts:
src/test/rgw/rgw_multi/tests.py

rgw: fetch_remote_obj filters out olh attrs

each zone manages its own olh log, so we don't want any of the source
zone's olh attributes. in one case (sync of an object overwrite in a
versioning-suspended bucket), these olh attributes were being written
directly to the head object and leading to link_olh errors due to the
olh_tag mismatch

Fixes: http://tracker.ceph.com/issues/37792
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a4195fc55fb76f0a2576d5ff0b2c99ad12fd7a49)

Merge pull request #29020 from smithfarm/wip-39047-mimic

mimic: rgw: update resharding documentation

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #29014 from smithfarm/wip-37891-mimic

mimic: doc: wrong value of usage log default in logging section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

build/ops: make "patch" build dependency explicit

This commit adds "patch" as an explicit build dependency. Before, the RPM
build was installing it in the build environment via some mechanism that I
don't entirely understand.

The "patch" command is used by the BuildBoost cmake module.

Fixes: http://tracker.ceph.com/issues/40175
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit fe632e2d886791e2168ee6200b5bf1bfe8c88d0e)

rgw: fix race b/w bucket reshard and ops waiting on reshard completion

A previous commit (f84f70d4) added functionality to clean up old
bucket instances and bucket shards once sharding completed
successfully. However the existing code that allowed a bucket
operation to wait until resharding completed was relying on a field in
the old bucket instance to know the bucket instance id of the new
bucket instance. This created a race condition as to whether the
clean-up or read of the bucket instance id would occur first.

This solution rereads the bucket entry point object when resharding
completes to determine the bucket instance id of the new bucket
thereby avoiding the race.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 26b793680805721cac86f64cdd234b273bf33be4)

Conflicts:
        src/rgw/rgw_rados.cc: equivalent changes made to rgw_reshard.cc

rgw: correct comment describing function try_refresh_bucket_info

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit d0894e6f6c062907796d10b524088c2abf7e4252)

rgw: fix log messages using "proceeding" to "processing"

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 1c2aa902cb2bec690387ace8eb6b8afb5678373f)

mgr/dashboard: Fix various RGW issues

* Fix handling of tenanted users
* Better exception handling if RGW backend is not available
* RGW user/bucket "max. size" should be hidden when "user/bucket quota" is not enabled
* Fix bug in loading user quota.

Fixes: https://tracker.ceph.com/issues/38800
partial manual backport of 61995970916cb11f66c3800bcf869755f8dae32c
partial manual backport of e6f130d47023c24f8a9c742f145abd34d7320cd8
partial manual backport of 945e790cab57785e1bc2dbf708d5ddafec618aba
partial manual backport of a98bca6a2f161140f78f34cfa5f5ebec07af2f82
partial manual backport of 975736aba546adb09ce4f38a1ae061621f7240cd

This fix is not cherry-picked because the code was completely re-worked for Nautilus (codebase is too far apart).

Signed-off-by: Volker Theile <vtheile@suse.com>

ceph-volume: add space between words

The current help of batch was showing:

rotational drivesare passed in DEVICES

This commit fixes it and now we have:

rotational drives are passed in DEVICES

Tremendous fix, I know.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit a151e111a2710e864bd66a44db38685459cf55c8)

ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 8ebff475dfa136c88101ef7364820fe3c5cee547)
(cherry picked from commit aa6da416cb14ba521796c950fba9667e7b581b27)

ceph-volume: set a lvm_size property on the fakedevice fixture

This is so tests can continue to set sys_api['size'] and the code
can retrieve that as Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9e2175c4001434f7b0ed8ce6f723e063cf001440)

ceph-volume: don't crash on unusable data devices

Also add test case for a MixedStrategy with unusable data devices.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_filestore.py
        Pick test addition and error-value change

ceph-volume: add a wal device specific unit test and size validator

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_bluestore.py
        Pick test addition and error.value change

ceph-volume: add osd_ids argument

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: strategies allow for external wal lv's

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: add default argument values to strategy

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 5465712e4d71315ff8daac9345be6ba96d1f44c4)

 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_bluestore.py
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_filestore.py
        pick with_auto_devices and error.value changes

Add a unit test that tests the ceph_volume.devices.lvm.batch.Batch::get_devices method.

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 19a38189fd15f57907449631a0f6e9faf908dc66)

ceph-volume strategies: rename db_devs to db_or_journal_devs

This aims at reducing confusion when reading the code that is shared
between the bluestore and filestore strategies.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 029414e89b773896fddebe0a5869b57ec13f527e)

ceph-volume batch: rename block_devs to data_devs for consistency

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 6dbf43e53ccd3ff06e3cdfc8e6de97475cc0c510)