ceph-volume: look for rotational data in lsblk

Also ask lsblk if a device is rotational if no information
is found in /sys/block, default to True if nothing is found.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit de748f9aef894b247622ee5dc49c1fbc31097ad0)

Merge pull request #28918 from smithfarm/wip-40222-mimic

mimic: mds: reset heartbeat during long-running loops in recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #29187 from xiaoxichen/wip-40875-mimic

mimic: cephfs: client: bump ll_ref from int32 to uint64_t

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29200 from smithfarm/wip-39685-mimic

mimic: cephfs: ceph-fuse: client hang because its bad session PipeConnection to mds

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #29208 from smithfarm/wip-40317-mimic

mimic: tests: test_volume_client: print python version correctly

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29210 from smithfarm/wip-38099-mimic

mimic: mds: remove cache drop admin socket command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29211 from smithfarm/wip-38687-mimic

mimic: kcephfs TestClientLimits.test_client_pin fails with client caps fell below min

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #27028 from mrandybu/mimic-alt

mimic: ceph_detect_init: Add support for ALT Linux

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28210 from votdev/fix_rgw_issue

mimic: mgr/dashboard: Fix various RGW issues

Reviewed-by:  Tiago Melo <tmelo@suse.com>

Merge pull request #28911 from bk201/wip-40660-mimic

mimic: mgr/dashboard: cephfs multimds graphs stack together

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29024 from ifed01/wip-ifed-fix-len0-mimic

mimic: os/bluestore: avoid length overflow in extents returned by Stupid Allocator

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #29174 from neha-ojha/wip-40583-mimic

mimic: common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29189 from badone/wip-mimic-lazy-omap-stats-backport-tracker-38552

mimic: core: lazy omap stat collection

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29196 from smithfarm/wip-37749-mimic

mimic: tools: Add clear-data-digest command to objectstore tool

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29016 from ivancich/mimic-abort-multipart-fix

mimic: rgw: abort multipart fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29021 from smithfarm/wip-39074-mimic

mimic: rgw: multisite: data sync loops back to the start of the datalog after reaching the end

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29139 from ivancich/mimic-fix-resharding-race

mimic: rgw: fix race b/w bucket reshard and ops waiting on reshard completion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29017 from smithfarm/wip-38080-mimic

mimic: multisite: overwrites in versioning-suspended buckets fail to sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tools: Add clear-data-digest command to objectstore tool.

There may be a situation where data digest in object info is
inconsistent with that computed from object data, then deep-scrub
will fail even though all three repicas have the same object data.

Fixes: https://tracker.ceph.com/issues/37935
Signed-off-by: Li Yichao <liyichao.good@gmail.com>
(cherry picked from commit da5832b2b49be4ab23f7ffc5ba1ba630168b886f)

Conflicts:
    src/tools/ceph_objectstore_tool.cc
- change cbegin() to begin() to get it to build

qa/cephfs: relax min_caps_per_client check

new kernel client proactively release caps. caps count can go below
mds_min_caps_per_client

Fixes: http://tracker.ceph.com/issues/38270
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 8e81bd74c59259a06f587bb425d58bf2b2176341)

Conflicts:
qa/tasks/cephfs/test_client_limits.py
- mimic TestClientLimits does not have expected_caps method at all

mds: remove cache drop asok command

`cache drop` is a long running command that will block the asok interface
(while the tell version does not). Attempting to abort the command with ^C or
equivalents will simply cause the `ceph` command to exit but won't stop the
asok command handler from waiting for the cache drop operation to complete.

Instead, just allow the tell version.

Fixes: http://tracker.ceph.com/issues/38020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7fa1e3c37f8c7fb709ae9e070a3154a7084c0584)

Conflicts:
PendingReleaseNotes
qa/tasks/cephfs/test_misc.py
src/mds/MDSDaemon.cc

qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Fixes: http://tracker.ceph.com/issues/40184
Signed-off-by: Lianne <liyan.wang@xtaotech.com>
(cherry picked from commit 7c7c7870d38902a0df83a0fdecaa56baad556d82)

client: fix fuse client hang because its bad session PipeConnection
to mds.

There still has a risk client will hang all the time and which
happened in my enviroment a few days ago. Well konw, the start
of mds will go through four stages at least(replay, reconnect,
rejoin, active). According to current logic, we recived
replay mdsmap and marked down pipe will return directly and if
we miss the next reconnect mdsmap the client may hang all the
time becuase we lost the opportunity due to old_inc equal to
new_inc. So we shuold judge if we have lost MDSMap::reconnect
by comparing newstate and oldstate.

Fixes: http://tracker.ceph.com/issues/39305
Signed-off-by: Guan yunfei <yunfei.guan@xtaotech.com>
(cherry picked from commit 51c9f5cf9af61fe317259ac79a503ef8a7139622)

Conflicts:
src/client/Client.cc
- mimic has "kick_maxsize_requests(session); wake_inode_waiters(session);"
  where master has just "wake_up_session_caps(session, true)

mimic: osd: Modify lazy omap stats to only show one copy

Only show stats from the primary to get more in line with other pg stats
representations.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 490829c8eaf14f41de978ac0f93ecda3df8bcdef)

mimic: tests: Add test for lazy omap stat collection

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 88e9ca58a06ba572ff4c6f6125a8ba7f8e0a7795)

mimic: osd: Implement lazy omap usage statistics

Opportunistic gathering of omap statistics during deep scrub.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 9e21ef0e40ff89b10dfdf857c703336496b3171f)

 Conflicts:
doc/dev/placement-group.rst: Trivial
src/osd/osd_types.h: Trivial

Client: bump ll_ref from int32 to uint64_t

ll_ref can surge up if application repeatly lookup same dentry
and finally cause overflow.

https://tracker.ceph.com/issues/40775 is an example.

in kernel fuse driver and libfuse, uint64_t is used, update
ceph-fuse side to match.

Fixes: https://tracker.ceph.com/issues/40775
Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>
(cherry picked from commit bb46de885336a8cd2adc9c0507e543c6832346e7)

 Conflicts:
src/client/Client.h
        use new code, skip all non-toched changes.
src/client/Inode.h
        use new code.

common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Warning when there are already 2M omap keys does not help, reduce this
threshold so that preemptive measures can be taken based on this.

Fixes: https://tracker.ceph.com/issues/40583
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 1ec948409ab547effbd5c656af8f0c9565f1717a)

rgw: add RGWRados::repair_olh() to reconstruct olh attrs

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1309e2d50ee8118afe1138cd6677d84438edb55a)

Conflicts:
    src/rgw/rgw_rados.cc
- no rgw_bl_str() in mimic
- mimic does not have a4d3977cfacc5eb0de16618f2c957bb417505597

rgw: add RGWRados::bi_get_olh() to read/decode olh entry

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b8d7f0ad1f882b999e0fd7f3e45f8579599714cf)

Conflicts:
    src/rgw/rgw_rados.cc
- bufferlist::cbegin()

rgw: bi_get() takes bucket_info

and calls the bucket_info overload of BucketShard::init() to avoid
reading the bucket info again

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a15e8736340c916120fef1f3a5ff1b982cf7c837)

rgw: add BucketShard::init overload for bucket_info,obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c6c445f6074df113a28adcae89248b909d031f8e)

test/rgw: add multisite test_version_suspended_incremental_sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 81fa354c1dadf4f957592f20fe9e02acd70671e0)

Conflicts:
src/test/rgw/rgw_multi/tests.py

rgw: fetch_remote_obj filters out olh attrs

each zone manages its own olh log, so we don't want any of the source
zone's olh attributes. in one case (sync of an object overwrite in a
versioning-suspended bucket), these olh attributes were being written
directly to the head object and leading to link_olh errors due to the
olh_tag mismatch

Fixes: http://tracker.ceph.com/issues/37792
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a4195fc55fb76f0a2576d5ff0b2c99ad12fd7a49)

Merge pull request #29020 from smithfarm/wip-39047-mimic

mimic: rgw: update resharding documentation

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #29014 from smithfarm/wip-37891-mimic

mimic: doc: wrong value of usage log default in logging section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix race b/w bucket reshard and ops waiting on reshard completion

A previous commit (f84f70d4) added functionality to clean up old
bucket instances and bucket shards once sharding completed
successfully. However the existing code that allowed a bucket
operation to wait until resharding completed was relying on a field in
the old bucket instance to know the bucket instance id of the new
bucket instance. This created a race condition as to whether the
clean-up or read of the bucket instance id would occur first.

This solution rereads the bucket entry point object when resharding
completes to determine the bucket instance id of the new bucket
thereby avoiding the race.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 26b793680805721cac86f64cdd234b273bf33be4)

Conflicts:
        src/rgw/rgw_rados.cc: equivalent changes made to rgw_reshard.cc

rgw: correct comment describing function try_refresh_bucket_info

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit d0894e6f6c062907796d10b524088c2abf7e4252)

rgw: fix log messages using "proceeding" to "processing"

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 1c2aa902cb2bec690387ace8eb6b8afb5678373f)

mgr/dashboard: Fix various RGW issues

* Fix handling of tenanted users
* Better exception handling if RGW backend is not available
* RGW user/bucket "max. size" should be hidden when "user/bucket quota" is not enabled
* Fix bug in loading user quota.

Fixes: https://tracker.ceph.com/issues/38800
partial manual backport of 61995970916cb11f66c3800bcf869755f8dae32c
partial manual backport of e6f130d47023c24f8a9c742f145abd34d7320cd8
partial manual backport of 945e790cab57785e1bc2dbf708d5ddafec618aba
partial manual backport of a98bca6a2f161140f78f34cfa5f5ebec07af2f82
partial manual backport of 975736aba546adb09ce4f38a1ae061621f7240cd

This fix is not cherry-picked because the code was completely re-worked for Nautilus (codebase is too far apart).

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28206 from pdvian/wip-39475-mimic

mimic: common/util: handle long lines in /proc/cpuinfo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28217 from smithfarm/wip-40005-mimic

mimic: build/ops: do_cmake.sh: source not found

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #28741 from smithfarm/wip-40547-mimic

mimic: common: Keyrings created by ceph auth get are not suitable for ceph auth import

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28894 from ifed01/wip-ifed-fix-no-compress-mimic

mimic: os/bluestore: load OSD all compression settings unconditionally.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28967 from liewegas/wip-set-aio-write-max-mimic

mimic: os/bluestore: fix >2GB bluefs writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28970 from neha-ojha/wip-40720-mimic

mimic: os/bluestore: default to bitmap allocator for bluestore/bluefs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28995 from smithfarm/wip-38751-mimic

mimic: core: should report EINVAL in ErasureCode::parse() if m<=0

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28220 from pdvian/wip-39513-mimic

mimic: osd: take heartbeat_lock when calling heartbeat()

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #26499 from k0ste/mimic_backports2

mimic: mgr/balancer: restrict automatic balancing to specific weekdays

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27986 from pdvian/wip-39311-mimic

mimic: crushtool crash on Fedora 28 and newer

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28089 from dzafman/wip-39720

mimic: osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28097 from pdvian/wip-39374-mimic

mimic: osd/OSD.cc: make osd bench description consistent with parameters

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28142 from pdvian/wip-39422-mimic

mimic: osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #28165 from pdvian/wip-39458-mimic

mimic: mgr/prometheus: replace whitespaces in metrics' names

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #28201 from dzafman/wip-38341

mimic: osd: pg stuck in backfill_wait with plenty of disk space

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29074 from ceph/backport-mimic-26397

mimic: ceph-volume: update testing playbook 'deploy.yml'

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27949 from smithfarm/wip-37613-mimic

mimic: rpm: missing dependency on python34-ceph-argparse from python34-cephfs (and others?)

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28090 from pdvian/wip-39233-mimic

mimic: cephfs: mount.ceph.c: do not pass nofail to the kernel

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28186 from pdvian/wip-39472-mimic

mimic: mds: fix SnapRealm::resolve_snapname for long name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #28274 from pdvian/wip-39669-mimic

mimic: mds: output lock state in format dump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28337 from pdvian/wip-39679-mimic

mimic: cephfs: pybind: added lseek()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28381 from pdvian/wip-39689-mimic

mimic: mds: check dir fragment to split dir if mkdir makes it oversized.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28501 from smithfarm/wip-40168-mimic

mimic: cephfs: client: ceph.dir.rctime xattr value incorrectly prefixes 09 to the nanoseconds component

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #28579 from ukernel/mimic-40342

mimic: mds: fix corner case of replaying open sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28650 from pdvian/wip-40042-mimic

mimic: mds: avoid sending too many osd requests at once after mds restarts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #28796 from shenhang/mimic

mimic: mds: destroy reconnect msg when it is from non-existent session to avoid memory leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

ceph-volume: fix deploy playbook for ansible~=2.6,<2.7

When testing the lumionus and mimic branch the import_role
ansible feature does not work as expected. Variables created by one
role is not available to the next role called when using
import_role. To fix this, we need to use the roles keyword instead.

This commit is only applies to luminous and mimic and should
only be applied there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b619988fbfafa7568e0ec51cfb56336a72460d44)

Merge pull request #28887 from pdvian/wip-40262-mimic

mimic: rgw: rgw_file: all directories are virtual with respect to contents

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #29018 from smithfarm/wip-38776-mimic

mimic: rgw: support delimiter longer then one symbol

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28073 from pdvian/wip-39229-mimic

mimic: rgw_file: save etag and acl info in setattr

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28757 from smithfarm/wip-38747-mimic

mimic: rgw: non-existent mdlog failures logged at level 0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28310 from pdvian/wip-39674-mimic

mimic: qa/workunits/rbd: wait for rbd-nbd unmap to complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28815 from trociny/wip-40593-mimic

mimic: rbd-mirror: handle duplicates in image sync throttler queue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: fix MDS charts are stacked in Filesystems page

Also include some UI enhancements:
- Fix MDS chart container is not aligned to other components.
- Show header on client table.

Fixes: https://tracker.ceph.com/issues/37579
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit a3e76ac5f5cf56f830fd2ae9d2e2ac4585195fc2)

Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-clients/cephfs-clients.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.html

files above are renamed, their paths in mimic are:
src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/clients/clients.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs/cephfs.component.html

Merge pull request #28852 from jdurgin/wip-18096-mimic

mimic: osd: report omap/data/metadata usage

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27971 from smithfarm/wip-39593-mimic

mimic: build/ops: Cython 0.29 removed support for subinterpreters: raises ImportError: Interpreter change detected ...

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28208 from votdev/fix_vstart_runner

mimic: qa: Fix issues in vstart runner

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28954 from bk201/wip-40707

mimic: mgr/dashboard: Fix run-frontend-e2e-tests.sh

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-volume: update deploy.yml

update deploy.yml accordingly with recent changes in ceph-ansible.

Fixes: ceph/ceph-ansible#3602
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit da13da67bc1e8ba062c61a716eea485ae462db48)

rgw: minor code clean-up

Adding some const correctness and reformatting.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit fb862909202c12c4bd908a0efe140c0f0ef0bd5a)

rgw: allow multipart upload abort to proceed

Cleaning up after aborting unfinished multipart uploads is a
best-effort process. Due to the history of the multipart upload, not
all components may be found and the process needs to be robust to
that.

Presently, if a component is not found after the final call to
list_bucket_multiparts, the error propogates to the caller, but not if
the component is not found in an earlier call. This change does not
propogate that error up ever.

Additional logging is also added to multipart-related functions.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 51aff62ac7cbb4653e0de667dccfa46795ffd8cb)

os/bluestore: avoid length overflow in extents returned by Stupid
Allocator.

Fixes: http://tracker.ceph.com/issues/40703
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit e1b4984a63e79ab2e2aec7aa17bd80a77e79ec8c)

 Conflicts:
src/test/objectstore/Allocator_test.cc
 inconsistency with the old gtest version

rgw: always generate after delimiter char to skip directory

the following code also need after_delim_s to skip directory after list
1000 objects

Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit 8c6678387b00c862d71cffd6fed6ca6c236e3fb7)

rgw: fix list bucket with start maker and delimiter '/' will miss next object with char '0'

'/' + 1 = '0', and the start marker will be skipped, so it will miss the
'0' object.

Fixes: http://tracker.ceph.com/issues/39989
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit 43b5264c7d61ae3be56d68f131b544edee307a46)

Conflicts:
src/rgw/rgw_rados.cc
- mimic does not have "unsigned" in the line "unsigned char e = delim.back();"

rgw: data sync checks empty next_marker for datalog

RGWReadRemoteDataLogShardCR tracks the marker and next_marker
separately, because next_marker will be empty when it reaches the end

this allows RGWDataSyncShardCR to avoid clearing its sync_marker and
restarting datalog listing from the beginning

Fixes: http://tracker.ceph.com/issues/39033
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ff63778342282ab5c676529f6e7d90f30e6da1a)

rgw: fix up indentation in incremental data sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 666bfe3a909ec83a3067398e7896527f99283e0a)

Conflicts:
src/rgw/rgw_data_sync.cc

rgw: updates to resharding documentation

Some minor updates to the resharding documentation with the aim of
clarifying the language and using terms consistently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 54b6b650eeeeac0e754bd5860cd53109ba7865bd)

Conflicts:
doc/radosgw/dynamicresharding.rst
- refrain from backporting changes to sections that aren't present in mimic

rgw: prefix-delimiter listing: support >1 character delimiter

Fix prefix and CommonPrefix extraction logic in
RGWRados::Bucket::List::list_objects_ordered so as to permit
arbitrary-length string delimiters.

Fixes: https://tracker.ceph.com/issues/24821
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit e3c1ea244234aace7368d5a5ee95af2f6a529b00)

doc: rgw: fix the default value of usage log setting

Fixes: http://tracker.ceph.com/issues/37856
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b21c96ec69ec65f359473a0f193c716925086828)

Merge pull request #28123 from dillaman/wip-39461-mimic

mimic: rbd-mirror: clear out bufferlist prior to listing mirror images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28133 from dillaman/wip-38441-mimic

mimic: librbd: ensure compare-and-write doesn't skip compare after copyup

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28770 from dillaman/wip-40573-mimic

mimic: librbd: properly track in-flight flush requests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28814 from trociny/wip-40461-mimic

mimic: journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/erasure-code/test-erasure-code: adjust test to avoid m=0

_DD is k=2 m=0, which we don't allow.  Switch it to cDD.

I confess I don't fully understand why this was _DD to begin with, but
I'm pretty sure mapping is there to control the order of results so that
it can be mapped to the CRUSH rule output sanely, and the coding portion
is not relevant to the test.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 52d5797c3d3d74dd88664cfe85e5d551e7b334a6)

erasure-code: ensure m >= 1

Fixes: http://tracker.ceph.com/issues/38682
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ab0a6528266ab30d442f351de97f5475399ff278)

Conflicts:
src/erasure-code/clay/ErasureCodeClay.cc
- file does not exist in mimic

mon/OSDMonitor: set ec min_size to k + min(1, m - 1)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7f49be2104f0a5df894a53e200c431c527781f9c)

deb,rpm,do_cmake: switch to cmake3

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit af2c91ace6c1082925f118d145937ae55fa780d4)

cmake: remove cython 0.29's subinterpreter check during install

Commit 3bde34af8a removed cython 0.29's subinterpreter check when
building the various python modules during `make`, but unforunately
they're *rebuilt* during `make install`, with the rebuild overwriting
the original build.  The original fix was of course missing from the
install stage...

Fixes: https://tracker.ceph.com/issues/38788
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 108462e3fb0e814c98ab32f02b79cf3ea1247692)

cmake: remove cython 0.29's subinterpreter check

cython 0.29 introduced a check which prevents multiple python
subinterpreters from loading the same module:

https://github.com/cython/cython/commit/7e27c7c

Unfortunately, this completely breaks ceph-mgr.  Until we can
figure out a better long term solution, this commit removes
cython's subinterpreter check, via some careful abuse of the
C preprocessor.

This works because when cython is invoked, it first generates
some C code, then compiles it.  We know it's going to generate
C code including:

  int __Pyx_check_single_interpreter(void) { ... }

and:

  if (__Pyx_check_single_interpreter())
      return NULL;

So, we can do the following:

  #define void0 dead_function(void)
  #define __Pyx_check_single_interpreter(ARG)=ARG ## 0

This replaces the call to __Pyx_check_single_interpreter()
with a literal 0, removing the subinterpreter check.

The void0 dead_function(void) thing is necessary because
the __Pyx_check_single_interpreter() macro also clobbers
that function definition, so we need to make sure it's
replaced with something that works as a function definition.

Fixes: https://tracker.ceph.com/issues/37472
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 3bde34af8a84d76744887b2cf10ee4ef36e3925b)

Conflicts:
cmake/modules/Distutils.cmake
- mimic doesn't have the immediately preceding ccache stuff

Merge pull request #28161 from pdvian/wip-39426-mimic

mimic: mgr: deadlock

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>