Merge pull request #27082 from pdvian/wip-38720-mimic

mimic: crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27257 from pdvian/wip-38858-mimic

mimic: crush: add root_bucket to identify underfull buckets

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27351 from liewegas/wip-fix-osd-check-mimic

mimic: mon/OSDMonitor: fix osd boot check

Merge pull request #27039 from pdvian/wip-38684-mimic

mimic: librbd: avoid aggregate-initializing IsWriteOpVisitor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27041 from pdvian/wip-38685-mimic

mimic: librbd: avoid aggregate-initializing any static_visitor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26529 from pdvian/wip-38353-mimic

mimic: rgw: check for non-existent bucket in RGWGetACLs

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #26641 from pdvian/wip-38447-mimic

mimic: rgw: only update last_trim marker on ENODATA

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27029 from pdvian/wip-38672-mimic

mimic: rgw: sync module: avoid printing attrs of objects in log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27174 from pdvian/wip-38756-mimic

mimic: rgw: ldap: fix LDAPAuthEngine::init() when uri !empty()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27179 from pdvian/wip-38772-mimic

mimic: rgw: nfs: skip empty (non-POSIX) path segments

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27297 from theanalyst/mimic-sse-c-fixes

Mimic sse c fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/OSDMonitor: relax osd check to allow octopus osd with mimic mon

This mirrors commit 1a689766a50d346bef9f74d45a5086f54f41cca5 in master.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27321 from liewegas/wip-valgrind-mimic

mimic: qa/valgrind: backport of latest valgrind suppressions

qa/valgrind: backport of latest valgrind suppressions

in the correct location

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: fixup_range() clamps offsets to valid part range

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4b3bac20d550cae274c07a1d555751483dde463b)

test_rgw_crypto: add invalid ranges beyond obj boundary

This is just to assert we're not doing an invalid memory access

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b34a00de0f6ecea6d97fd80f65d2a2dca0876e6e)

tests: rgw crypto tests with partlen vector

4 tests are added when either obj_size for an aligned object, and when an object
is not aligned to a boundary (1 B last part), or part_size is not aligned to a
boundary. Comments on boundary case explains the logic on choosing whether to
align to a 4095B boundary or not

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 1bf59f68db608084a1e7a05aca4cca8a8ab76062)

test_rgw_crypt: BlockCryptNone takes a block_size param

So that block sizes like 4096 can be tested

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9e13a42daaba26751f56f2454f9ec314d7d0d460)

rgw_crypt: make decrypt's parts_len protected

This is so that testing classes can set this and use that to test the branches
in fixup_range tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 78b37a5ccd67c464f3c6e61568e9e827c327ad6d)

rgw_crypt: take care of start & end boundaries when parsing range

As http range header is inclusive of start and end ranges, this otherwise breaks
at ranges close to multipart boundaries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9c92b56d77932283ccc0336645dc3b76bd0fffa2)

rgw: decrypt filter does not cross multipart boundaries

multipart uploads with sse encrypts each part separately, using an
initialization vector based on the part offset

decryption must respect the same part boundaries, and start each part
with a fresh initialization vector. this means that the decrypt filter
must flush data up to part boundaries before starting the next

Fixes: http://tracker.ceph.com/issues/38700
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b782902954abd20e49c0ed1c14cebee4c8ab6f88)

Merge pull request #26983 from ifed01/wip-ifed-bmap-alloc-mimic

mimic: os/bluestore: backport new bitmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crush: add root_bucket to identify underfull buckets

All underfull buckets under root_buckets will be taken as target

For the crule rule:
    step take datacenter0
    step chooseleaf firstn 2 type host
    step emit
    step take datacenter1
    step chooseleaf firstn 2 type host
    step emit

If one host contains overfull osd but no underfull osd,
it will use other underfull buckets as target, which
maybe not in the same datacenter, that will
broke the rule.

Fixes: http://tracker.ceph.com/issues/38826
Signed-off-by: huangjun <huangjun@xsky.com>
(cherry picked from commit 3d5678d3561d90a10d9de3cb6e7e0405dbe8fdfe)

Merge pull request #26961 from pdvian/wip-38666-mimic

mimic: qa: ignore slow metadata io wrn during osd thrash

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27053 from rhcs-dashboard/38284-mimic-show-io-stats

mimic: mgr/dashboard: show I/O stats in Pool list

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/suites: fix bluestore links

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 269910fc8b2bdf64b98534ce525318e108aa9409)

rgw: nfs: skip empty (non-POSIX) path segments

Such a path could be created through S3 upload,
e.g., s3://my_files//data/file.pdf.

Previously we asserted for this case, but it's harmless--such a
path segment should just be ignored.

Fixes: http://tracker.ceph.com/issues/38744
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit a53e6c1d5b785a2a46007292041d0a8cb552e3e6)

Conflicts:
src/rgw/rgw_file.h : Resolved in operator

rgw: ldap: fix LDAPAuthEngine::init() when uri !empty()

Fixes: https://tracker.ceph.com/issues/38699
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 6ef98c6e0fcf4f9b6e431b3409975e0966c5c21a)

Merge pull request #27081 from dzafman/wip-38275

mimic: Fix recovery and backfill priority handling

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27032 from cbodley/wip-38797

mimic: doc/rgw: document placement target configuration

Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: mimic fix: show I/O stats in Pool list

* Applied same fix as in nautilus.
* Adapted Pool list columns to avoid constant resizing.
* Added 'decimals' argument to dimless pipe in order for
  stats to fit in Pool list columns.

Fixes: https://tracker.ceph.com/issues/38284
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets

The crush/builder.c crush_add_bucket method resizes the max_buckets array
but a power of 2 when it has to expand, but the code in CrushWrapper was
assuming that if the array grew the pos for the new bucket would be the
last position in the new array.  This led to a situation where the
crush_choose_arg_map args array size didn't match max_buckets, and
eventually caused a crash.

Fixes: http://tracker.ceph.com/issues/38664
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ccda488815eeca608a58092d0d917c8a68a7b93c)

Conflicts:
src/crush/CrushWrapper.cc : Resolved in add_bucket

osd: Add OSD_BACKFILL_PRIORITY_FORCED with lower priority
than OSD_RECOVERY_PRIORITY_FORCED

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d088ffb09f46c6207cfedd2cdab9f074197af4bb)

test: Add scripts to test backfill/recovery priority handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ca5cf14fa83ee89260688d467257bb3d81efdd36)

osd AsyncReserver: Update priority in AsyncReserver for force and cancel-force

Fixes: http://tracker.ceph.com/issues/38041
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 11097f1235265a3eb753adc7e6d2818e5902e39e)

Backport change:
src/common/AsyncReserver.h (Mimic locking in update_priority())

test: Add option to wait_for_clean() to execute at every sleep

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 70b51362084d2ae5f2a6e9e5856f17452c53e3f3)

Merge pull request #26578 from badone/wip-mimic-test-dup_index-trim

mimic: test: Verify a log trim trims the dup_index

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26583 from pdvian/wip-38398-mimic

mimic: common: ceph_timer: stop timer's thread when it is suspended

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26612 from badone/wip-mimic-drop-GIL-in-get_config

mimic: mgr: drop GIL in get_config

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26707 from pdvian/wip-38505-mimic

mimic: rbd: API list_images() Segmentation fault

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26944 from ashishkumsingh/wip-38725-mimic

mimic: mgr/dashboard: Search broken for entries with null values

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #26517 from yehudasa/wip-rgw-es-fixes-mimic

mimic: rgw: multiple es related fixes and improvements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26687 from pdvian/wip-38502-mimic

mimic: rgw: rgw_file: use correct secret key to check auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26664 from pdvian/wip-38415-mimic

mimic: rgw: bug in versioning concurrent, list and get have consistency issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26945 from ashishkumsingh/wip-38726-mimic

mimic: rgw: bucket limit check misbehaves for > max-entries buckets (usually 1000)

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #26702 from ceph/backport-mimic-26685

mimic: ceph-volume: use our own testinfra suite for functional testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

librbd: avoid aggregate-initializing any static_visitor

This will result in an error complaining about a protected dtor.
also see:
    https://github.com/ceph/ceph/pull/26838

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 8abfb42856bfdef1318ba4a178555d95aeeaaf8b)

librbd: avoid aggregate-initializing IsWriteOpVisitor

Clang 8 complains about missing dtor, and this circumvents this problem.
```
/home/jenkins/workspace/ceph-master/src/librbd/io/ImageDispatchSpec.cc:136:48:
error: temporary of type 'boost::static_visitor<bool>' has protected
destructor
   return boost::apply_visitor(IsWriteOpVisitor{}, m_request);
                                                ^
/home/jenkins/workspace/ceph-master/src/librbd/io/ImageDispatchSpec.cc:152:28:
note: in instantiation of member function
'librbd::io::ImageDispatchSpec<librbd::ImageCtx>::is_write_op' requested
here
template class librbd::io::ImageDispatchSpec<librbd::ImageCtx>;
                            ^
/usr/local/include/boost/variant/static_visitor.hpp:53:5: note: declared
protected here
     ~static_visitor() = default;
     ^
1 error generated.
gmake[2]: *** [src/librbd/CMakeFiles/rbd_internal.dir/build.make:895:
src/librbd/CMakeFiles/rbd_internal.dir/io/ImageDispatchSpec.cc.o] Error 1
```

The details of the problem are explained in more detail in:
https://github.com/ceph/ceph/pull/26301/commits/09870d7a083913b5a3830ba8b0be04d0b0996aca

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 1d7b291874b65770c0f0a56f2df59326687e7083)

doc/rgw: document placement target configuration

Fixes: http://tracker.ceph.com/issues/24508
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c95585b1984e60708addee436a774d0c92ce57a6)

rgw: sync module: avoid printing attrs of objects in log

Since attrs are verbose and not necessarily needed as a debug info (as they can
be obtained from the required object itself) avoid logging them. Also
additionally removed another place where such a log is done by the ES sync
module

Fixes: http://tracker.ceph.com/issues/37646
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 8f37b70630ceb62aff4019e9ad2508977590a873)

Merge pull request #27014 from badone/wip-mimic-CA-mods

mimic: qa/ceph-ansible: modifications for mimic branch

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/objectstore: bluestore -> bluestore-{bitmap,stupid}

The new default is bitmap, so we were testing bitmap twice.  Instead,
explicitly call out stupid and bitmap cases so a future default change
won't break coverage.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f390704b4616c837352e7d6a8df91526cb047f0d)

os/bluestore: fix minor issues in bitmap logging output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dc4b5fc5afcea4be9e3b4c91dcc9dcce38d4e906)

os/bluestore: fix overlappedd allocation returned by bitmap allocator

Fixes: https://tracker.ceph.com/issues/24598
Signed-off-by: Igor Fedotov <ifedotov@suse.som>
(cherry picked from commit 539af4f27e773cb21d187b75086c7c9498dd7749)

os/bluestore: provide a rough estimate for bitmap allocator
fragmentation.
The approach is counting 'partial' and 'free' slots at L1 on the fly and
use partial / (partial+free) value as rough fragmentation estimate.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dbbe12fe6f543fc6942f41a108effc7eb1632f67)

os/bluestore: rename new bitmap allocator class to BitmapAllocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit a8e8c192acb48ce93553b04d1953ce8bee1b16e4)

os/bluestore: perform allocations aligned with min_length  in new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 51974c48e38e7080eb12c6028f95f4da666fbe76)

test/objectstore/unitetest_fastbmap_allocator: replace ASSERT_TRUE with
ASSERT_EQ

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 69ccb4367dcdcc51fbcfdaaea308c761291621e1)

os/bluestore: respect min_length as allocation granularity for new
bitmap allocator.
It was used a real minimum threshold before this fix which allowed e.g.
allocated extent length to be equal to min_length + 1.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 8a188b2aa3892b7f5a6c85042ffae8c2847ca869)

os/bluestore: cosmetic new allocator internal method rename.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fa06ff234f11f266bb10d7fa9ea28847e5ce72fe)

os/bluestore: properly respect min_length when analysing partially free
slots in bitmap allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 288bbd42531bebd8f6c619c848e75bfd9b8f93fd)

os/bluestore: cosmetic cleanup in new bitmap allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 3db9d9a66cbd62c543cdd88ff4ab636fba2c2a2b)

os/bluestore: more verbose logging in new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 160c73032db2e122f5e643090c86eed85e679eb4)

mimic: qa/ceph-ansible: modifications for mimic branch

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #26860 from ashishkumsingh/wip-38648-mimic

mimic: doc: Fix incorrect mention of 'osd_deep_mon_scrub_interval'

Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: remove original bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 95bd72fc22aa313112de096ac3ef98e45c71bce5)

os/bluestore: align BitMap allocator's init_rm_free/init_add_free parameters with min_alloc_size

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 1b6a56e80066decf8c3727090a2cd51ba9ded9bc)

os/bluestore: fix improper access to a BitmapFastAllocator::last_pos
from multiple threads.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 0c69672fce55362a7e6eb8ed6955f163afe85482)

test/allocator: move bluestore allocator's benchmarks to a standalone UT

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ac2ec80cddc51f1642849c7bf226e9bed116c0d6)

os/bluestore: add new bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 631041a7b788eaaf010f1e39f58b5546ab5580fc)

test/allocator: get rid off bitmap allocator specifics and introduce new
performance test cases

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dcd30610fb1c87711c9f43f7d3d6636888792345)

os/bluestore: add release(PExtentVector) helper to Allocator class to
free PExtentVector explicitly.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 328ea72a84edef4a6c4e0558dff52d0e85f05122)

os/bluestore: get rid off allocator's reserve method

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 88bea46242d4eb6bdfd8309ddf8843c9a14e5f25)

Merge pull request #26968 from pdvian/wip-38668-mimic

mimic: doc/radosgw: Document mappings of S3 Operations to ACL grants

Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/radosgw: Document mappings of S3 Operations to ACL grants

Since the ACL documentation is a bit sparse, let people know exactly
what operations they can expect to enable with a given ACL grant.

Fixes: https://tracker.ceph.com/issues/38523
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit bd94c93fada952537c7015743db55bff7619afa8)

qa: ignore slow metadata io wrn during osd thrash

Fixes: http://tracker.ceph.com/issues/38651
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5abcc32ff642b03c4b53e565e995ebb94802579a)

rgw: bucket limit check misbehaves for > max-entries buckets (usually 1000)

RGWBucketAdminOp::limit_check does not correctly handle
truncation/continuation markers across calls to rgw_read_user_buckets(...).
Also, it should, but does not, flush its formatter periodically.

Fixes: https://tracker.ceph.com/issues/35973
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 675b775ea52e0d7b81efb32c6cffa6987190184b)

mgr/dashboard: Search broken for entries with null values

Fixes: http://tracker.ceph.com/issues/38583
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
(cherry picked from commit ee988b90b84a484811b3351ff9eee242ae5b2ebd)

Conflicts:
- src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.ts
    Modified the file to keep changes to relevant to this version.

13.2.5

Merge pull request #26735 from aclamk/mimic-fix-1600138

mimic: os/bluestore: Fix bluefs - block balance.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #26789 from theanalyst/mimic-resharding-fixes

mimic: rgw: resharding fixes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

doc: Fix incorrect mention of 'osd_deep_mon_scrub_interval'

Fixed the incorrect mention of 'osd_deep_mon_scrub_interval' in health-checks.rst.
Changed it to 'osd_deep_scrub_interval'.

Fixes: https://tracker.ceph.com/issues/38310
Signed-off-by: Ashish Singh <assingh@redhat.com>
(cherry picked from commit 7108e6a3c7ec1f3ae3c39682f43db99efb2b7a6c)

Conflicts:
doc/rados/operations/health-checks.rst
           - Fixed conflicts to keep relevant changes.

Merge pull request #26814 from tspmelo/wip-fix-resolve-mimic

mimic: mgr/dashboard: Fix dashboard compile error

Reviewed-By: Sebastian Wagner <swagner@suse.com>
Reviewed-By: Laura Paduano <lpaduano@suse.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Fix dashboard compile error

New version of lodash was generating a compile error.
With this change npm will install an older version of lodash.

Fixes: https://tracker.ceph.com/issues/38590
Fixes: https://tracker.ceph.com/issues/38602
Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: fix typo

This fixes a typo in a log message. It's a separate commit so
downstream commits point to the right upstream commits via
cherry-pick.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 7d1768f2a332c5ac389cbb4bbeda28752e4f17bc)

rgw: perf -- remove bucket shards asynchronously rather than synchronously

We can now take advantage of the new asynchronous bucket shard removal
code and where we used to remove each shard synchronously now remove
them asynchronously. This would be a huge win when we have tens of
thousands of shards.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit cb0da45c5d6e4dff6514919b3b894df0eea7c1e4)

 Conflicts:
src/rgw/rgw_rados.cc
conflicts with placement set and rgw rados refactor

rgw: clean up unused bucket index shards

Clean up old bucket index shards when a resharding is complete. Also,
when a resharding fails, clean up unfinished bucket index shards. Do
both clean-ups asynchronously.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit f84f70d451036ee99011b8e2c7b974f15d3a005a)

 Conflicts:
src/rgw/rgw_rados.h
merge conflict as bucket_placement functions were moved after the rgw rados
refactor

rgw: a little code clean-up by adding const

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 48e22fb76feffb551074f274986bc4125a02003f)

rgw: remove never-called RGWRados member function

Because RGWRados::cls_rgw_init_index is never called, remove it.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 4593778c6d830a1a0d33528507f1795aee019ce4)

rgw: rename function for clarity

The function cls_rgw_bucket_init was renamed to
cls_rgw_bucket_init_index in order to better describe its
functionality.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 20868bd3b893ba2e17d2bfdc63d6544dbe677a9b)

rgw: batch and process bucket instances

Sort through and batch bucket instances so that multiple calls to reading
current bucket info and locking can be avoided. For the most trivial case when
the bucket is already deleted we exit early with all the stale instances. When
the bucket reshard is in progress we only process the stale entries with status
done, if the bucket is available for locking then we lock down and mark the
other instances as well.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit fb9c0497621fce34283094cfe260480f9122573f)

 Conflicts:
src/rgw/rgw_bucket.cc
Get rid of the following c++17isms:
- split_tenant auto return type -> trailing return type
- tuple destructuring bind for split tenant with std::tie

cli: use rgw-admin reshard stale-instances list/delete

Add a delete command as well that clears the resharded instances. We print out
the json status to indicate success or error state

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit bf8f885016a588130f05c25fdef3d3b6e1216c40)

rgw: implement listing stale instances from a reshard

Dynamic resharding used to leave behind stale bucket instances; walk through the
metadata pool and identify these instances by comparing the reshard status. If
the reshard status is done, these instances are ok to be cleared. For reshard
status of none we compare against the bucket entry point to ensure that we don't
match the current entry point.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 0c35a6fab250a817b2752f319b5c9f1da13bb3df)

ceph-volume: set number of osd ports in the tests

The number of ports the OSDs listen on depend on the version of ceph
being used, so we need to test for that number accordingly.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1132cb3c25fef9f853488cc8aab1327aff6bcf96)

os/bluestore: Fix problem with bluefs's freespace not being balanced when kv_sync_thread is sleeping.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

librbd: set *image_size to 0 when getting ENOENT in rbd_group_image_list()

Fixes: http://tracker.ceph.com/issues/38468
Signed-off-by: songweibin <song.weibin@zte.com.cn>
(cherry picked from commit 141e3a74b6fb10ca923c61dc90e7573f1becf002)

rbd/pybind: add __enter__ and __exit__ for class Group

Fixes: http://tracker.ceph.com/issues/38468
Signed-off-by: songweibin <song.weibin@zte.com.cn>
(cherry picked from commit bb8df22e4ae851d858b081a47f063a1f33b4d0c9)

mgr: drop GIL in get_config

Fixes: https://tracker.ceph.com/issues/35985
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d590a5391876eda23ef286af5800e5bdbb2b4b97)

Conflicts:
        src/mgr/ActivePyModules.cc

ceph-volume: use our own testinfra suite for functional testing

This allows us to run a smaller subset of tests from the ones provided
by ceph-ansible and test only the things that pertain to ceph-volume.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit ccfc8e6dc42b55f14fa3b4ef210cb639b9a7e20b)

rgw_file: use correct secret key to check auth

Currently, the authorize function in rgw_file only use the secret key of
first key pair to compare, add a function which get secret key by user's
access key and check the result is match with user's secret key.

Fixes: https://tracker.ceph.com/issues/37855
Signed-off-by: MinSheng Lin <minsheng.l@inwinstack.com>
(cherry picked from commit e92755c60325475342afcafcd5293a20e3e5216a)