Merge pull request #28310 from pdvian/wip-39674-mimic

mimic: qa/workunits/rbd: wait for rbd-nbd unmap to complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28815 from trociny/wip-40593-mimic

mimic: rbd-mirror: handle duplicates in image sync throttler queue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28852 from jdurgin/wip-18096-mimic

mimic: osd: report omap/data/metadata usage

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27971 from smithfarm/wip-39593-mimic

mimic: build/ops: Cython 0.29 removed support for subinterpreters: raises ImportError: Interpreter change detected ...

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28208 from votdev/fix_vstart_runner

mimic: qa: Fix issues in vstart runner

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28954 from bk201/wip-40707

mimic: mgr/dashboard: Fix run-frontend-e2e-tests.sh

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #28123 from dillaman/wip-39461-mimic

mimic: rbd-mirror: clear out bufferlist prior to listing mirror images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28133 from dillaman/wip-38441-mimic

mimic: librbd: ensure compare-and-write doesn't skip compare after copyup

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28770 from dillaman/wip-40573-mimic

mimic: librbd: properly track in-flight flush requests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28814 from trociny/wip-40461-mimic

mimic: journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

deb,rpm,do_cmake: switch to cmake3

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit af2c91ace6c1082925f118d145937ae55fa780d4)

cmake: remove cython 0.29's subinterpreter check during install

Commit 3bde34af8a removed cython 0.29's subinterpreter check when
building the various python modules during `make`, but unforunately
they're *rebuilt* during `make install`, with the rebuild overwriting
the original build.  The original fix was of course missing from the
install stage...

Fixes: https://tracker.ceph.com/issues/38788
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 108462e3fb0e814c98ab32f02b79cf3ea1247692)

cmake: remove cython 0.29's subinterpreter check

cython 0.29 introduced a check which prevents multiple python
subinterpreters from loading the same module:

https://github.com/cython/cython/commit/7e27c7c

Unfortunately, this completely breaks ceph-mgr.  Until we can
figure out a better long term solution, this commit removes
cython's subinterpreter check, via some careful abuse of the
C preprocessor.

This works because when cython is invoked, it first generates
some C code, then compiles it.  We know it's going to generate
C code including:

  int __Pyx_check_single_interpreter(void) { ... }

and:

  if (__Pyx_check_single_interpreter())
      return NULL;

So, we can do the following:

  #define void0 dead_function(void)
  #define __Pyx_check_single_interpreter(ARG)=ARG ## 0

This replaces the call to __Pyx_check_single_interpreter()
with a literal 0, removing the subinterpreter check.

The void0 dead_function(void) thing is necessary because
the __Pyx_check_single_interpreter() macro also clobbers
that function definition, so we need to make sure it's
replaced with something that works as a function definition.

Fixes: https://tracker.ceph.com/issues/37472
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 3bde34af8a84d76744887b2cf10ee4ef36e3925b)

Conflicts:
cmake/modules/Distutils.cmake
- mimic doesn't have the immediately preceding ccache stuff

Merge pull request #28161 from pdvian/wip-39426-mimic

mimic: mgr: deadlock

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #28326 from cbodley/wip-40088

mimic: common: parse ISO 8601 datetime format

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28399 from xiexingguo/wip-fudege-for-m

mimic: mgr/balancer: fix fudge

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28645 from ifed01/wip-ifed-fix-40080-mimic

mimic: os/bluestore: fix duplicate allocations in bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28948 from ceph/backport-mimic-28866

mimic: ceph-volume broken assertion errors after pytest changes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #28947 from ceph/backport-mimic-28836

mimic: ceph-volume tests add a sleep in tox for slow OSDs after booting

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Fix run-frontend-e2e-tests.sh

e2e tests were always returning positive result, even when they failed.

Fixed problem with jq, where it was failing when it was not receiving a string.

Fixes: http://tracker.ceph.com/issues/40707
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit ad293e1961065ea4b92244c8f059dff0ef73a016)

Conflicts:
src/pybind/mgr/dashboard/run-frontend-e2e-tests.sh

ceph-volume api.lvm catch IndexError when parsing dmmapper output

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfff70d07a1ccf7fb11a96dd7c63335cfc9b5e26)

ceph-volume tests update to use error.value instead of str(error)

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 484be7ffb1e0865ff840090a73036567a1f4f5d2)

ceph-volume tests add a sleep in tox for slow OSDs after booting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b398f99f810d97c94af6d943aed4c11a5fd69c19)

Merge pull request #28930 from ceph/backport-mimic-26739

mimic: ceph-volume: replace testinfra command with py.test

Reviewed-by: Alfredo Deza

Merge pull request #28209 from votdev/add_tenant_to_bucket

mimic: rgw: Return tenant field in bucket_stats function

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28249 from pdvian/wip-39532-mimic

mimic: rgw: resharding of a versioned bucket causes a bucket stats discrepancy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28348 from smithfarm/wip-39179-mimic

mimic: rgw: remove_olh_pending_entries() does not limit the number of xattrs to remove

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28409 from cbodley/wip-40147

mimic: rgw: fix bucket may redundantly list keys after BI_PREFIX_CHAR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28422 from pdvian/wip-39697-mimic

mimic: rgw: return ERR_NO_SUCH_BUCKET early while evaluating bucket policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28483 from pdvian/wip-39734-mimic

mimic: rgw: multisite: mismatch of bucket creation times from List Buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28668 from smithfarm/wip-40348-mimic

mimic: tests: use curl in wait_for_radosgw() in util/rgw.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28711 from pdvian/wip-40133-mimic

mimic: rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28716 from smithfarm/wip-40507-mimic

mimic: rgw: conditionally allow builtin users with non-unique email addresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.simple replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e90ebc3264004cab3286895c843db43f20a4840b)

ceph-volume tests.lvm replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c4597fc63195c654b960f747f9d50a558fa719c0)

ceph-volume tests.batch replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc7501af69ccbdabb5e6a77a91696454855201e9)

rbd-mirror: handle duplicates in image sync throttler queue

Fixes: http://tracker.ceph.com/issues/40519
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 2f35ab70a59fc22c02c98e5aa80a1e6cde29a226)

Conflicts:
    src/test/rbd_mirror/test_mock_ImageSyncThrottler.cc:
       no g_ceph_context arg for MockImageSyncThrottler constructor

os/bluestore: factor VDO into bluestore's 'allocated' value

If we are on VDO, report the *raw* storage we've allocated, as reported
by VDO.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 04b47939b84f388c3a8c0b09077fe56b27d2f9f2)

os/filestore: estimate omap_allocated

Assume all of leveldb/rocksdb is omap.  This is an overestimate, but
better than nothing.

We don't populate the metadata overhead (no easy way to calculate this
that comes to mind).  And we don't populate the compression-related
fields.  It's possible we could make something up here in the VDO
case...

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 186edca81ef09058b9c121967ab1850fbb5812b6)

os/ObjectMap: add get_db() accessor

This is just to let us get at the underlying KeyValueDB for DBObjectMap.
It is not really better or worse than adding accessors for things like
GetEstimatedSize() to ObjectMap.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6c82e66dde715e5e4e12261655c5e7b521871453)

osd/OSDMap: include data, omap in 'osd df' output

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 13295a38839d3df18994be101eb69361ce30cd34)

os/bluestore: report omap and internal metadata usage

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fc5561ed6e1d44cda83a7c31d27f7392ee2e18ee)

os/bluestore/BlueFS: drop get_fs_usage(); add get_used()

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 52cfc505fb36f6cc77e29b246db8e0e833116a23)

kv: add per-prefix space estimation

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c6da00b418764ca8fec248b768f3744ca1e655ed)

osd: decouple statfs update from hb peers, pg count

These don't need to be updated in synchrony.  (In fact, the statfs update
could be much more infrequent.)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 71d8fe7faf4952f1743103b16f590d46a9488684)

osd: drop useless OSDService::update_osd_stat()

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a66f8bfe7d49f7d33ec027eeeb67870c610610db)

osd/osd_types: separate accounting for data, omap, and metadata

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fa120cec38b11378a7e6dbe37397f9a203f2f75c)

Conflicts:
src/os/bluestore/BlueStore.cc
src/test/objectstore/store_test.cc

Trivial - gconf and ceph_assert

osd: re-add get_num_pgs()

We removed this but it's a useful accessor.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b116d2140871e0bb6d562cb3303ad3116145d55d)

Merge pull request #28029 from ifed01/wip-ifed-dump-before-nospanid-mimic

mimic: os/bluestore: dump before "no-spanning blob id" abort

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28619 from xiexingguo/wip-40230

mimic: mon, osd: parallel clean_pg_upmaps

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28138 from dillaman/wip-39186-mimic

mimic: rbd: filter out group/trash snapshots from snap_list

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28139 from dillaman/wip-38563-mimic

mimic: librbd: race condition possible when validating RBD pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28150 from trociny/wip-37691-mimic

mimic: librbd: disable image mirroring when moving to trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28151 from dillaman/wip-38509-mimic

mimic: librbd: add missing shutdown states to managed lock helper

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28202 from dzafman/wip-39698

mimic: osd: Don't include user changeable flag in snaptrim related assert

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28232 from pdvian/wip-39518-mimic

mimic: osd: Don't evict after a flush if intersecting scrub range

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #28259 from pdvian/wip-39538-mimic

mimic: osd/PG: fix last_complete re-calculation on splitting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28503 from pdvian/wip-39737-mimic

mimic: osd: Output Base64 encoding of CRC header if binary data present

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28540 from pdvian/wip-39744-mimic

mimic: mon: paxos: introduce new reset_pending_committing_finishers for safety

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #28574 from smithfarm/wip-40280-mimic

mimic: bluestore: 50-100% iops lost due to bluefs_preextend_wal_files = false

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

journal: properly advance read offset after skipping invalid range

Fixes: https://tracker.ceph.com/issues/40409
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 7b8fd11fc6f49ca69a8484ee4b175b530b2dd2a3)

librbd: remove special case for starting AioCompletion ops

All ops can be immediately started now that flush ops won't
accidentally block themselves.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b5fc7ecaf7a3741d227ab5a9108392813dd03495)

Conflicts:
src/librbd/io/AioCompletion.cc: assert/ceph_assert conflicts
src/librbd/io/ImageRequest.cc: assert/ceph_assert conflicts

librbd: simplify IO flush handling through AsyncOperation

Allow ImageFlushRequest to directly execute a flush call through
AsyncOperation. This will allow the flush to be directly linked
to its preceeding IOs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 09e4127d5df1e2a79f2536dc784ec3730feea9ce)

Conflicts:
src/librbd/io/AsyncOperation.cc: trivial resolution
src/test/librbd/io/test_mock_CopyupRequest.cc: DNE

rgw: conditionally allow non-unique email addresses

Conditionally allow non-unique email address values for builtin
RGW users.

Fixes: http://tracker.ceph.com/issues/40089
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 974791522007cca6d8fb30e83677f0ddd7c4e71d)

Conflicts:
    src/rgw/rgw_user.cc
- changed '_conf.get_val<bool>' to '_conf->get_val<bool>'

rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Fixes: http://tracker.ceph.com/issues/20797
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 40e602bc3866598952eb0dd68ecec947dd7b70d6)

qa: use curl in wait_for_radosgw() in util/rgw.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 9c6afa3fb570629a57612f250a6a4890a65cbc6e)

os/bluestore: BitmapAllocator::allocate: log new extents only

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit a90d57ca05c902eab8ca19090b856ea0e08f5d2b)

os/bluestore: distinguis request and results in bitmap alloc log

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 84c2d4148574373c678ad8f00e35f9e209ba1f18)

os/bluestore: fix duplicate allocations in bmap allocator

Fixes: http://tracker.ceph.com/issues/40080
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit eac81bc85970287fce60178e80c37628b3df0e7d)

 Conflicts:
src/os/bluestore/fastbmap_allocator_impl.cc
        assert vs. ceph_assert mismatch

test: add parallel clean_pg_upmaps test

With parallel clean_pg_upmaps feature on, the total time cost
of the performance test which now can utilize up to 8 threads for
parallel upmap validating decreased from:

maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:104s

to:

clean_pg_upmaps (~10000 pg_upmap_items) latency:7s

Note that by default the mon uses only 4 worker threads for
CPU intensive background work, you could further increase
the "mon_cpu_threads" option value if you decided the
time-consuming of clean_pg_upmaps still matters.

Fixes: http://tracker.ceph.com/issues/40104
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a45112a9761d7b11f62cf4dac9f3a8e093cdd78f)

mon/OSDMonitor: do clean_pg_upmaps the parallel way if necessary

There could definitely be some certain cases we could reliably
skip this kind of checking, but there is no easy way to separate
those out.
However, this is clearly the general way to do the massive pg
upmap clean-up job more efficiently and hence should make sense
in all cases.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit c395f45f1f4d6f5e2b538a34730d9c92d8f9ae8b)

osd/OSDMap: split clean_pg_upmaps into smaller helpers

- it's good to read.
- the updating pending_inc part should be made independent
  since it is going to be racy while running in parallel.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4d5cf1e4173e5151cc571af571edb2eab0bb46a7)

Conflicts:
slight conflicts with pg-merge

osd: maybe_remove_pg_upmaps -> clean_pg_upmaps

It should always be the preferred option to kill the unnecessary
or duplicated code, which is good for maintenance.
Also I've noticed there is already a clean_temps helper, so re-naming
maybe_remove_pg_upmaps to clean_pg_upmaps to at least keep pace with
that sounds to be a natural choice for me..

Master PR: https://github.com/ceph/ceph/pull/28373

This does not follow the normal backport process
since this piece of code has been changed a lot for the past 18 months..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMapMapping: make ParallelPGMapper can accept input pgs

The existing "prime temp" machinism is a good inspiration
for cluster with a large number of pgs that need to do various
calculations quickly.
I am planning to do the upmap tidy-up work the same way, hence
the need for an alternate way of specifying pgs to process other
than taking directly from the map.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit f6fd4a312e0dda260f2c150334f06b531678ce47)

Conflicts:
s/g_conf./g_conf->/

osd/OSDMap: maybe_remove_pg_upmaps - avoid do_crush twice

which is extremely time-consuming.
Half of the amount of time of calling maybe_remove_pg_upmaps
has been saved by applying this patch as a result..

Was: maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:104s
Now: maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:56s

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 02e5499b350bcd7d9eac98b2072052a9a4a1f535)

Conflicts:
- s/nextmap/tmpmap/
- drop std:: namespace

osd/OSDMap: maybe_remove_pg_upmaps - s/pg_to_raw_up/pg_to_raw_upmap/

The upmap results are directly applied after calling
_pg_to_raw_osds, which means it basically has nothing to do
with the up/down status.
In other words, if a pg_upmap/pg_upmap_items remapped a pg
into some down osds and is now causing collided result,
we should still be able to detect and cancel that.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit d9ed406a61c46858dd8350af5c72d7b8824dcdd3)

Conflicts:
s/nextmap/tmpmap/

test/osd: add performance test case for maybe_remove_pg_upmap

Tom Byrne reported that maybe_remove_pg_upmap might become
super inefficient for large clusters with balancer on.
To identify and resolve the problem, we need to add some good
measurements first.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit c0ce22b8c861fb76957b4cbbd59d9800e1ec09c3)

osd/bluestore: Actually wait until completion in write_sync

This function is only used by RocksDB WAL writing so it must sync data.

This fixes #18338 and thus allows to actually set `bluefs_preextend_wal_files`
to true, gaining +100% single-thread write iops in disk-bound (HDD or bad SSD) setups.
To my knowledge it doesn't hurt performance in other cases.
Test it yourself on any HDD with `fio -ioengine=rbd -direct=1 -bs=4k -iodepth=1`.

Issue #18338 is easily reproduced without this patch by issuing a `kill -9` to the OSD
while doing `fio -ioengine=rbd -direct=1 -bs=4M -iodepth=16`.

Fixes: https://tracker.ceph.com/issues/18338 https://tracker.ceph.com/issues/38559
Signed-off-by: Vitaliy Filippov <vitalif@yourcmc.ru>
(cherry picked from commit c703cf9a7632cbd9f17e148ef203509549a28571)

Conflicts:
src/os/bluestore/KernelDevice.cc
- mimic has a single variable "fd_buffered" where master has an array "fd_buffereds"

mon: paxos: introduce new reset_pending_committing_finishers for safety

There are asserts about the state of the system and pending_finishers which can
be triggered by running arbitrary commands through again. They are correct
when not restarting, but when we do restart we need to take care to preserve
the same invariants as appropriate. Use this function to be careful about
the order of committing_finishers v pending_finishers and to make sure they're
both empty before any Contexts actually get called.

We also reorder a call to finish_contexts on the waiting_for_writeable list for
similar reasons.

Fixes: http://tracker.ceph.com/issues/39484
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit b17caec586ba2801593db61f91d66719d40b905e)

osd: Output Base64 encoding of CRC header if binary data present

Add optional paramter so cleanbin() for bufferlist can include "Base64:"

Fixes: https://tracker.ceph.com/issues/39582
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eea239c03ca8aeb9a1da742b07d8627fbe2317e2)

Conflicts:
src/include/util.h : Resolved for cleanbin
src/osd/ReplicatedBackend.cc : Resolved for util.h

test/rgw: add multisite test_bucket_creation_time()

does not create any buckets itself, but checks that all creation times
match on buckets created by other tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8df991c5cc48a6709185edf506d8bdec4d28e371)

cls/user: cls_user_set_buckets_info overwrites creation_time

metadata sync of a new bucket entrypoint may call rgw_link_bucket()
(which in turn calls into cls user) without deleting/unlinking the
previous bucket entrypoint. this prevented the new bucket entrypoint
from overwriting the creation_time of the old one

Fixes: http://tracker.ceph.com/issues/39635
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c469e5150a3e95b547a7d05d1ae08fa88fad7159)

Merge pull request #28457 from yuriw/wip-yuriw-40208-mimic

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: return ERR_NO_SUCH_BUCKET early while evaluating bucket policy

Right now we create a ERR_NO_SUCH_BUCKET ret code but continue further
processing. Since this ret code isn't returned at any stage we end up creating a
bucket instance anyway which shouldn't happen and then succeeding the client
call in cases like put bucket versioning. Return an error code early in these
cases

Fixes: http://tracker.ceph.com/issues/38420
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit cf66a6d5a7eee294313a1a08d0524daf939747e5)

cls/rgw: test before accessing pkeys->rbegin()

if pkeys is empty here, dereferencing rbegin() will crash

Fixes: http://tracker.ceph.com/issues/39984
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a936a06a657ce1cc118d75f66773aeda4167bbf3)

rgw: fix bucket may redundantly list keys after BI_PREFIX_CHAR

Fixes: http://tracker.ceph.com/issues/39984
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit dc3ffe99b910d04b68d70a1af2f38d877e05cd25)

mgr/balancer: fix fudge

next_* simply record some intermediate results when balancer tries
to find something to optimize, whereas best_* are used for the
final outputs.

Fix by comparing origin_ow with best_ow when determining if we
should use a valid fudge value.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 8f46a351f1fa6f921b373aed4038efc0898a8a44)

Merge pull request #28389 from yuriw/wip-yuriw-whitelist-mimic

qa/tests: whitelisted  'application not enabled'

qa/tests: whitelisted  'application not enabled'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

13.2.6

Merge pull request #28356 from yuriw/wip-yuriw-whitelist-mimic

qa/tests: whitelisted POOL_APP_NOT_ENABLED

qa/tests: whitelisted POOL_APP_NOT_ENABLED

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28301 from yuriw/wip-yuriw-exl-python-mimic

qa/tests: added 'python3-cephfs','python3-rados' to excluded packadges

rgw: limit entries in remove_olh_pending_entries()

If there are too many entries to send in a single osd op, the osd rejects
the request with EINVAL. This error happens in follow_olh(), which means
that requests against the object logical head (requests with no version
id) can't be resolved to the current object version. In multisite, this
also causes data sync to get stuck in retries

Fixes: http://tracker.ceph.com/issues/39118
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3805ea635a5f28656072b2cf5cdc9b5bf53b5e97)

Conflicts:
src/rgw/rgw_rados.cc
- mimic ref.ioctx.operate() call takes ref.oid instead of ref.obj.oid

qa/tests: added 'python3-cephfs','python3-rados' to excluded packadges

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28129 from dillaman/wip-39728-mimic

mimic: qa/workunits/rbd: use https protocol for devstack git operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

common/utime: don't pass %z to utime if there is a 'Z'

This fails on centos 7 for some reason.

If it's UTC, we don't need to parse the tz portion, so avoid the issue by
leaving the %z out of the fmt string.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1d1c4782eea1fac0e9230ed5ac32b19517fdd616)

common/utime: make parse() handle (our) ISO 8601 output

- optional subsecond values
- optional timezone offset

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 668d6e7f97bd37f1896611249cfae85bced4b096)

qa/workunits/rbd: wait for rbd-nbd unmap to complete

The "unmap" request is asynchronous, so wait for a short amount
of time for the "rbd-nbd" daemon process to exit.

Fixes: http://tracker.ceph.com/issues/39598
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 87278548a5a476692511b76b661278501bf72429)

Conflicts:
qa/workunits/rbd/rbd-nbd.sh : Resolved for map/unmap namespace test