ceph-volume: use two NVME drives in centos7 mixed_type tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 934c32acb318aac9aff3c800cdbf99bbdee04d06)

ceph-volume: add explicit mixed-type functional tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bea35c63c83fc2ab87fc983774ed8181cbe74e5b)

ceph-volume: add tests for Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 01185fb32479f271beeb14dcf94e32b1401ad2d9)

ceph-volume: adds a lvm_size property to util.Device

This will give us the size lvm should report because it takes into
account the 1GB physical extent size we set.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5197c16b44fcf344862201549777dc2935fc19a3)

ceph-volume: functional tests, add libvirt customization

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c330a4758755e3815b30e4f00a4a00e8319db399)

ceph-volume: add space between words

The current help of batch was showing:

rotational drivesare passed in DEVICES

This commit fixes it and now we have:

rotational drives are passed in DEVICES

Tremendous fix, I know.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit a151e111a2710e864bd66a44db38685459cf55c8)

ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 8ebff475dfa136c88101ef7364820fe3c5cee547)
(cherry picked from commit aa6da416cb14ba521796c950fba9667e7b581b27)

ceph-volume: set a lvm_size property on the fakedevice fixture

This is so tests can continue to set sys_api['size'] and the code
can retrieve that as Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9e2175c4001434f7b0ed8ce6f723e063cf001440)

ceph-volume: don't crash on unusable data devices

Also add test case for a MixedStrategy with unusable data devices.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_filestore.py
        Pick test addition and error-value change

ceph-volume: add a wal device specific unit test and size validator

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_bluestore.py
        Pick test addition and error.value change

ceph-volume: add osd_ids argument

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: strategies allow for external wal lv's

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: add default argument values to strategy

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 5465712e4d71315ff8daac9345be6ba96d1f44c4)

 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_bluestore.py
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_filestore.py
        pick with_auto_devices and error.value changes

Add a unit test that tests the ceph_volume.devices.lvm.batch.Batch::get_devices method.

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 19a38189fd15f57907449631a0f6e9faf908dc66)

ceph-volume strategies: rename db_devs to db_or_journal_devs

This aims at reducing confusion when reading the code that is shared
between the bluestore and filestore strategies.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 029414e89b773896fddebe0a5869b57ec13f527e)

ceph-volume batch: rename block_devs to data_devs for consistency

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 6dbf43e53ccd3ff06e3cdfc8e6de97475cc0c510)

ceph-volume batch: refactor filtered_devices and strategy retrieval

Keep filtered_devices as a member variable of the batch object and pass
this to the report functionality instead of having the report functions
retrieve filtered_devices. Also keep selected strategy as a batch member
variable to avoid re-retrival in e.g. the report methods.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit dc2c4c597249d61ea6f48d37dbe02d8b97e3d55d)

ceph-volume: refactor strategies for explicit device arguments

This commit refactors lvm strategies to:
a) change the constructor to receive usage specific device lists
b) add classmethod with_auto_devices of automatic splitting by the
rotational flag (keep backwards compatible behavior)
c) rename ssds and hdds member variables to wal_devs and block_devs
d) add db_devs member variable

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 60dec8b1dfdd056662ba6a030de291f93b2f3650)

ceph-volume: move arg parsing to __init__ and add explicit device lists

Moving the arg parsing to __init__ allows to get the help message with
running as root.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit a7717818ab3790b226067da3dee8f8c216ab8f4b)

ceph-volume: use Device.lvm_size in batch strategy validators

This will validate the devices on the size that lvm will allow to be
used, not the raw physical size of the device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit aaa915cfe6678d5326b2c15730febfff79e40463)

ceph-volume: batch mixed type scenarios have no need to calulate data size

We know with a mixed type scenario the device used for data will be used
at 100% capacity. This means we do not need to be explict when asking
for the size of the data lvs, which avoids rounding errors with very
small device sizes.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0023961397d19659200eaf420f2789db17a71b34)

ceph-volume: set use_large_block_db in validate, not compute

The self.use_large_block_db property was never getting set because
the block in compute was never called as block_db_size was reset in
validate if it was 0. We needed to set self.use_large_block_db in
validate instead of compute.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit fdfb79b91c52f82877d08190cb626ed00b45f3e8)

ceph-volume: use Device.lvm_size in batch strategies

We should show the user what the size of the device will be after lvm
creates a pv out of it. This way there isn't a discrepency between the
sizes that are reported to the user and what is actually created.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 071e7ce41a58702c0eb84c35bd9388e2ec24ac13)

ceph-volume: use extents instead of size when creating block.db lvs

In the batch command use extents instead of size when creating lvs. This
gives a more precise size and avoids rounding errors.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit add9f88dc800913adf387c90cbf4352e392b0a75)

 Conflicts:
src/ceph-volume/ceph_volume/devices/lvm/strategies/bluestore.py
        remove json import; pick ours and add self.use_large_block_db = False

(cherry picked from commit 83e71842d8cd77b7d45efdd344bd8fac31078745)

Merge pull request #28887 from pdvian/wip-40262-mimic

mimic: rgw: rgw_file: all directories are virtual with respect to contents

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #29018 from smithfarm/wip-38776-mimic

mimic: rgw: support delimiter longer then one symbol

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28073 from pdvian/wip-39229-mimic

mimic: rgw_file: save etag and acl info in setattr

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28757 from smithfarm/wip-38747-mimic

mimic: rgw: non-existent mdlog failures logged at level 0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28310 from pdvian/wip-39674-mimic

mimic: qa/workunits/rbd: wait for rbd-nbd unmap to complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28815 from trociny/wip-40593-mimic

mimic: rbd-mirror: handle duplicates in image sync throttler queue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28852 from jdurgin/wip-18096-mimic

mimic: osd: report omap/data/metadata usage

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27971 from smithfarm/wip-39593-mimic

mimic: build/ops: Cython 0.29 removed support for subinterpreters: raises ImportError: Interpreter change detected ...

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28208 from votdev/fix_vstart_runner

mimic: qa: Fix issues in vstart runner

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28954 from bk201/wip-40707

mimic: mgr/dashboard: Fix run-frontend-e2e-tests.sh

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

rgw: always generate after delimiter char to skip directory

the following code also need after_delim_s to skip directory after list
1000 objects

Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit 8c6678387b00c862d71cffd6fed6ca6c236e3fb7)

rgw: fix list bucket with start maker and delimiter '/' will miss next object with char '0'

'/' + 1 = '0', and the start marker will be skipped, so it will miss the
'0' object.

Fixes: http://tracker.ceph.com/issues/39989
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit 43b5264c7d61ae3be56d68f131b544edee307a46)

Conflicts:
src/rgw/rgw_rados.cc
- mimic does not have "unsigned" in the line "unsigned char e = delim.back();"

rgw: prefix-delimiter listing: support >1 character delimiter

Fix prefix and CommonPrefix extraction logic in
RGWRados::Bucket::List::list_objects_ordered so as to permit
arbitrary-length string delimiters.

Fixes: https://tracker.ceph.com/issues/24821
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit e3c1ea244234aace7368d5a5ee95af2f6a529b00)

Merge pull request #28123 from dillaman/wip-39461-mimic

mimic: rbd-mirror: clear out bufferlist prior to listing mirror images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28133 from dillaman/wip-38441-mimic

mimic: librbd: ensure compare-and-write doesn't skip compare after copyup

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28770 from dillaman/wip-40573-mimic

mimic: librbd: properly track in-flight flush requests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28814 from trociny/wip-40461-mimic

mimic: journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

deb,rpm,do_cmake: switch to cmake3

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit af2c91ace6c1082925f118d145937ae55fa780d4)

cmake: remove cython 0.29's subinterpreter check during install

Commit 3bde34af8a removed cython 0.29's subinterpreter check when
building the various python modules during `make`, but unforunately
they're *rebuilt* during `make install`, with the rebuild overwriting
the original build.  The original fix was of course missing from the
install stage...

Fixes: https://tracker.ceph.com/issues/38788
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 108462e3fb0e814c98ab32f02b79cf3ea1247692)

cmake: remove cython 0.29's subinterpreter check

cython 0.29 introduced a check which prevents multiple python
subinterpreters from loading the same module:

https://github.com/cython/cython/commit/7e27c7c

Unfortunately, this completely breaks ceph-mgr.  Until we can
figure out a better long term solution, this commit removes
cython's subinterpreter check, via some careful abuse of the
C preprocessor.

This works because when cython is invoked, it first generates
some C code, then compiles it.  We know it's going to generate
C code including:

  int __Pyx_check_single_interpreter(void) { ... }

and:

  if (__Pyx_check_single_interpreter())
      return NULL;

So, we can do the following:

  #define void0 dead_function(void)
  #define __Pyx_check_single_interpreter(ARG)=ARG ## 0

This replaces the call to __Pyx_check_single_interpreter()
with a literal 0, removing the subinterpreter check.

The void0 dead_function(void) thing is necessary because
the __Pyx_check_single_interpreter() macro also clobbers
that function definition, so we need to make sure it's
replaced with something that works as a function definition.

Fixes: https://tracker.ceph.com/issues/37472
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 3bde34af8a84d76744887b2cf10ee4ef36e3925b)

Conflicts:
cmake/modules/Distutils.cmake
- mimic doesn't have the immediately preceding ccache stuff

Merge pull request #28161 from pdvian/wip-39426-mimic

mimic: mgr: deadlock

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #28326 from cbodley/wip-40088

mimic: common: parse ISO 8601 datetime format

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28399 from xiexingguo/wip-fudege-for-m

mimic: mgr/balancer: fix fudge

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28645 from ifed01/wip-ifed-fix-40080-mimic

mimic: os/bluestore: fix duplicate allocations in bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28948 from ceph/backport-mimic-28866

mimic: ceph-volume broken assertion errors after pytest changes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #28947 from ceph/backport-mimic-28836

mimic: ceph-volume tests add a sleep in tox for slow OSDs after booting

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Fix run-frontend-e2e-tests.sh

e2e tests were always returning positive result, even when they failed.

Fixed problem with jq, where it was failing when it was not receiving a string.

Fixes: http://tracker.ceph.com/issues/40707
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit ad293e1961065ea4b92244c8f059dff0ef73a016)

Conflicts:
src/pybind/mgr/dashboard/run-frontend-e2e-tests.sh

ceph-volume api.lvm catch IndexError when parsing dmmapper output

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfff70d07a1ccf7fb11a96dd7c63335cfc9b5e26)

ceph-volume tests update to use error.value instead of str(error)

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 484be7ffb1e0865ff840090a73036567a1f4f5d2)

ceph-volume tests add a sleep in tox for slow OSDs after booting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b398f99f810d97c94af6d943aed4c11a5fd69c19)

Merge pull request #28930 from ceph/backport-mimic-26739

mimic: ceph-volume: replace testinfra command with py.test

Reviewed-by: Alfredo Deza

Merge pull request #28209 from votdev/add_tenant_to_bucket

mimic: rgw: Return tenant field in bucket_stats function

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28249 from pdvian/wip-39532-mimic

mimic: rgw: resharding of a versioned bucket causes a bucket stats discrepancy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28348 from smithfarm/wip-39179-mimic

mimic: rgw: remove_olh_pending_entries() does not limit the number of xattrs to remove

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28409 from cbodley/wip-40147

mimic: rgw: fix bucket may redundantly list keys after BI_PREFIX_CHAR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28422 from pdvian/wip-39697-mimic

mimic: rgw: return ERR_NO_SUCH_BUCKET early while evaluating bucket policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28483 from pdvian/wip-39734-mimic

mimic: rgw: multisite: mismatch of bucket creation times from List Buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28668 from smithfarm/wip-40348-mimic

mimic: tests: use curl in wait_for_radosgw() in util/rgw.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28711 from pdvian/wip-40133-mimic

mimic: rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28716 from smithfarm/wip-40507-mimic

mimic: rgw: conditionally allow builtin users with non-unique email addresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.simple replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e90ebc3264004cab3286895c843db43f20a4840b)

ceph-volume tests.lvm replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c4597fc63195c654b960f747f9d50a558fa719c0)

ceph-volume tests.batch replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc7501af69ccbdabb5e6a77a91696454855201e9)

rbd-mirror: handle duplicates in image sync throttler queue

Fixes: http://tracker.ceph.com/issues/40519
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 2f35ab70a59fc22c02c98e5aa80a1e6cde29a226)

Conflicts:
    src/test/rbd_mirror/test_mock_ImageSyncThrottler.cc:
       no g_ceph_context arg for MockImageSyncThrottler constructor

rgw_file: all directories are virtual with respect to contents

This change causes directory handles to always report an mtime of
"now."  This is not an invalidate per se--it interacts with the
nfs implementation to produce that result when the implementation
updates its cached attributes.  Hence, it can be modulated by timers
or other rules governing attribute caching at the upper layer.

Fixes: http://tracker.ceph.com/issues/40204
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit b4c7d0faeff667c25ab255786999ef0cc844ea2b)

os/bluestore: factor VDO into bluestore's 'allocated' value

If we are on VDO, report the *raw* storage we've allocated, as reported
by VDO.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 04b47939b84f388c3a8c0b09077fe56b27d2f9f2)

os/filestore: estimate omap_allocated

Assume all of leveldb/rocksdb is omap.  This is an overestimate, but
better than nothing.

We don't populate the metadata overhead (no easy way to calculate this
that comes to mind).  And we don't populate the compression-related
fields.  It's possible we could make something up here in the VDO
case...

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 186edca81ef09058b9c121967ab1850fbb5812b6)

os/ObjectMap: add get_db() accessor

This is just to let us get at the underlying KeyValueDB for DBObjectMap.
It is not really better or worse than adding accessors for things like
GetEstimatedSize() to ObjectMap.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6c82e66dde715e5e4e12261655c5e7b521871453)

osd/OSDMap: include data, omap in 'osd df' output

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 13295a38839d3df18994be101eb69361ce30cd34)

os/bluestore: report omap and internal metadata usage

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fc5561ed6e1d44cda83a7c31d27f7392ee2e18ee)

os/bluestore/BlueFS: drop get_fs_usage(); add get_used()

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 52cfc505fb36f6cc77e29b246db8e0e833116a23)

kv: add per-prefix space estimation

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c6da00b418764ca8fec248b768f3744ca1e655ed)

osd: decouple statfs update from hb peers, pg count

These don't need to be updated in synchrony.  (In fact, the statfs update
could be much more infrequent.)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 71d8fe7faf4952f1743103b16f590d46a9488684)

osd: drop useless OSDService::update_osd_stat()

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a66f8bfe7d49f7d33ec027eeeb67870c610610db)

osd/osd_types: separate accounting for data, omap, and metadata

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fa120cec38b11378a7e6dbe37397f9a203f2f75c)

Conflicts:
src/os/bluestore/BlueStore.cc
src/test/objectstore/store_test.cc

Trivial - gconf and ceph_assert

osd: re-add get_num_pgs()

We removed this but it's a useful accessor.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b116d2140871e0bb6d562cb3303ad3116145d55d)

Merge pull request #28029 from ifed01/wip-ifed-dump-before-nospanid-mimic

mimic: os/bluestore: dump before "no-spanning blob id" abort

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28619 from xiexingguo/wip-40230

mimic: mon, osd: parallel clean_pg_upmaps

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28138 from dillaman/wip-39186-mimic

mimic: rbd: filter out group/trash snapshots from snap_list

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28139 from dillaman/wip-38563-mimic

mimic: librbd: race condition possible when validating RBD pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28150 from trociny/wip-37691-mimic

mimic: librbd: disable image mirroring when moving to trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28151 from dillaman/wip-38509-mimic

mimic: librbd: add missing shutdown states to managed lock helper

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28202 from dzafman/wip-39698

mimic: osd: Don't include user changeable flag in snaptrim related assert

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28232 from pdvian/wip-39518-mimic

mimic: osd: Don't evict after a flush if intersecting scrub range

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #28259 from pdvian/wip-39538-mimic

mimic: osd/PG: fix last_complete re-calculation on splitting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28503 from pdvian/wip-39737-mimic

mimic: osd: Output Base64 encoding of CRC header if binary data present

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28540 from pdvian/wip-39744-mimic

mimic: mon: paxos: introduce new reset_pending_committing_finishers for safety

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #28574 from smithfarm/wip-40280-mimic

mimic: bluestore: 50-100% iops lost due to bluefs_preextend_wal_files = false

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

journal: properly advance read offset after skipping invalid range

Fixes: https://tracker.ceph.com/issues/40409
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 7b8fd11fc6f49ca69a8484ee4b175b530b2dd2a3)

librbd: remove special case for starting AioCompletion ops

All ops can be immediately started now that flush ops won't
accidentally block themselves.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b5fc7ecaf7a3741d227ab5a9108392813dd03495)

Conflicts:
src/librbd/io/AioCompletion.cc: assert/ceph_assert conflicts
src/librbd/io/ImageRequest.cc: assert/ceph_assert conflicts

librbd: simplify IO flush handling through AsyncOperation

Allow ImageFlushRequest to directly execute a flush call through
AsyncOperation. This will allow the flush to be directly linked
to its preceeding IOs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 09e4127d5df1e2a79f2536dc784ec3730feea9ce)

Conflicts:
src/librbd/io/AsyncOperation.cc: trivial resolution
src/test/librbd/io/test_mock_CopyupRequest.cc: DNE

rgw_sync: drop ENOENT error logs from mdlog

Since this commonly occurs and isn't an error of concern at level 1.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit c246abb3935266f9335cab97e52ab489cc7ebfd6)

Conflicts:
src/rgw/rgw_sync.cc
- mimic has "ldout(cct, 1)" where master has "ldpp_dout(sync_env->dpp, 1)"

rgw: conditionally allow non-unique email addresses

Conditionally allow non-unique email address values for builtin
RGW users.

Fixes: http://tracker.ceph.com/issues/40089
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 974791522007cca6d8fb30e83677f0ddd7c4e71d)

Conflicts:
    src/rgw/rgw_user.cc
- changed '_conf.get_val<bool>' to '_conf->get_val<bool>'

rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Fixes: http://tracker.ceph.com/issues/20797
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 40e602bc3866598952eb0dd68ecec947dd7b70d6)

qa: use curl in wait_for_radosgw() in util/rgw.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 9c6afa3fb570629a57612f250a6a4890a65cbc6e)

os/bluestore: BitmapAllocator::allocate: log new extents only

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit a90d57ca05c902eab8ca19090b856ea0e08f5d2b)