osd/bluestore: Actually wait until completion in write_sync

This function is only used by RocksDB WAL writing so it must sync data.

This fixes #18338 and thus allows to actually set `bluefs_preextend_wal_files`
to true, gaining +100% single-thread write iops in disk-bound (HDD or bad SSD) setups.
To my knowledge it doesn't hurt performance in other cases.
Test it yourself on any HDD with `fio -ioengine=rbd -direct=1 -bs=4k -iodepth=1`.

Issue #18338 is easily reproduced without this patch by issuing a `kill -9` to the OSD
while doing `fio -ioengine=rbd -direct=1 -bs=4M -iodepth=16`.

Fixes: https://tracker.ceph.com/issues/18338 https://tracker.ceph.com/issues/38559
Signed-off-by: Vitaliy Filippov <vitalif@yourcmc.ru>
(cherry picked from commit c703cf9a7632cbd9f17e148ef203509549a28571)

Conflicts:
src/os/bluestore/KernelDevice.cc
- mimic has a single variable "fd_buffered" where master has an array "fd_buffereds"

Merge pull request #28457 from yuriw/wip-yuriw-40208-mimic

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28389 from yuriw/wip-yuriw-whitelist-mimic

qa/tests: whitelisted  'application not enabled'

qa/tests: whitelisted  'application not enabled'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

13.2.6

Merge pull request #28356 from yuriw/wip-yuriw-whitelist-mimic

qa/tests: whitelisted POOL_APP_NOT_ENABLED

qa/tests: whitelisted POOL_APP_NOT_ENABLED

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28301 from yuriw/wip-yuriw-exl-python-mimic

qa/tests: added 'python3-cephfs','python3-rados' to excluded packadges

qa/tests: added 'python3-cephfs','python3-rados' to excluded packadges

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28129 from dillaman/wip-39728-mimic

mimic: qa/workunits/rbd: use https protocol for devstack git operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28184 from yuriw/wip-yuriw-fix-distro-mimic

qa/tests: cleaned up supported distro

qa/tests: cleaned up supported distro

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/workunits/rbd: use https protocol for devstack git operations

Fixes: http://tracker.ceph.com/issues/39656
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fb4f9a8a08edd18b1a23e1be4c1285d0ec4d1de6)

Merge pull request #28096 from ivancich/mimic-wip-rgw-admin-unordered

mimic: rgw: allow radosgw-admin to list bucket w --allow-unordered

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #26762 from pdvian/wip-38530-mimic

mimic: rgw: data sync drains lease stack on lease failure

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27973 from iain-buclaw-sociomantic/mimic-rgw-cls-bi-list-log-level

mimic: cls/rgw: raise debug level of bi_log_iterate_entries output

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28086 from cbodley/wip-39411

mimic: rgw: cls_bucket_list_unordered lists a single shard

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: allow radosgw-admin to list bucket w --allow-unordered

Presently the `radosgw-admin bucket list --bucket=<bucket>` lists the
objects in lexical order. This can be an expensive operation since
objects are not stored in bucket index shards in order and a selection
sort process is done across all bucket index shards.

By allowing the user to add the "--allow-unordered" command-line flag,
a more efficient bucket listing is enabled. This is particularly
important for buckets with a large number of objects.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 12452c4a91ee56c10d3dcf970dd2b0dd5aeb2401)

rgw: cls_bucket_list_unordered lists a single shard

CLSRGWIssueBucketList sends the request to every shard, but this loop
intended to list only the current_shard

Fixes: http://tracker.ceph.com/issues/39393
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d37d0339ff61a293f2f9fd6dff3fbd630efce2a0)

Conflicts:
src/rgw/rgw_rados.cc: remove unnessary "struct"s

cls/rgw: expose cls_rgw_bucket_list_op for single shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cd1fc96c5ca5254eb8343509b11a59514d62e532)

Conflicts:
src/cls/rgw/cls_rgw_client.cc: remove unnessary "struct"s

cls/rgw: raise debug level of bi_log_iterate_entries output

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9c8a207cef8792fc5a467b27b500153eef455c04)

Merge pull request #27432 from smithfarm/wip-38540-mimic

mimic: qa: fsstress with valgrind may timeout

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27847 from ashishkumsingh/wip-39469-mimic

mimic : s: better output of 'ceph health detail'

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27906 from smithfarm/wip-38736-mimic

mimic: qa: [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN) in cluster log

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27916 from pdvian/wip-39193-mimic

mimic: mds: drop reconnect message from non-existent session

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27917 from pdvian/wip-39200-mimic

mimic: mds/server: check directory split after rename.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28014 from cbodley/wip-39614

mimic: rgw: use chunked encoding to get partial results out faster

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27938 from pdvian/wip-39206-mimic

mimic: osd: shutdown recovery_request_timer earlier

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27963 from xiexingguo/wip-mimic-upmap-fixes

mimic: crush: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #27940 from pdvian/wip-39220-mimic

mimic: osd: FAILED ceph_assert(attrs || !pg_log.get_missing().is_missing(soid) || (it_objects != pg_log.get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) in PrimaryLogPG::get_object_context()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27055 from ifed01/wip-ifed-fix-storetest-mimic

mimic: test/store_test: fix/workaround for BlobReuseOnOverwriteUT and garbageCollection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27907 from smithfarm/wip-38443-mimic

mimic: tests: osd-markdown.sh can fail with CLI_DUP_COMMAND=1

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27943 from smithfarm/wip-38879-mimic

mimic: core: ENOENT in collection_move_rename on EC backfill target

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: use chunked encoding to get partial results out faster

Some operations can take a long time to have their complete result.

If a RGW operation does not set a content-length header, the RGW
frontends (CivetWeb, Beast) buffer the entire request so that a
Content-Length header can be sent.

If the RGW operation takes long enough, the buffering time may exceed
keepalive values, and because no bytes have been sent in the connection,
the connection will be reset.

If a HTTP response header contains neither Content-Length or chunked
Transfer-Encoding, HTTP keep-alive is not possible.

To fix the issue within these requirements, use chunked
Transfer-Encoding for the following operations:

* RGWCopyObj_ObjStore_S3 **
* RGWDeleteMultiObj_ObjStore_S3 **
* RGWGetUsage_ObjStore_S3
* RGWListBucketMultiparts_ObjStore_S3
* RGWListBucket_ObjStore_S3
* RGWListBuckets_ObjStore_S3
* RGWListMultipart_ObjStore_S3

RGWCopyObj & RGWDeleteMultiObj specifically use send_partial_response
for long-running operations, and are the most impacted by this issue,
esp. for large inputs. RGWCopyObj attempts to send a Progress header
during the copy, but it's not actually passed on to the client until the
end of the copy, because it's buffered by the RGW frontends!

The HTTP/1.1 specification REQUIRES chunked encoding to be supported,
and the specification does NOT require "chunked" to be included in the
"TE" request header.

This patch has one side-effect: this causes many more small IP packets.
When combined with high-latency links this can increase the apparent
deletion time due to round trips and TCP slow start. Future improvements
to the RGW frontends are possible in two seperate but related ways:
- The FE could continue to push more chunks without waiting for the ACK
  on the previous chunk, esp. while under the TCP window size.
- The FE could be patched for different buffer flushing behaviors, as
  that behavior is presently unclear (packets of 200-500 bytes seen).

Performance results:
- Bucket with 5M objects, index sharded 32 ways.
- Index on SSD 3x replicas, Data on spinning disk, 5:2
- Multi-delete of 1000 keys, with a common prefix.
- Cache of index primed by listing the common prefix immediately before
  deletion.
- Timing data captured at the RGW.
- Timing t0 is the TCP ACK sent by the RGW at the end of the response
  body.
- Client is ~75ms away from RGW.
BEFORE:
Time to first byte of response header: 11.3 seconds.
Entire operation: 11.5 seconds.
Response packets: 17
AFTER:
Time to first byte of response header: 3.5ms
Entire operation: 16.36 seconds
Response packets: 206

Backport: mimic, luminous
Issue: http://tracker.ceph.com/issues/12713
Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>
(cherry picked from commit d22c1f96707ba9ae84578932bd4d741f6c101a54)

Merge pull request #27972 from smithfarm/wip-37348-mimic

mimic: rgw: when using nfs-ganesha to upload file, rgw es sync module get failed

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27948 from smithfarm/wip-37498-mimic

mimic: rgw: get or set realm zonegroup zone should check user's caps for security

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27356 from pdvian/wip-38922-mimic

mimic: rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27367 from pdvian/wip-38924-mimic

mimic: rgw: Adding tcp_nodelay option to Beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27421 from pdvian/wip-38959-mimic

mimic: rgw-admin: fix data sync report for master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27603 from pdvian/wip-39017-mimic

mimic: rgw admin: add tenant argument to reshard cancel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27661 from pdvian/wip-39049-mimic

mimic: rgw: beast: set a default port for endpoints

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27666 from smithfarm/wip-39359-mimic

mimic: rgw: failed to pass test_bucket_create_naming_bad_punctuation in s3test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27796 from cbodley/wip-38713

mimic: rgw: resolve bugs and clean up garbage collection code

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27828 from ashishkumsingh/wip-39498-mimic

mimic: rgw: admin: handle delete_at attr in object stat output

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26140 from ashishkumsingh/wip-37909-mimic

mimic: rbd_mirror: don't report error if image replay canceled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27391 from pdvian/wip-38955-mimic

mimic: tests: krbd discard qa fixes

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27588 from pdvian/wip-38976-mimic

mimic: rbd: krbd: return -ETIMEDOUT in polling

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #27958 from dillaman/wip-39585-mimic

mimic: qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #27959 from dillaman/wip-39583-mimic

mimic: qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #26655 from jan--f/c-v-simple-activate-all-mimic

mimic: ceph-volume: add --all flag to simple activate

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: ES sync: wrap all the decode bls in try block

since decode can throw, wrap all the try block

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 12b12ccea23871688cc4101c72f00b0575f1c01a)

Conflicts:
src/rgw/rgw_sync_module_es.cc
- mimic uses val.begin() where master has val.cbegin()

osd/OSDMap: add log for better debugging

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a89281ffbb50a4dfc700398e864138b5faaf00f5)

osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 01e8e9482ce7194d347e02ef41acfa6d8d14f614)

osd/OSDMap: drop local pool filter in calc_pg_upmaps

The local pre-loaded pool filter is completely redundant since
the below check:

if (!only_pools.empty() && !only_pools.count(i.first))

could reliably catch both cases - either optimization should be
restricted to specific pools feeded, or all existing pools.

Let's clean it up.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3e6bad9821b5fb3e780d970666fbdfbf217d905e)

crush: fix upmap overkill

It appears that OSDMap::maybe_remove_pg_upmaps's sanity checks
are overzealous. With some customized crush rules it is possible
for osdmaptool to generate valid upmaps, but maybe_remove_pg_upmaps
will cancel them.

Fixes: http://tracker.ceph.com/issues/37968
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 5c4d241c7f796cb685e9944bf237028162122725)

Conflicts:
- maybe_remove_pg_upmaps input changing
        - slight c++11 auto conflicts

osd/OSDMap: using std::vector::reserve to reduce memory reallocation

In C++ vectors are dynamic arrays.
Vectors are assigned memory in blocks of contiguous locations.
When the memory allocated for the vector falls short of storing
new elements, a new memory block is allocated to vector and all
elements are copied from the old location to the new location.
This reallocation of elements helps vectors to grow when required.
However, it is a costly operation and time complexity is involved
in this step is linear.
Try to use std::vector::reserve whenever possible if performance
matters.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4a0eabb3a65107cbee5e692ade564102e2b2f8aa)

osd/OSDMap: more improvements to upmap

- add ability of appending a 2nd, 3rd, etc... pair to existing upmaps
  when possible, rather than just continuing to the next PG
- handle the underfull case: we can rm-pg-upmap-items if there exist
  any upmaps which remapped a PG out from an underfull OSD

See-also: http://tracker.ceph.com/issues/37940
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a7d2adf4283cc11ed567b0ad07c4076a50d7d2a0)

Conflicts:
- various option changes since luminous
- s/assert/ceph_assert/
        - s/conf./conf->/

Merge pull request #26194 from pdvian/wip-38078-mimic

mimic: multisite: bucket full sync does not handle delete markers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26553 from pdvian/wip-38396-mimic

mimic: rgw: when exclusive lock fails due existing lock, log add'l info

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #26615 from pdvian/wip-38413-mimic

mimic: multisite: rgw_data_sync_status json decode failure breaks automated datalog trimming

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27122 from smithfarm/wip-38886-mimic

mimic: rgw: GetBucketCORS API returns Not Found error code when CORS configuration does not exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27304 from pdvian/wip-38907-mimic

mimic: rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27335 from liaoxin01/mimic

mimic: rgw: fix bug of apply default quota, for this create new a user may core using beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27354 from pdvian/wip-38919-mimic

mimic: rgw: don't crash on missing /etc/mime.types

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26792 from cbodley/wip-38609

mimic: qa/rgw: reduce number of multisite log shards

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #26670 from tspmelo/wip-npm-update-mimic

mimic: mgr/dashboard: Update npm packages

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #27638 from tchaikov/mimic-39132

mimic: mgr: enable inter-module calls

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Fixes: http://tracker.ceph.com/issues/24668
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b184acd218b36b85f5a25678d773c35407a2b745)

qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 83c11efb98f28c19f70eac04a6a84fe820f968cb)

qa/rgw: add credentials to zone 'remove' in test_zonegroup_remove()

zone credentials are required to 'period update --commit' from
--rgw-zone remove

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Conflicts:
src/test/rgw/rgw_multi/tests.py
(cherry picked from commit 56021eab9584f10fa43053621cce93b623c36265)

rgw: get or set realm zonegroup zone need check user's caps

fix: https://tracker.ceph.com/issues/37352

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 6ecaec926fb81810f6be43744cd5c48d6ccfaf5a)

Conflicts:
src/rgw/rgw_rest_config.h
src/rgw/rgw_rest_realm.cc
- mimic lacks "override" in some places

osd/PG: !transaction_applied is true for async_recovery_targets as well

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 71996da6be171cd310f8cae48c95bedb02f000b8)

osd/PG: change error to dout in _scan_rollback_obs()

This message is now expected during scrubbing, since we do not delete
objects during a rollforward when !transaction_applied.

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit b8e62e6ee235e43e9930faf486ca8bea985c25b9)

osd/PG: skip rollforward when !transaction_applied during append_log()

Earlier, we did pg_log.roll_forward(&handler), when
!transaction_applied, which advanced the crt and trimmed the entries
in rollforward(). Due to this, during _merge_object_divergent_entries(),
when we tried to rollback entries, those objects were not found in the
backend, and thus we hit this bug http://tracker.ceph.com/issues/36739.

With this change, we are advancing the crt value, without deleting the
objects, so that _merge_object_divergent_entries() does not fail
because of deleted objects.

Fixes: http://tracker.ceph.com/issues/36739
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 17419ee39342257f5a6f41c792e8e45a8e243720)

osd/PG: move down peers out from peer_purged

In purge_strays(), we'll aggressively clear stray_set and
add all related peers into peer_purged.

However, if the corrsponding peer is down and becomes
up again, (unconditionally) adding it to peer_purged
will prevent primary from re-purging it.
(See Active::react(const MNotifyRec& notevt))

On consuming a new osdmap, let's move any down peers out from
peer_purged simutaneously. This way we can lower the risk
of leaving any leftover PGs behind.

Related-to: http://tracker.ceph.com/issues/38931
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit f7c5b01e181630bb15e8b923b0334eb6adfdf50a)

osd/PG: introduce all_missing_unfound helper

We use pg_log.missing to track each peer's missing objects separately,
whereas missing_loc records the location of all (probably existing) good copies
for both primary and replicas' missing objects. Hence an item from
pg_log.missing or missing_loc is of different meaning and is not comparable.

During recovery, we can skip recovering primary only if
- primary is good, e.g., has no missing at all
- or all of the primary's missing objects do exist in missing_loc and are
  currently unfound

Obviously, the current "all missing objects are unfound" checker is broken.
Fix by introducing an independent all_missing_unfound helper to make the
count of missing objects that are currently unfound correct.

Fixes: http://tracker.ceph.com/issues/38784
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit d9497139a6f516ce015bb43a2e7f8958638cf8f8)

osd: shutdown recovery_request_timer earlier

recovery_request_timer may hold some QueuePeeringEvts which PGRef,
if we dont shutdown it earlier, it potentially cause the PGRef leak
when kicking pg.

Fixes: https://tracker.ceph.com/issues/38945
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
(cherry picked from commit bb41bdfdb14768b5d46b7b1334bba1a3909e2618)

Conflicts:
src/osd/OSD.cc : Resolved in shutdown and start_shutdown

Merge pull request #27451 from badone/wip-mimic-spdk-rte_table_hash_cuckoo_create-build-error

mimic: spdk: update to latest spdk-18.05 branch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26495 from pdvian/wip-38337-mimic

mimic: test: run-standalone.sh set local library location so mgr can find li…

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #26777 from pdvian/wip-38532-mimic

mimic: mgr/PyModule: put mgr_module_path first in sys.path

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #26811 from ashishkumsingh/wip-38565-mimic

mimic: run-standalone.sh: Need double-quotes to handle | in core_pattern on all distributions

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27860 from ashishkumsingh/wip-39464-mimic

mimic: msg: output peer address when detecting bad CRCs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #26709 from pdvian/wip-38507-mimic

mimic: osd/PrimaryLogPG: handle object !exists in handle_watch_timeout

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26760 from pdvian/wip-38511-mimic

mimic: CLI: ability to change file ownership

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26810 from ashishkumsingh/wip-38587-mimic

mimic: common/str_map: fix trim() on empty string

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26833 from pdvian/wip-38561-mimic

mimic: mgr/BaseMgrModule: drop GIL for ceph_send_command

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26862 from ashishkumsingh/wip-38646-mimic

mimic: osd: fixup OpTracker destruct assert, waiting_for_osdmap take ref      with OpRequest

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mds/server: check directory split after rename.
fixes: http://tracker.ceph.com/issues/38994
Signed-off-by: Shen Hang <harryshen18@gmail.com>
(cherry picked from commit 3e82fc51db3d222af009c7e2e70c284bb9d37a85)

Merge pull request #27078 from yuriw/wip-yuriw-clients-mimic_2

qa/tests: initial checkin for `client-upgrade-mimic` suite

mds: drop reconnect message from non-existent session

fixes:http://tracker.ceph.com/issues/39026

Signed-off-by: Shen Hang <harryshen18@gmail.com>
(cherry picked from commit 017cd2a663e71039c54c57d70fec722c0c4aba81)

Conflicts:
src/mds/Server.cc : Resolved in handle_client_reconnect

qa/tests: removed conflicting tests rbd_api_tests and rbd_import_export
and added devstack-tempest-gate.yaml instead

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27447 from ifed01/wip-ifed-fix-39144

mimic: os/bluestore_tool: bluefs-bdev-expand: indicate bypassed for main dev

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27558 from tspmelo/wip-26986-mimic

mimic: mgr/dashboard: Use human readable units on the OSD I/O graphs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27587 from pdvian/wip-38972-mimic

mimic: osd: process_copy_chunk remove obc ref before pg unlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27366 from ifed01/wip-ifed-fix-alloc-overflow-mimic

mimic: os/bluestore: fix length overflow.

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27570 from ifed01/wip-ifed-fix-21312-mimic

mimic: os/bluestore: call fault_range properly prior to looking for blob to …

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27629 from pdvian/wip-39044-mimic

mimic: osd/PGLog: preserve original_crt to check rollbackability

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

qa/standalone/osd/osd-markdown: fix dup command disabling

The ceph cli tool checks for the presence of the variable, not its value.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f7216d0b2c3a08d0f6e6106e838f84a7af29d396)

qa/standalone/osd/osd-markdown: disable CLI command dups

The markdown test is based on marking down a specific number of times, but
the duplicate commands from the CLI may not get absorbed/batched by the
mon, breaking the test.  Override the default qa/tasks/workunit.py
behavior of sending dups.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c6a7b2cbd1430849ee92c627ff870f181697282d)