common/ceph_context: avoid unnecessary wait during service thread shutdown

Fixes: https://tracker.ceph.com/issues/42332
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e8249d3b28f8789b2d4aca6fb75c75103a5cbea1)

Conflicts:
src/common/ceph_context.cc
- Mutex::Locker

Merge pull request #28585 from ukernel/mimic-40327

mimic: mds: change how mds revoke stale caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30841 from smithfarm/wip-42263-mimic

mimic: tests: do not take ceph.conf.template from ceph/teuthology.git

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30918 from smithfarm/wip-42122-mimic

mimic: cephfs: client: add procession of SEEK_HOLE and SEEK_DATA in lseek.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30979 from smithfarm/wip-41464-mimic

mimic: tools: ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31017 from smithfarm/wip-40896-mimic-revert

mimic: cephfs: Revert "ceph_volume_client: convert string to bytes object"

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29219 from smithfarm/wip-38875-mimic

mimic: mds: high debug logging with many subtrees is slow

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30932 from smithfarm/wip-42034-mimic

mimic: cephfs: client: EINVAL may be returned when offset is 0.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30933 from smithfarm/wip-42038-mimic

mimic: cephfs: client: _readdir_cache_cb() may use the readdir_cache already clear

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #31090 from smithfarm/wip-42416-mimic

mimic: doc/rbd: s/guess/xml/ for codeblock lexer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: s/guess/xml/ for codeblock lexer

this change silences the warning of

```
doc/rbd/qemu-rbd.rst:174: WARNING: Pygments lexer name 'guess' is not
known
```

see http://pygments.org/docs/lexers/, we should use "xml" for XML .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit df226da996e468d2707b08eb012d54b4e37ffdc6)

Merge pull request #30775 from smithfarm/wip-41979-mimic

mimic: rgw: fix list versions starts with version_id=null

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30868 from smithfarm/wip-41324-mimic

mimic: rgw: datalog/mdlog trim commands loop until done

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30980 from smithfarm/wip-41496-mimic

mimic: rgw: fix the bug of rgw not doing necessary checking to website configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30891 from smithfarm/wip-41715-mimic

mimic: rgw: fix refcount tags to match and update object's idtag

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30977 from theanalyst/wip-41570-mimic

mimic: rgw: asio: check the remote endpoint before processing requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/tasks/ceph.conf: do not warn on TOO_FEW_OSDS

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 0483c1c3e7ffdfa6a6f65c5ef000c45d2f096428)

Merge pull request #30713 from smithfarm/wip-40258-mimic

mimic: cmake: detect armv8 crc and crypto feature using CHECK_C_COMPILER_FLAG

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30893 from smithfarm/wip-41964-mimic

mimic: tools/rados: list objects in a pg

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30898 from smithfarm/wip-42128-mimic

mimic: osd/OSDMap: do not trust partially simplified pg_upmap_item

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30903 from smithfarm/wip-42154-mimic

mimic: mon/OSDMonitor: trim not-longer-exist failure reporters

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30924 from vumrao/wip-vumrao-42240

mimic: osd/PG: Add PG to large omap log message

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30846 from wido/mimic-42116

mimic: mgr/telemetry: Ignore crashes in report when module not enabled

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30895 from smithfarm/wip-42036-mimic

mimic: osd/PeeringState: recover_got - add special handler for empty log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30901 from smithfarm/wip-42137-mimic

mimic: osd: Remove unused osdmap flags full, nearfull from output

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30916 from smithfarm/wip-41457-mimic

mimic: osd: merge replica log on primary need according to replica log's crt

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30982 from tchaikov/wip-mimic-42362

mimic: build/ops: python3-cephfs should provide python36-cephfs

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30991 from smithfarm/wip-37520-mimic-revert

mimic: msg: Revert "msg/async: do not trigger RESETSESSION from connect fault during connection phase"

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert "ceph_volume_client: make UTF-8 encoding explicit"

This reverts commit ddb8cfa072cc19fb6cb61128b4fe4b8ffe1a1742.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "pybind/ceph_volume_client: allow volume_client to"

This reverts commit 1ab18fcf056ef8e4fcb9ed27e64ee4f8a75866dc.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "ceph_volume_client: convert string to bytes object"

This reverts commit e081e97190b58d3a09377275f6d484bf9020fe51.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "ceph_volume_client: don't convert None to str object"

This reverts commit e085982def42738d181240c7b6e97b30f6bc0cf8.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mds: check last laggy before marking unresponsive client stale

Current mds may evict unresponsive client without going through session
stale. So we need to adjust the last laggy check.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e5cc6f52feb64fa1c7ba1ee2b304cd6588f16e7a)

 Conflicts:
src/mds/Server.cc

mds: remove the code that skip evicting the only client

There is already logic that defer marking unresponsive client stale.
No reason to defer evicting the only stale client.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit cd29206974427a4f6ab410b1482bbd8ebfb55fbd)

 Conflicts:
qa/tasks/cephfs/test_misc.py

qa/cephfs: update tests for stale session handling

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 1c8be588e32f47ca712561711ad1ffdddc54b330)

 Conflicts:
qa/tasks/cephfs/test_client_recovery.py

mds: change how mds revoke stale caps

- Only revokes conflicting caps from stale client.
- If stale client holds conflicting CEPH_CAP_ANY_WR,
  blacklist and kill it.

Fixes: https://tracker.ceph.com/issues/38326
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit dcae1ea2d30398f7b6493a74b482e964a21fcfeb)

 Conflicts:
src/mds/CInode.cc
src/mds/Capability.cc
src/mds/Locker.cc
src/mds/MDSRank.h
src/mds/Server.cc

mds: don't mark unresponsive sessions holding no caps stale

When an unresponsive MDS session holds no caps, do not mark it stale
even after session_timeout; at session_autoclose, evict it directly.

Fixes: http://tracker.ceph.com/issues/17854
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 98af31d10f362c05ea8ed57495973b08599431e7)

 Conflicts:
src/mds/Server.cc

mds: optimize resuming stale caps

If client doesn't want any cap, there is no need to re-issue stale
caps.

A special case is that client wants some caps, but skipped updating
'wanted'. For this case, client needs to update 'wanted' when stale
session get renewed.

Fixes: http://tracker.ceph.com/issues/38043
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e824b3d2024db36789fdf579f0af9bf3bbe55d51)

 Conflicts:
src/client/Client.cc
src/mds/cephfs_features.h

client: avoid unnecessary wakeup when handling RENEWCAPS

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit c744bc1673ca0b4e50f41516fdf49c8560db073a)

client: don't wakeup cap waiters twice when mds recovered

Both kick_maxsize_requests() and wake_inode_waiters() wake up cap
waiters

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 5993a93e606d23ef311831863444294d7c64cc04)

 Conflicts:
src/client/Client.cc
src/client/Client.h

client: set cap->wanted when adding new cap

This avoids unnecessary cap message if cap is added by open/create
request reply.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 314660a46a8a1af97f70b2bac05b2f6fa5d23bc4)

 Conflicts:
src/client/Client.h

mds: optimize revoking stale caps

For caps that are not being revoked and don't have writeable range
and don't want exclusive caps or file read/write. there is no need
to call Locker::revoke_stale_caps(Capability*). Because these caps
don't need recover and don't affect eval_gather()/try_eval().

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit cb62030f0468fc04735c1b4cff73da779cb11ad8)

 Conflicts:
src/mds/CInode.cc
src/mds/Capability.h
src/mds/Locker.cc
src/mds/Migrator.cc

mds: put notable caps at the front of session's caps list

Notable Capabilities are ones that are being revoked, ones that
have writeable ranges and ones that want exclusive caps or want
file read/write.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit cb6e7184458e64720ad83c266e5e393a80c32697)

mds: track if client has writeable range in Capability

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 370ae1cb3e5dc07867d80e998082bc514e8fccfd)

 Conflicts:
src/mds/Locker.cc
src/mds/MDCache.h
src/mds/Server.cc

mds: add session pointer to Capability

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 7c628472a86c6acebb20f0a2504744b10f250587)

 Conflicts:
src/mds/CInode.cc
src/mds/Capability.h
src/mds/Locker.cc

client: sync 'retain caps' logical from kernel client

The main change is keeping CEPH_CAP_ANY_RD for unused file inodes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 29034396398b1706db625e1abf2e6682be7130a5)

client: skip updating 'wanted' caps if caps are already issued

When reading cached inode that already has Fscr caps, this can avoid
two cap messages (one updats 'wanted' caps, one clears 'wanted' caps).

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit d20b260ecf3f323c87ad1e6865f87e2381444546)

Merge pull request #30808 from jan--f/wip-42233-mimic

mimic: ceph-volume: VolumeGroups.filter shouldn't purge itself

Merge pull request #30936 from smithfarm/wip-42130-mimic

mimic: doc/ceph-fuse: mention -k option in ceph-fuse man page

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #30806 from jan--f/wip-42235-mimic

mimic: ceph-volume: PVolumes.filter shouldn't purge itself

Revert "msg/async: do not trigger RESETSESSION from connect fault during connection phase"

This reverts commit 00b163564c6cafd4edf54d470cc708eab9dae10e.

Xie Xingguo and Ricardo Dias looked at this, and both agreed that the bug
was caused with the reduction of states of the V1 protocol during its
refactoring. In other words, the bug is not present in mimic.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: provide python2-<modname>

to be consistent with other python2 packages, and their python3
counterparts

the `python_provide` macro is offered by `python-rpm-macros` package,
which is in turn required by python*-devel

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit fb6290b1fab0bd8a73043f2d68210c010e2fb425)

ceph.spec.in: use python_provide macro

our python3 bindings are now named `python3-<modname>` after python3 is
now maintained by RHEL/CentOS instead EPEL. to help the users using
`python36-<modname>`, we should "Provide" `python36-<modname>`.

the `python_provide` macro is offered by `python-rpm-macros` package,
which is in turn required by python*-devel. and we do install
`python36-devel` in install-deps.sh, and install `python3-devel` in
ceph-*build/build/setup_rpm

see also
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_provides

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 50b19e673d8200306c3e36d1abaec414a3d336b9)

rgw: fix checking index_doc_suffix when getting effective key

Currently, if the index_doc_suffix is empty which is leaded
by the IndexDocument field is not configurated or is set to
empty string during enabling bucket website function, the
rgw will crash when accessing the static website through the
S3Website enabled RGW instance.

In actually, we have add the necessary checking in the commit
355f392ad26631f44dac250296e96f421d86fb8f, but double checking
in here is better.

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>
(cherry picked from commit c96f415dafe176b1b8d10ff9456d13fb76c79baa)

Conflicts:
src/rgw/rgw_rest_s3.cc
- ldpp_dout

rgw: fix the bug of rgw not doing necessary checking to website configuration

Fixes: http://tracker.ceph.com/issues/40678
Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>
(cherry picked from commit 355f392ad26631f44dac250296e96f421d86fb8f)

Conflicts:
src/rgw/rgw_rest_s3.cc
- 3275dffa45ae08d3818562d2d11a8d1c0afa326b is not being backported

Merge pull request #29224 from smithfarm/wip-39223-mimic

mimic: mds: behind on trimming and [dentry] was purgeable but no longer is!

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29232 from smithfarm/wip-40439-mimic

mimic: mds: cannot switch mds state from standby-replay to active

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29479 from xiaoxichen/wip-41001

mimic: cephfs: client: unlink dentry for inode with llref=0

Reviewed-by: Venky Shankar <vshankar@redhat.com>

ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

there is chance that we could use an OSD which does not have incmap of a
certain epoch for rebuilding the monstore. and since OSD does not read
and store the incmap if the MOSDMap message already has the fullmap of
that fullmap, and if an OSD does not have previous fullmap, monitor
will just send it the fullmao. so it's not unusual that an OSD has
a fullmap of some epoch without corresponding incmap.

Fixes: https://tracker.ceph.com/issues/41177
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2e2414b3df97b22ccc54500830c24b28597ce75f)

Merge pull request #29218 from smithfarm/wip-38709-mimic

mimic: tests: kclient unmount hangs after file system goes down

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29220 from smithfarm/wip-39210-mimic

mimic: mds: mds_cap_revoke_eviction_timeout is not used to initialize Server::cap_revoke_eviction_timeout

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29222 from smithfarm/wip-39212-mimic

mimic: cephfs: MDSTableServer.cc: 83: FAILED assert(version == tid)

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #29223 from smithfarm/wip-39215-mimic

mimic: mds: there is an assertion when calling Beacon::shutdown()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29228 from smithfarm/wip-40219-mimic

mimic: tests: cephfs: TestMisc.test_evict_client fails

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29230 from smithfarm/wip-40437-mimic

mimic: cephfs: getattr on snap inode stuck

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30796 from dillaman/wip-36122-mimic

mimic: librbd: properly handle potential object map failures

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30828 from dillaman/wip-41882-mimic

mimic: rbd-mirror: cannot restore deferred deletion mirrored images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw: asio: check the remote endpoint before processing requests

`socket.remote_endpoint()` can throw exceptions corresponding to errors in the
`getpeername` syscall, make sure these are handled.

Fixes: CVE-2019-10222, https://tracker.ceph.com/issues/40018
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit caa653196856ecdf50519a9a33195d5c4e3372af)

 Conflicts:
src/rgw/rgw_asio_frontend.cc
conflicts due to optional-yield-ctx changes in master

Merge pull request #30213 from smithfarm/wip-41449-mimic

mimic: core: mon: C_AckMarkedDown has not handled the Callback Arguments

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30150 from neha-ojha/wip-40769-mimic

mimic: bluestore: common/options: Set concurrent bluestore rocksdb compactions to 2

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: check earlier if directories are already exported

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 46fb90734f371b5467d81cae25c06f8a487a3041)

Conflicts:
    src/mds/MDCache.cc
- g_conf-> instead of g_conf()->

mds: dont print auth trees if they are too many

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 13c4152c8b1756a871f817d4b503396b6c9cc81d)

Conflicts:
src/mds/MDBalancer.cc
- g_conf
- authsubs in method MDBalancer::handle_export_pins() is redeclared from set
  to vector in master branch. Removed vector declaration as per the luminous
  backport.

mds: dont print subtrees if they are too many/big

Also, add an argument to force print subtrees and let
MDBalancer::tick() always print subtrees.

Fixes: https://tracker.ceph.com/issues/37726
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 50d28ec4f842670fb96ab3b0d2b37a9c2e282236)

Conflicts:
    src/mds/MDCache.cc
- g_conf

Merge pull request #30069 from smithfarm/wip-40124-mimic

mimic: qa/rgw: don't use ceph-ansible in s3a-hadoop suite

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30133 from smithfarm/wip-40850-mimic

mimic: rgw/multisite: Don't allow certain radosgw-admin commands to run on non-master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29203 from smithfarm/wip-40320-mimic

mimic: tests: make: *** [hello_world_cpp] Error 127 in rados

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30219 from vumrao/wip-vumrao-bluefs-shared-alloc-with-log-level-change-mimic

mimic: os/bluestore: apply shared_alloc_size to shared device with log level change

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30260 from smithfarm/wip-41723-mimic

mimic: build/ops: fix build fail related to PYTHON_EXECUTABLE variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30355 from pdvian/wip-41765-mimic

mimic: build/ops: ceph.spec.in: reserve 2500MB per build job

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30672 from smithfarm/wip-37520-mimic

mimic: msg/async: do not trigger RESETSESSION from connect fault during connection phase

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30784 from smithfarm/wip-41918-mimic

mimic: core: osd: scrub error on big objects; make bluestore refuse to start on big objects

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/ceph-fuse: mention -k option in ceph-fuse man page

Fixes: https://tracker.ceph.com/issues/42044
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit a96a32bd2ef15e963bdb8c46e45edcb83e2361bb)

client: _readdir_cache_cb() may use the readdir_cache already clear

Calling function _readdir_cache_cb() means to get dir information from the cache, but in the while loop,
the contents of readdir_cache are not static.

For example, _getattr() will trigger insert_trace to update readdir_cache, such as:
1. insert_trace()->clear_dir_complete_and_ordered() or
2. insert_trace()->add_update_inode()->add_update_cap()-> check_cap_issue()->clear_dir_complete_and_ordered()
So, there is a possibility that dn is empty after the _getattr() call.

Fixes: http://tracker.ceph.com/issues/41148
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>
(cherry picked from commit 404c06784e46016703104db59ea73f36d09b578b)

client:EINVAL may be returned when offset is 0 ,Loff_t pos = f->pos should be the best.
Fixes:https://tracker.ceph.com/issues/41837
Signed-off-by: wenpengLi <liwenpeng@inspur.com>
(cherry picked from commit 0572e54318cebdfa83d67d47be9a6d110489e808)

tools/rados: call pool_lookup() after rados is connected
This commit fixes a segmentation fault when using --pgid
option in rados ls command in combination with --pool/-p option.
The reason for the crash was that we can not use the rados object
before connecting it with the cluster using rados.init_with_context().

Fixes: https://tracker.ceph.com/issues/41875
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit 56b6dd9bd7a50371062128122e6a899f2c79c5f5)

Conflicts:
src/tools/rados/rados.cc

tools/rados: always call rados.shutdown() before exit()

When connected to ceph cluster, if call exit(1) directly, will cause the finisher
thread segmentation fault as follows,

Caught signal (Segmentation fault) **in thread 7f45377fe700 thread_name:fn_anonymous

run 'rados mapext' without other arguments could easily get it. This patch fixes it
by calling rados.shutdown() before exit()

Fixes: http://tracker.ceph.com/issues/36732
Signed-off-by: Li Wang <laurence.liwang@gmail.com>
(cherry picked from commit 2c149262888c50beb9f480a6cd78f77fd5920d1c)

Conflicts:
src/tools/rados/rados.cc
- revert "std::make_unique" change that, in master, should have been included in
  3804f52b109558591d699609749a789b5cfc72b9 (which is not being backported)

tools/rados: allow list objects in a pg

When the cluster is not healthy, normally only a very few pgs encounter
problems, and the operator wants to know which objects may potentially
get lost.

Signed-off-by: Li Wang <laurence.liwang@gmail.com>
(cherry picked from commit 825c7b9729b2410a116e123d4b21210c646704d4)

Conflicts:
src/tools/rados/rados.cc
- 6e4db7b0a41f8d56c8725ea27352e85a13273888 is not being backported to mimic

tools/rados: add --pgid in help

This commit 825c7b9729b2410a116e123d4b21210c646704d4 brought
the support for --pgid in rados ls but was not in help.
Fixes: https://tracker.ceph.com/issues/41876
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit ae9a24eaba330fd3f6946063c2bc926f58c93772)

mon/OSDMonitor: trim no-longer-exist failure reporters

remove the report from failure_info in OSDMonitor::check_failure(), if
the reporter does not exist in the osdmap any longer. otherwise, we will
run into assert() failure when trying to lookup the osd in osdmap using
`osdmap.get_xinfo()`.

Fixes: https://tracker.ceph.com/issues/41680
Signed-off-by: NancySu05 <su_nan@inspur.com>
(cherry picked from commit 99ccc166ede5f296b86abfe9db0a3d597100edde)

Conflicts:
src/mon/OSDMonitor.cc
- ceph_assert, g_conf()

osd/PG: Add PG to large omap log message

Fixes: https://tracker.ceph.com/issues/41359
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit a71d51bfa154a278a350e6dbeb842fb3585b74f5)

client: add procession of SEEK_HOLE and SEEK_DATA in lseek.

Fixes: https://tracker.ceph.com/issues/42107
Signed-off-by: Shen Hang <harryshen18@gmail.com>
(cherry picked from commit 394720ca6b731e4698f1dbc7896d294363ecb466)

osd: merge replica log on primary need according to replica log's crt

Fixes: https://tracker.ceph.com/issues/41194
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
(cherry picked from commit ac387844e12edc9a1f481bdd1ba307ac4dc60e82)

Conflicts:
src/osd/PGLog.h
- ceph_assert

osd: Remove unused osdmap flags full, nearfull from output

Fixes: https://tracker.ceph.com/issues/42015
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d9c645d7d8c7355c8aaf3ff9121f2b8fc3b1290a)

osd/OSDMap: do not trust partially simplified pg_upmap_item

If we simplified a partially no-op pg_upmap_item, we shall still
continue to verify that the remaining part is valid.
The bug is introduced by 02e5499b350bcd7d9eac98b2072052a9a4a1f535,
before which we always validate the correctness of a pg_upmap_item
before trying to cancel or simplify it.

Fixes: https://tracker.ceph.com/issues/42052
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4196b13283144de966eeba40e6765f10b254dac6)

osd/PeeringState: recover_got - add special handler for empty log

If we use a small osd_min[max]_pg_log_entries setting, say, 10, and
do splitting, a child pg may only be able to keep some divergent log
entries in the list. Any follow-up peering cycle on that child pg
would then rewind all the divergent log entries and lead to an empty
log list, which as a result could reliably fire the

  ceph_assert(info.last_complete == info.last_update)

assert if we start to recover any of the missing divergent objects
later.

Fixes: https://tracker.ceph.com/issues/41816
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a5e9e9099865be7a3d90c58cf3f3fc50ed6eed6d)

Conflicts:
src/osd/PeeringState.cc
- file does not exist in the stable branch: made the changes manually to
  src/osd/PrimaryLogPG.cc

rgw: make rollback refcount tag match

When the refcount does a "get", the tag ends with a null character,
but during a rollback, when the tag is "put" it does not end with a
null character, so the same tag is not being manipulated and the
refcounts are therefore off, preventing future deletion of an rgw
object. This adds the null character to the "put" operation.

This change was originally submitted by lltlo <120680451@qq.com> but
wasn't signed-off.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit a0fee94982d89f0210e1c526605831105cd90ef4)

Conflicts:
src/rgw/rgw_rados.cc

rgw: make sure object's idtag is updated when available

This change was originally submitted by lltlo <120680451@qq.com> but
had some formatting issues and wasn't signed-off.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 7d32b082f826181b605ae02ebc914fba9ac0d449)

Conflicts:
src/rgw/rgw_rados.cc

Merge pull request #29258 from smithfarm/wip-39741-mimic

mimic: rgw: swift object expiry fails when a bucket reshards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29483 from pdvian/wip-40761-mimic

mimic: rgw: Save an unnecessary copy of RGWEnv

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29500 from pdvian/wip-40847-mimic

mimic: rgw: Don't crash on copy when metadata directive not supplied

Reviewed-by: Casey Bodley <cbodley@redhat.com>