client: fix issue of revoking non-auth caps

when non-auth mds revokes caps, Fcb caps can still be issued by auth
auth mds. It's wrong to flush buffer or invalidate cache when non-auth
mds revokes other caps. This bug can cause client to not respond the
revoke.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24172
(cherry picked from commit 341a9114e0726e1a7cbb7e6f22adb54c2024c506)

Merge pull request #22076 from tchaikov/wip-cmake-build-rocksdb-no-Werror

luminous: cmake: disable FAIL_ON_WARNINGS for rocksdb

Reviewed-by: Nathan Cutler <cutler@suse.cz>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22197 from dzafman/wip-test-fixes-luminous

luminous: test fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: wait_for_pg_stats() should do another check after last 13 second sleep

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 151de1797b9163918b95a5996f422688e0964126)

os/bluestore: fix data read error injection in bluestore

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit be32d15a04d9d900f604aa366e82791249f1bdb2)

Merge pull request #21502 from smithfarm/wip-23782-luminous

luminous: table of contents doesn't render for luminous/jewel docs

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test: Whitelist corrections

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ee4acb6e1ff7458ceaefdb288cbcb158c6a3bed3)

Add "erasure code profile property .ruleset-failure-domain. is no longer supported" for luminous

Merge pull request #22134 from dzafman/wip-missed-backport

test: Add CACHE_POOL_NO_HIT_SET to whitelist for mon/pool_ops.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Add CACHE_POOL_NO_HIT_SET to whitelist for mon/pool_ops.sh

Ignore
  cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4fad800043d44024a496f78869e9bb02a16af063)

Merge pull request #22044 from dzafman/wip-24045-luminous

luminous: osd: Don't evict even when preemption has restarted with smaller chunk

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22131 from ceph/wip-yuriw-clients-fix-luminous

qa/tests: added supported distro

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22128 from liewegas/wip-rbd-msgr-luminous

luminous: qa/suites/rbd/basic/msgr-failures: remove many.yaml

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd/basic/msgr-failures: remove many.yaml

Overkill, and triggers some failures, see
http://tracker.ceph.com/issues/23789

Removed in master by 4046f46d0e6a70d860d74945dfb95c2511394640

Fixes: http://tracker.ceph.com/issues/23789
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21547 from VictorDenisov/backport

luminous: tests: filestore journal replay does not guard omap operations

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21515 from tchaikov/wip-luminous-pr-21469

luminous: mon/LogMonitor: do not crash on log sub w/ no messages

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21376 from pdvian/wip-23666-luminous

luminous: msg/async/AsyncConnection: Fix FPE in process_connection

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21405 from pdvian/wip-23672-luminous

luminous: os/bluestore: alter the allow_eio policy regarding kernel's error list.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21407 from tchaikov/wip-luminous-23246

luminous: os/bluestore: fix exceeding the max IO queue depth in KernelDevice.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #21514 from smithfarm/wip-posix-zfs-luminous

luminous: common: posix_fallocate on ZFS returns EINVAL

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #21818 from xiexingguo/wip-23925

luminous: osd/OSDMap: check against cluster topology changing before applying pg upmaps

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: added supported distro

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21575 from ceph/wip-cd-fix-pool-create

luminous: tests: ceph-deploy: create the rbd pool right after install

test: Fix omap_digest changes in osd-scrub-repair.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test: No more omap_digest being set

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Luminous specifc changes

*** Not sure why this wasn't seen earlier

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Need to escape parens in log-whitelist for grep

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a9e43ed85236c8412679da58d068253e80d21d05)

Conflicts:
qa/suites/rados/monthrash/ceph.yaml (no changes needed)

Additional changes for luminous:
qa/suites/rados/basic/tasks/rados_api_tests.yaml
qa/suites/rados/singleton/all/thrash-eio.yaml
qa/suites/smoke/basic/tasks/rados_api_tests.yaml

osd: Clear part of cleaned_meta_map in case of a restarted smaller chunk

This can not happen at the primary because scrub_compare_maps() is only
called once per chunk start.

Preemption causes a smaller chunk from start to be processed again at
replicas.  We clear any of the previous chunk's information.

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9e0ac797c602a088447679b04e14ec0cfaf9dd7b)

osd: Don't evict even when preemption has restarted with smaller chunk

Fixes: https://tracker.ceph.com/issues/24045
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 818b59fa95ee60e86991276f18c4dee405dc79b1)

Conflicts:
src/osd/PG.h (trivial)

osd/PrimaryLogPG: defer evict if head *or* object intersect scrub interval

Consider a scenario like:
- scrub [3:2525d100:::earlier:head,3:2525d12f:::foo:200]
 - we see 3:2525d12f:::foo:100 and include it in scrub map
- scrub [3:2525d12f:::foo:200, 3:2525dfff:::later:head]
- some op(s) that cause scrub to be preempted
- agent_work wants to evict 3:2525d12f:::foo:100
  - write_blocked_by_scrub sees scrub is preempted, returns false
  - 3:2525d12f:::foo:100 is removed, :head SnapSet is updated
- scrub rescrubs [3:2525d12f:::foo:200, 3:2525dfff:::later:head]
  - includes (updated) :head SnapSet
  - issues error like "3:2525d12f:::foo:100 is an unexpected clone"

Fix the problem by checking if anything part of the object-to-evict and
its head touch the scrub range; if so, back off.  Do not let eviction
preempt scrub; we can come back and do it later.

Fixes: http://tracker.ceph.com/issues/23646
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c20a95b0b9f4082dcebb339135683b91fe39ec0a)

osd: If ending on a head object get all of meta map

When ending on a head object, the head and snapshots would stay in
cleaned_meta_map until more maps arrive.  The problem as that
during a scrub an eviction could occur because scrubber.start
is already past the stray object(s) so range_intersects_scrub() is false.

Fixes: http://tracker.ceph.com/issues/23909
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 83861a5b75ddb98366f1ec106487b88703f25cf7)

test: Add test cases for multiple copy pool and snapshot errors

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 1a7fa9a62a62a35c645757287917101925044df1)

test: Fix comment at end of scrub test scripts

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bae4940574fa0ee267e40785c88ee6baa3fba96b)

test: Prepare for second test and minor improvements

Check list-inconsistent-obj output
Check how many _scan_snap groupings
Use more general check for crashed osd(s)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 2fa596dc0c515b757bce3bd3089a2ed32304d976)

osd: process _scan_snaps() with all snapshots with head

Fixes: http://tracker.ceph.com/issues/22881
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 8f0514bf59bad486df63d078b57df636eb969bc5)

Conflicts:
src/osd/PG.cc (trivial)

osd/PG: kill extra scrubber state transition

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 323dca0c82b710766ece06da8efe8d99cf3c07ab)

osd/PG: decay scrub_chunk_max too if scrub is preempted

In normal case we'll at least scrub as many objects as
osd_scrub_chunk_max specified at a time, so the current
backoff mechanism should have very limit effect.
Decay both osd_scrub_chunk_min and osd_scrub_chunk_max
should instead be a better resolution.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a9260524676ac28742e5a945de93b87ae985017e)

osd/ReplicatedBackend: turn more be_deep_scrub options into legacy

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 946b6dde76e513af3e28a8725c873c414f4ad40b)

osd/ReplicatedBackend: turn be_deep_scrub options into legacy

See 588f0643f12ac842ff68cacd4d10d57f9f3ed3fe.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 048e638e3335c32e767f0767c8aa64eedfb675db)

osd/ECBackend: inject sleep during deep scrub

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3cec7bfd819deea609a2996b2a9d118968fa6128)

osd/PG: pass scrub priority to replica

If we are scrubbing with high priority on the primary, pass that along
to the replica so that it can schedule its scrub work accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d9fd07696058cf79a62c327ecf08a5f8fb5b6a28)

- fixed encode vs ::encode conflict

osd/ReplicatedBackend: 'osd_deep_scrub_keys' doesn't work

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>
(cherry picked from commit ad6039bbab42137b748d2377fb402e31f4e0dcfe)

osd/osd_types.h: default to no data/omap digest for new object

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 5345afa5d5c984a173fcf6d5a4447b71eb864070)

osd/PG: drop 'seed' property from Scrubber

This has been -1 for many releases now.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2d34e380c8df465bfb3968fd10550285faa4a9b9)

Conflicts:
src/messages/MOSDRepScrub.h

- encode vs ::encode etc

qa/suites/rados/singleton/all/divergent_priors*: unsquelch osd debug

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5ac3bfa34c40d2beb79ad189e6a98033b981e75c)

osd/ECBackend: debug ec scrub error paths

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a188cb27dd0d458362181205915bf17df61595e6)

osd: document scrub options

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4e0f4238b92ac212f941592745978620a1967cd2)

osd: allow limited scrub preemption

If we receive a write within the scrub range, abort the scrub chunk and
shrink the chunk size.  If we do this too many times do not preempt and
allow the scrub to complete (to avoid scrub starvation due to client io).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6dd42392c0f00011059ffa5de74cace7d1e911bd)

Conflicts:
src/messages/MOSDRepScrub.h
src/messages/MOSDRepScrubMap.h
src/osd/PrimaryLogPG.cc
src/osd/PrimaryLogPG.h

- encode vs ::encode etc
- dragged in waiting for scrub events from 508ea640e3b
- ignore change in chunked manifest code (which dne in luminous)

osd: piecewise scrub

Perform scrub in stages, with each unit of work requeuing an item in the
work queue.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf16f59887d6b7624112212cecead3ebec48b6f9)

Conflicts:
src/osd/PG.cc
src/osd/ReplicatedBackend.cc

- encode -> ::encode

osd: flush before collection_list()

We would get this implicitly with FileStore if we waited for the onreadable
callbacks, but in some cases the OSD has already done that.  With BlueStore,
we need to explicitly flush().

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b877860e4246cdd21b5ee79f17756efcf71b311e)

osd/ECBackend: turn be_deep_scrub options into legacy

We don't have a lightweight mechanism for doing trivial config options
that is better than legacy_config_opts.h yet.  Until then,

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 588f0643f12ac842ff68cacd4d10d57f9f3ed3fe)

qa/tasks/ceph: disable osd_debug_deep_scrub_sleep in case it is set

Otherwise the final scrub may take too long.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3f922e79c3c39710f5fbabc0dacef5f4ab19885b)

osd/*Backend: debug: inject sleep during deep scrub

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3e66d88f308af2a9bd3410f7476af342acf48b91)

osd/PG: drop waiting_on, use waiting_on_whom

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 004ee202fac7a2f3fba2b018426474eeae7f913b)

Add changes to PG::sub_op_scrub_map() which exists in Luminous

osd/PrimaryLogPG: do not generate data digest for BlueStore by default

BlueStore enables CRC by default, so this is a dup and gains
no more benefits.

Turn this off by default, which is good for performance.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit afcb617dc9791aa6551a1856c89b3e8e2648eabc)

Conflicts:
../qa/standalone/scrub/osd-scrub-repair.sh (Modify json object info instead of string)

osd/PrimaryLogPG: add condition "is_chunky_scrub_active" to check object in chunky_scrub.

Avoid every time call scrubber.write_block_by_scrub. Most time scrubber
is inactive. And compare to write_block_by_scrub, is_chunky_scrub_active
is light.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit 6c81c9bb0979c101c112e8ccd45880e08bfdb945)

osd: add scrub week day constraint

if add week day constraint, we can set scrub permit time
more flexible. eg. we can set scurb in Monday-Wednesday 0-12 o'clock
according set this parameter:
osd_scrub_begin_week_day = 1
osd_scrub_end_week_day = 3
osd_scrub_begin_hour = 0
osd_scrub_end_hour = 12

Signed-off-by: kungf <yang.wang@easystack.cn>
(cherry picked from commit 87be7c70a17492c9e5f06e01722690acec7a2c51)

osd/PG: flush osr before listing scrub objects

We need to make sure previous transactions are readable before we list
objects.  This avoids races like, say,

- ecbackend rolls forward and removes a rollback object
- bluestore delays the txn apply due to some prior op
- scrub lists objects, sees rollback object it doesn't expect

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dca1257ab13e7871f25ef1c8179e45c199b252f7)

cmake: disable FAIL_ON_WARNINGS for rocksdb

otherwise -Werror=implicit-fallthrough will fail the build with GCC-7

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6559a850512ddefef384d2733823ac2cb8027e3a)

Conflicts:
cmake/modules/BuildRocksDB.cmake: we have not extract the
BuildRocksDB to this cmake module yet in luminous. so update
src/CMakeLists.txt instead.

Merge pull request #21705 from smithfarm/wip-23913-luminous

luminous: rbd-nbd can deadlock in logging thread

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21939 from pdvian/wip-23986-luminous

luminous: osdc/Objecter: fix recursive locking in _finish_command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21965 from pdvian/wip-24014-luminous

luminous: mgr/influx: Only split string on first occurence of dot (.)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #21378 from pdvian/wip-23668-luminous

luminous: mon: add  'ceph osd pool get erasure allow_ec_overwrites' command

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #21556 from tchaikov/wip-luminous-23763

luminous: test/librados: increase pgp_num along with pg_num

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21764 from pdvian/wip-23914-luminous

osd/PrimaryLogPG: avoid infinite loop when flush collides with write …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21964 from pdvian/wip-23988-luminous

luminous: osd/PG: fix DeferRecovery vs AllReplicasRecovered race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21993 from pdvian/wip-24015-luminous

luminous: osd/PG: fix uninit read in Incomplete::react(AdvMap&)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22012 from b-ranto/wip-bp-selinux

luminous: Backport recent SELinux changes

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21674 from pdvian/wip-23865-luminous

luminous: rgw: fix error handling for GET with ?torrent

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #19884 from pdvian/wip-22637-luminous

luminous: rgw:lc: RGWPutLC return ERR_MALFORMED_XML  when missing <Rule> tag in…

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #21382 from smithfarm/wip-23681-luminous

luminous: mg_read() call has wrong arguments

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21435 from pdvian/wip-23682-luminous

luminous: rgw: return EINVAL if max_keys can not convert correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21438 from pdvian/wip-23685-luminous

luminous: rgw_file: conditionally unlink handles when direct deleted

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21660 from pdvian/wip-23862-luminous

luminous: rgw: aws4 auth supports PutBucketRequestPayment

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #21661 from pdvian/wip-23861-luminous

luminous: rgw: admin rest api shouldn't return error when getting user's stats if

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21669 from oritwas/wip-rgw-luminous-22721

luminous: rgw: fix bi_list to reset is_truncated flag if it skips entires

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21675 from pdvian/wip-23866-luminous

luminous: rgw: move all pool creation into rgw_init_ioctx

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21735 from pdvian/wip-23868-luminous

luminous: rgw: do not reflect period if not current

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21736 from pdvian/wip-23869-luminous

luminous: rgw: es module: set compression type correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21758 from cbodley/wip-23864

luminous: rgw: add buffering filter to compression for fetch_remote_obj

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #21792 from cbodley/wip-23977

luminous: rgw: raise log level on coroutine shutdown errors

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #21908 from cbodley/wip-24060

luminous: radosgw-admin sync status improvements

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #21917 from smithfarm/wip-24063-luminous

luminous: rgw: Misnamed S3 operation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21741 from pdvian/wip-23902-luminous

luminous: [rbd-mirror] local tag predecessor mirror uuid is incorrectly replaced with remote

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21869 from pdvian/wip-23945-luminous

luminous: qa/workunits/rbd: potential race in mirror disconnect test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21944 from dillaman/wip-23809-luminous

luminous: qa/workunits/rbd/import_export: update unit formatting

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #21995 from tchaikov/luminous-24051

luminous: os/bluestore: fix the demotion in StupidAllocator::init_rm_free

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
ifed01

selinux: Allow ceph to execute ldconfig

The ceph-volume testing showed that the ceph daemons can run ldconfig in
a corner case when they are forbidden access to some files. This patch
allows ceph to execute ldconfig in Enforcing mode.

Fixes: https://tracker.ceph.com/issues/22302
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit fa5071b6d7182f54cd7b1ffe171a4b006f5255cb)

selinux: Allow ceph to block suspend

Ceph is now trying to block suspend in certain cases, we need to update
the policy accordingly.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit e3be372d854f9014ca711f14525ca9e510d34f1a)

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

* we should update the _size field when demotion kicks in, otherwise the
  interval_set::m and interval_set::_size won't be consistent.
* also rename post_process to claim, as it "steals" the leftover(s) of
chopped segment.

Fixes: https://tracker.ceph.com/issues/24051
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d43c8da83329dd743c635869b1443366d03839fd)

Merge pull request #21872 from tchaikov/wip-luminous-jdk

luminous: cmake: fix the cepfs java binding build on Bionic

Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>

osd/PG: fix uninit read in Incomplete::react(AdvMap&)

If a PG is incomplete when the pool is deleted we'll dereference invalid
iterators here.

Fixes: http://tracker.ceph.com/issues/23980
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3187b036d83adae3646d411e6705e056189f33bb)

radosgw-admin: add pretty 'bucket sync status' command

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 650aef59cfaf429951a82a9d65a37b8c4bd1b942)

Conflicts:
src/rgw/rgw_admin.cc (no readonly_ops_list, no std::string_view)

rgw: expose struct bucket_index_marker_info in header

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6274368d9cc6566e85b52e53d6949a7814a5ea27)

rgw: rgw_bucket_sync_status takes bucket info

rgw_bucket_sync_status() no longer reads the bucket instance info, and
instead requires the caller to pass it in

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3640ee844a9d77f0fa6a6849c4d8b51cc932f4e2)

radosgw-admin: rename 'bucket sync status' to 'bucket sync markers'

renames the command to 'bucket sync markers' because it only returns the
local sync status markers for use in multisite tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2df2b0665d12dd9e47bf7155a1c6335c2d1a3bf4)

Conflicts:
src/rgw/rgw_admin.cc (no readonly_ops_list)

rgw: translate the state in rgw_data_sync_marker

Signed-off-by: lvshanchun <lvshanchun@gmail.com>
(cherry picked from commit 7ddca6e50d85380505134481d37734b4f1433f1a)

doc: update radosgw-admin.rst and help.t about data sync status

Signed-off-by: lvshanchun <lvshanchun@gmail.com>
(cherry picked from commit 33a0ec7164eff9fb66f05d984eb92b0de4ced9b1)

rgw: add --shard-id for data sync status

when specified --shard-id with data sync status, it'll
display the realated pending and recovering bucket shards, as well as sync
marker, besides this, we can specified --max-entries to limit the number
of datalog entries(for pending buckets) or omapkey entries(for recovering
buckets) be consumed.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>
(cherry picked from commit 1dc7f5ea207dc17c743bfa1e06db446e16d31aa6)

rgw: read behind bucket shards of a specified data log shard

add read_shard_status to read pending bucket shards,
recovering bucket shards and sync marker

Signed-off-by: lvshanchun <lvshanchun@gmail.com>
(cherry picked from commit 9c5ff46ad9c849eb0c99a4d78be3f9dd2299779c)

Conflicts:
rgw_data_sync.cc (make_move_iterator fails to build with gcc<5)

osd/PG: fix DeferRecovery vs AllReplicasRecovered race

- DeferRecovery event queued by AsyncReserver due to preemption
  event.  We are in Recovering state with RECOVERING bit set.
- We finish recovery, clear RECOVERING state bit, and queue
  AllReplicasRecovered from PrimaryLogPG::start_recovery_ops()
- DeferRecovery event arrives, moving us from Recovering -> NotRecovering
- AllReplciasRecovered event arrives, crashing us.

This is all hard to deal with because the events are queued and may
arrive later.  Solve the problem here by tolerating a delayed
DeferRecovery event: if the RECOVERING pg state bit isn't set, ignore
it (it's old).  The async reserver cancel events are unpredictable.

Fixes: http://tracker.ceph.com/issues/23860
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cfe59cf20c4b09aa7b25c3f9a724a01380699744)