test: fix threading for FaultInjector death tests

addresses test timeout and warning message:

[WARNING] /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-death-test.cc:1121:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads. See https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests-and-threads for more explanation and suggested solutions, especially if this is the last message you see before your test times out.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove fault injection options from usage

these are only used for testing, not administration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBucketReshard doesn't need a friend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix decode of cls_rgw reshard types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager takes layouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up uses of BucketShard::init() without info

the rgw_bucket overload of BucketShard::init() has to look up the bucket
info. use the RGWBucketInfo overload when we have one

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable coredumps for reshard fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: ssl support for test_rgw_reshard.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add test_bucket_reshard() for fault injection testing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_rgw_reshard.py exec_cmd() can return error code

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket reshard' returns positive error codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: removed unused BucketInfoReshardUpdate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard first cleans up old-style reshards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor reshard init/cleanup with fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add typedef for ReshardFaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager stores BucketReshardShards by value

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused attrs from RGWBucketReshard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard tests with fault injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

common: add generic FaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: - Better error handling
     - function update_bucket() handles updating bucket state

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: use a helper function to handle repetitive num_shards check

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: code organization fixes and error handling

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: clean up fixes for extra spaces and indentation.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: helper function to set reshard status from rgw::BucketReshardState.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: minor fixes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: failing to reshard, restore old indexes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: - allow clean_index() to take const bucket_index_layout_generation&
     - fix bi_get() to get objects after being resharded

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: - take gen_id without optional as parameter in svc_bi_rados functions.

     - call init_index() on target layout during reshard process.
       Takes const rgw::bucket_index_layout_generation&

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: - make RGWBucketInfo a non-const reference in functions using guard_reshard()
     - remove unused 'instance_id' from cls_rgw_reshard_entry
     - other minor fixes

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>

rgw: target_layout to be optional

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>

rgw: use rgw::BucketReshardState for reshard state change

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: fetch updated bucketinfo rather than bucket_id in guard_reshard() to be
able to read the updated layout.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: use BucketReshardState states in appropriate places replacing cls_rgw_reshard_status.
Remove unused parameters in cls_rgw_bucket_instance_entry.
Other minor cleanup fixes.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: Allow clean_index to take generation number as a parameter.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: reshard bucket using the same bucket instance id

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: remove creation of new bucket instance

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

test/rgw: add --gen to radosgw-admin/help.t

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generations to error repo with binary format

adds a backward-compatible binary encoding for error repo keys that can
contain a generation number along with the bucket and shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generation support to BILog admin APIs

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: remove rgw_bucket_shard_sync_info::full_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: disable backoff on data sync error_retry_time

this backoff is triggered often by the per-bucket lease for full sync,
and causes tests to fail with checkpoint timeouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw_multisite_tests task loads default plugins

the ability to filter tests by attribute is provided by the
nose.plugins.attrib plugin, which wasn't being loaded by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable multisite tests for 'bucket sync disable'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add missing meta checkpoint to test_multipart_object_sync

if metadata sync hasn't finished, the 'bucket checkpoint' commands may
not find its bucket info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSyncBucketCR holds lease over Init state too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix up BucketShardIncrementalSync log message

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move full sync from SyncBucketShard to SyncBucket

renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id
parameter

renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack,
which now updates the bucket-level rgw_bucket_sync_status

renamed BucketShardFullSyncCR to BucketFullSyncCR

BucketSyncCR now takes a bucket-wide lease during full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync checkpoint' waits for full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync status' displays new full sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_read_bucket_full_sync_status()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: InitBucketShardSyncStatus always sets state to Incremental

full sync happens as the bucket level, so the shards will always start
in StateIncrementalSync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split SyncBucket from SyncBucketShard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add InitBucketFullSyncStatusCR

a coroutine to initialize a bucket for full sync using a new bucket-wide
sync status object

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosWriteAttrsCR supports exclusive create

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: system objects can set exclusive on set_attrs()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWSimpleRadosWriteCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: generalize error handling in RGWShardCollectCR

RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a
'failed to fetch log status' error message. this moves that logic into a
handle_result() virtual function. it also exposes the member variables
'status' and 'max_concurrent' as protected, so they can be consulted or
modified by overrides of handle_result() and spawn_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_bucket_shard_cr() factory function

allows other code to spawn this coroutine without having the class
definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to RGWSyncBucketShardCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add full_status_oid() for buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use const for string constants in rgw_data_sync.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to inc_status_oid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data structures for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to rgw_read_bucket_inc_sync_status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRunBucketsSyncBySourceCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosReadCR copies out objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39176 from ivancich/wip-rgw-gap-list

rgw: tooling to locate rgw objects with missing rados components

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Kidd <linuxkidd@gmail.com>

Merge PR #39213 into master

* refs/pull/39213/head:
qa/suites/upgrade/octopus-x/parallel: include cephfs in upgrade cluster

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39135 from orozery/librbd-crypto-performance

librbd: crypto performance optimization

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39298 from dillaman/wip-rbd-suite-readahead

qa/suites/rbd: drop require-osd-release command

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #39147 into master

* refs/pull/39147/head:
qa/tasks/ceph_fuse: do not createfs
qa/tasks/cephfs/fuse_mount: pass admin_socket path
qa/suites/fs/cephadm/multivolume: add basic multivolume test
mgr/mds_autoscaler: some fixes and cleanup
mgr/volumes: deploy MDSs when creating fs

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #39262 from zdover23/wip-doc-removing-workbench-from-docs-2021-Feb-first

doc/dev: Remove workbench mentions

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/rbd: drop require-osd-release command

Teuthology already defaults to quincy now and results in a failure
when trying to set to pacific. Additionally, drop the LUKS readbalance
test since it's unnecessary to duplicate that test.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38715 from lxbsz/rest_api

qa: add REST API method support for ceph-iscsi

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39265 from zdover23/wip-doc-rados-realy-typo-2020-feb

doc/rados: s/realy/really/

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39263 from cbodley/wip-48963

cmake: partial revert of BOOST_USE_VALGRIND when ALLOCATOR=libc

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #39182 from cbodley/wip-49065

rgw multisite: fix RGWCoroutine error handling

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39241 from dillaman/wip-49117

test/rbd-mirror: fix broken ceph_test_rbd_mirror_random_write

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/suites/upgrade/octopus-x/parallel: include cephfs in upgrade cluster

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39210 into master

* refs/pull/39210/head:
qa/suites/rados: move thrash-old-clients out of cephadm

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39229 from rhcs-dashboard/label-badge

mgr/dashboard: Add badge to the Label column in Host List

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39269 from tchaikov/wip-crimson-seastore-cleanup

crimson/os/seastore: do not capture unused variables

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #39162 from sebastian-philipp/cephadm-find-orch-ls-assert

mgr/cephadm: Add strings to assert statements

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38927 from sebastian-philipp/cephadm-upgrade-retry-ceph-volume

mgr/cephadm: try again calling ceph-volume without --filter-for-batch

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39242 from vshankar/wip-cephfs-mirroring-doc-cleanup

doc: cleanup cephfs-mirroring document

Merge pull request #39208 from sebastian-philipp/pyhton-common-more-typing

python-common: Add more type annotations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38685 from tchaikov/wip-gil

mgr/ActivePyModules.cc: always acquire locks without holding GIL

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39217 from myoungwon/wip-fix-seg-fault-manifest-ops

osd: fix to call nullptr when cancel_manifest_ops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39224 from petrutlucian94/rbd_fixes

Windows RBD fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39145 from ronen-fr/wip-ronenf-scrub-48793

osd: fix the scrubber behavior on multiple preemption attempts

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #39279 from cbodley/wip-cmake-spawn-options

cmake/rgw: forward spawn's compile options to rgw_common object library

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: add rgw-gap-list-comparator tool

The rgw-gap-list tool can produce a number of false positives when the
cluster is being used during its run. One technique to minimize the
number of false positives is to run the tool twice and look for the
objects that appear in both lists. The rgw-gap-list-comparator tool is
designed to do this comparison.

Signed-off-by: Michael Kidd <linuxkidd@gmail.com>

rgw: add rgw-gap-list tool

Due to a prior bug (pr: 38228) tail rados objects of some RGW objects
could have been incorrectly deleted. This tool is designed to look for
such cases. It essentially does the opposite of rgw-orphan-list,
looking for rados objects that RGW expects to be there, but which are
not to be found.

IMPORTANT: This is very experimental at this point in time, and any
"results" produced should be verified by other means.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Signed-off-by: Michael Kidd <linuxkidd@gmail.com>

Merge pull request #39275 from neha-ojha/wip-49139

qa/suites/rados/perf: pin to 18.04

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake/rgw: forward spawn's compile options to rgw_common object library

since rgw_common is an OBJECT library, we can't use
target_link_libraries() for its dependency on spawn. we add its
include directories manually already with
$<TARGET_PROPERTY:spawn,INTERFACE_INCLUDE_DIRECTORIES>, but this didn't
pull in the compile definitions. this ultimately prevented the
WITH_BOOST_VALGRIND option from passing the BOOST_USE_VALGRIND
definition attached to boost::context

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: remove unnecessary target_include_directories() for spawn

targets that depend on spawn via target_link_libraries() automatically
pull in their compile options and include directories

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39274 from yuriw/wip-yuriw-crontab-master

qa/tests: removed ceph-ansible (master and pacific)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/suites/perf-basic: pin to 18.04

This will have same problems as https://tracker.ceph.com/issues/49139

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tests: removed ceph-ansible (master and pacific)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/rados/perf: pin to 18.04

Caused by a07ca4c5c0f04c43c71231a898c2a89e99a80606

Fixes: https://tracker.ceph.com/issues/49139
Signed-off-by: Neha Ojha <nojha@redhat.com>