rgw/multisite: fix bucket shard state init function

* make sure src/dest shard ids are the same in sync pair
* copy sync pair by value in coroutine loop

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw: update bucket sync status after bucket shards finishes current gen

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: reshard preserves old index in multisite

if the old index is still referenced by an InIndex log layout, we can't
call clean_index() to remove the index objects yet. log trimming will do
that later, once the bilogs are no longer needed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BILog_List handles requests for generation=0

previous logic treated requests for generation=0 as the latest gen

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: zero-initialize rgw_bucket_sync_status::incremental_gen

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: handle older/newer generations after reading bucket sync status

wait until we've read the bucket sync status and found that we're in
incremental sync before we start using incremental_gen for comparison

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Handle entries of the wrong generation

Drop entries from past generations.

Send entries of future generations to the error repo for retry.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: rgw_data_change can decode v1 format if gen was 0

but if gen>0, require decoders to understand the v2 format. this way,
old clients can't decode entries with gen>0, so they won't be able to
serve them to other zones

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename rgw_data_change::gen_id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add gen parameter to RGWDataChangesLog::add_entry

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add gen_id to rgw_data_change

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: vector<rgw_data_change_log_entry> not list

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: add json encoding of bucket layout types

adds a "layout" section to RGWBucketInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

json: add encode_json() overload for string_view

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor per-entry reshard logic into separate function

this cuts down on nesting and avoids the need for goto

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard adds a log layout for the new index

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: try reshard even if bucket is resharding

allow reshard in case a previous reshard failed. if the reshard is
actually still in progress, we'll fail to get the reshard lock

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: fix threading for FaultInjector death tests

addresses test timeout and warning message:

[WARNING] /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-death-test.cc:1121:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads. See https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests-and-threads for more explanation and suggested solutions, especially if this is the last message you see before your test times out.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove fault injection options from usage

these are only used for testing, not administration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBucketReshard doesn't need a friend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix decode of cls_rgw reshard types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager takes layouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up uses of BucketShard::init() without info

the rgw_bucket overload of BucketShard::init() has to look up the bucket
info. use the RGWBucketInfo overload when we have one

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable coredumps for reshard fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add test_bucket_reshard() for fault injection testing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_rgw_reshard.py exec_cmd() can return error code

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket reshard' returns positive error codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard first cleans up old-style reshards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor reshard init/cleanup with fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add typedef for ReshardFaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager stores BucketReshardShards by value

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused attrs from RGWBucketReshard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard tests with fault injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

common: add generic FaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Better error handling

  - function update_bucket() handles updating bucket state

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: use a helper function to handle repetitive num_shards check

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: code organization fixes and error handling

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: clean up fixes for extra spaces and indentation.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: helper function to set reshard status from rgw::BucketReshardState.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: minor fixes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: failing to reshard, restore old indexes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: allow clean_index to take const bucket_index_layout_generation&

fix bi_get() to get objects after being resharded

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: - take non-optional gen_id in svc_bi_rados functions

     - call init_index() on target layout during reshard process.
       Takes const rgw::bucket_index_layout_generation&

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: make RGWBucketInfo non-const in functions using guard_reshard()

 - remove unused 'instance_id' from cls_rgw_reshard_entry
 - other minor fixes

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>

rgw: target_layout to be optional

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>

rgw: use rgw::BucketReshardState for reshard state change

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: fetch updated bucketinfo rather than bucket_id in guard_reshard

To be able to read the updated layout.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: use BucketReshardState states in appropriate places

This replaces cls_rgw_reshard_status.

Remove unused parameters in cls_rgw_bucket_instance_entry.
Other minor cleanup fixes.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: Allow clean_index to take generation number

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: reshard bucket using the same instance id

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: remove creation of new bucket instance

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

test/rgw: add --gen to radosgw-admin/help.t

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generations to error repo with binary format

adds a backward-compatible binary encoding for error repo keys that can
contain a generation number along with the bucket and shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generation support to BILog admin APIs

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: remove rgw_bucket_shard_sync_info::full_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: disable backoff on data sync error_retry_time

this backoff is triggered often by the per-bucket lease for full sync,
and causes tests to fail with checkpoint timeouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw_multisite_tests task loads default plugins

the ability to filter tests by attribute is provided by the
nose.plugins.attrib plugin, which wasn't being loaded by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable multisite tests for 'bucket sync disable'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add missing meta checkpoint to test_multipart_object_sync

if metadata sync hasn't finished, the 'bucket checkpoint' commands may
not find its bucket info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSyncBucketCR holds lease over Init state too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix up BucketShardIncrementalSync log message

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move full sync from SyncBucketShard to SyncBucket

renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id
parameter

renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack,
which now updates the bucket-level rgw_bucket_sync_status

renamed BucketShardFullSyncCR to BucketFullSyncCR

BucketSyncCR now takes a bucket-wide lease during full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync checkpoint' waits for full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync status' displays new full sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_read_bucket_full_sync_status()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: InitBucketShardSyncStatus always sets state to Incremental

full sync happens as the bucket level, so the shards will always start
in StateIncrementalSync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split SyncBucket from SyncBucketShard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add InitBucketFullSyncStatusCR

a coroutine to initialize a bucket for full sync using a new bucket-wide
sync status object

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosWriteAttrsCR supports exclusive create

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: system objects can set exclusive on set_attrs()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWSimpleRadosWriteCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: generalize error handling in RGWShardCollectCR

RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a
'failed to fetch log status' error message. this moves that logic into a
handle_result() virtual function. it also exposes the member variables
'status' and 'max_concurrent' as protected, so they can be consulted or
modified by overrides of handle_result() and spawn_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_bucket_shard_cr() factory function

allows other code to spawn this coroutine without having the class
definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to RGWSyncBucketShardCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add full_status_oid() for buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use const for string constants in rgw_data_sync.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to inc_status_oid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data structures for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to rgw_read_bucket_inc_sync_status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRunBucketsSyncBySourceCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosReadCR copies out objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44833 from fstachura/wip-doc-more-pdf-links

doc: Fix links to CRUSH, RADOS and DSP research papers

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc: Fix links to CRUSH, RADOS and DSP research papers.

Signed-off-by: Franciszek Stachura <fbstachura@gmail.com>

Merge pull request #44707 from liumiaomiaoIntel/qatdoc

doc/radosgw: update QAT acceleration to specify the configuration

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/radosgw: update QAT acceleration to specify the configuration

add and change the configuration steps to enable latest version of QAT
driver on QAT based encryption and compression

Signed-off-by: Miaomiao Liu <miaomiao.liu@intel.com>

Merge pull request #44816 from cbodley/wip-rgw-cmake-system

cmake/rgw: src/rgw and src/rgw/services are not system headers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #44801 from sebastian-philipp/doc-cephadm-count-per-host

doc/cephadm: Co-location of daemons

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by:Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #44823 from andriytk/fix-install-deps2

install-deps.sh: fix cortx-motr installation condition

install-deps.sh: fix cortx-motr installation condition

Don't try to install cortx-motr if it is already installed
or not needed. (Currently, it installs on any run from an
interactive terminal session).

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

Merge pull request #44711 from zdover23/wip-doc-2022-01-21-rados-bluestore-cache-autotune-interval-rewrite-2

src/rados: rewrite bs cache autotune instructions

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44703 from phlogistonjohn/jjm-issue46773

cephadm: validate that the constructed YumDnf baseurl is usable

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44517 from adk3798/agent-cache

mgr/cephadm: store agent metadata in its own cache

Reviewed-By: Sebastian Wagner <sewagner@redhat.com>

cmake/rgw: src/rgw and src/rgw/services are not system headers

include these with -I instead of -isystem so we don't hide their
compiler warnings

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44719 from zdover23/wip-doc-2021-01-22-man-page-radosgw-multitenancy-remove-double-quotes

doc/radosgw: replace double quotes with single

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44379 from andriytk/add-motr-rgw-sal

rgw: add CORTX Motr SAL implementation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #44654 from rhcs-dashboard/page-title-feature

mgr/dashboard: Page title should indicate current location

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44793 from anthonyeleven/anthonyeleven/cephfs-mirroring

doc/cephfs: Clean up mirroring

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw: add CORTX Motr SAL implementation

To build with Motr backend, use -DWITH_RADOSGW_MOTR=YES cmake
option. cortx-motr-devel rpm should be installed beforehand.

To connect to Motr cluster, add the following configuration
parameters to ceph.conf:

    [client]
            ...
            rgw backend store = motr
            motr profile fid  = 0x7000000000000001:0x4f
            motr ha endpoint  = inet:tcp:10.0.0.1@2001
    ...
    [client.rgw.8000]
            ...
            motr my endpoint  = inet:tcp:10.0.0.1@5001
            motr my fid       = 0x7200000000000001:0x29

The correct values for the Motr connection parameters can be
taken from `hctl status` cmd output after Motr cluster is
bootstrapped. In the example above, the values were taken from
the following output:

    Profile:
        0x7000000000000001:0x4f
    Services:
        centos8n1
        [started] hax       0x7200000000000001:0x6  inet:tcp:10.0.0.1@5001
        ...
        [unknown] m0_client 0x7200000000000001:0x29 inet:tcp:10.0.0.1@5001

Motr pkgs for the build/run can be taken from
https://github.com/Seagate/cortx-motr/releases/tag/2.0.0-rgw.

Co-authored-by: Sining Wu <sining.wu@seagate.com>
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

mgr/dashboard: Page title should indicate current location

Added titleService to BreadcrumbsComponent.

Fixes: https://tracker.ceph.com/issues/36587
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>

doc/cephadm: Co-location of daemons

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>