rgw: `radosgw-admin bucket stats` on indexless bucket crashes

The new bucket layout code didn't check whether the bucket is
indexless prior to asking for the last entry in the layout log. The
layout log appears to be empty for an indexless bucket, thereby
putting the runtime in an undefined state that later may cause a
failed assertion.

This commit adds two safety checks and returns -EINVAL along with
putting useful information on stderr when either stats are requested
on an indexless bucket or when the layout log is empty.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: fix reshard cancelling race condition

this is happening when resharding while objects are uploaded
tests steps are here:
https://gist.github.com/yuvalif/060f66f03511bff881e952287df3087b

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw: preserve 'bucket sync disable' over reshard

if bucket sync is disabled, apply that flag to new index objects on
bucket reshard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: handle shard_progress correctly in RunBucketSources

we run bucket sync on each of the sync pipes, so size the vector
accordingly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Revert "rgw: cr: add prealloc_stack()"

This reverts commit 7970f355497f48ee5a18bf3a0bc034226c6d225c.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Revert "rgw: bucket sync: track progress by stack id"

This reverts commit c0baf3eb34c6c1de7e4de2e35cb62e219c174b0b.

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Conflicts:
src/rgw/rgw_data_sync.cc no longer loops over num_shards

rgw/multisite: RunBucketSourcesSync no longer takes optional target

RGWDataSyncSingleEntryCR is the only caller of RGWRunBucketSourcesSyncCR

it always provides a source_bs, and never provides a target_bs. so remove
all the complexity related to target_bs, and the idea that we'd need to
sync several source bucket shards related to the target bucket

we now just have the single loop over the target buckets that use the
given bucket as a source

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: allow reshard commands in multisite on secondary

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix for uninitialized oldest_gen/latest_gen

when data sync queries RGWOp_BILog_Info from an un-upgraded gateway, it
doesn't include the oldest_gen/latest_gen fields. so initialize these
variables to 0 by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: enable RGWReshard thread on any zone that supports it

enable the background dynamic resharding thread based on
RGWSI_Zone::can_reshard(), which takes the zonegroup features into
account

Fixes: https://tracker.ceph.com/issues/52877
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: prevent reshard from creating too many log generations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove per-shard sync status object after incremental sync finishes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: bucket sync status guards against shard count mismatch

if the remote gives us more shards than we expect, just count those
shards as 'behind' and avoid out-of-bounds access of shard_status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: bucket sync status handles missing full status

if the full sync status object is missing, it's possible that we just
haven't started syncing it again after upgrading from just the per-shard
status objects

in this case, as long as we have a log generation 0, assume that we just
haven't initialized the full status object and try to read the gen=0
per-shard incremental status for comparison

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rgw_read_bucket_inc_sync_status doesn't need bucket info

all we need to construct the per-shard bucket sync status object names
are the bucket names themselves, which we already have from
rgw_sync_bucket_pipe

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: resize status vector before reading inc_sync_status

rgw_read_bucket_inc_sync_status() uses the size of this vector as the
'num_shards', so we need to resize it appropriately beforehand

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWOp_BILog_Status reads full status unconditionally

the calls to rgw_read_bucket_inc_sync_status() depend on
sync_status.incremental_gen, which we need to read via
rgw_read_bucket_full_sync_status() regardless of whether
we're returning it to the client (version > 1)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWCollectBucketSyncStatusCR doesn't need the shard count

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RunBucketSourceSync uses num_shards from remote bilog info

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RGWListBucketIndexesCR only needs zero shard

We only need to check one shard, and everything has shard zero.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: sync checkpoint gets num_shards from remote bilog info

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: RGWRemoteBucketManager constructor takes num_shards

The logic for getting it was moved to its caller.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: InitBucketFullSyncStatusCR gets num shards from remote

As specified in rgw_bucket_index_marker_info, unless we're doing the
compatibility check, in which case we look at generation 0.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: read shard count using remote bilog info during bucket sync

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

doc/rgw: document zone features

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSI_Zone::can_reshard() respects zonegroup 'resharding' feature

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'sync status' shows enabled/disabled zonegroup features

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: zone/zonegroup commands support --enable-feature=x --disable-feature=y

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add set of 'features' to zone and zonegroup

Signed-off-by: Casey Bodley <cbodley@redhat.com>

json: encoding for flat_set accepts all template params

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: don't delete per shard status on init

and pass correct generation and num shards when deleting
per shard status objects when disabling during incremental sync

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/multisite: support enable right after disable

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/multisite: remove the retry mechanism

when writign the sync status object

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/multisite: allow bucket sync disable/enable

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/multisite: track shard sync status objects per generation

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw: remove destination shard id from rgw_bucket_sync_pair_info

the sync_pair is used as input to RGWBucketPipeSyncStatusManager::status_oid()
to generate the per-shard sync status object names

this sync status tracks incremental bucket sync, which reads changes
from a source bucket's bilog shard, and copies objects from the remote
source bucket to the local destination bucket

this doesn't require sync to know anything about the destination bucket
shards, so rgw_bucket_sync_pair_info and status_oid() now only track the
the destination's rgw_bucket instead of rgw_bucket_shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Trim old generations in BucketTrimInstanceCR

Only one generation per call.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add RGWRadosRemoveOidCR

A more generally applicable way of removing objects in coroutines.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add RGWAsyncPutBucketInstanceInfoCR

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Trim bilog with generation

From the REST interface and radosgw-admin. Assume Generation 0 if none
provided and error if it doesn't exist.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Bilog trim takes markers as string view

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: get_str_vec takes std::string_view

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: BucketInstanceTrimCR knows about generations

Fetch the current generation from remote peers and trim the minimum
marker on the minimum generation.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add cast from bucket_index_log_layout

To bucket_index_layout_generation

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: add sync_info to BILog_Status output

Needed so we can get the incremental generation.

Guard this behind a version check and return the original output if
less than 2.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: fix incremental sync by using the right generation for bilog listing

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: on bucket reshard, write datalog entries for each shard of the previous generation

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/rgw: temporarily disable multisite reshard tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add multisite test for full sync after reshard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add simple multisite reshard test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync checkpoint' waits for generation to catch up

poll on rgw_read_bucket_full_sync_status() until
full_status.incremental_gen catches up to the latest_gen we got from
rgw_read_remote_bilog_info()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: get_bucket_instance_ids() uses num_shards from layout

knock out a TODO that was causing this assertion failure in
RGWRados::get_bucket_stats() after a reshard:

  ceph_assert(headers.size() == bucket_instance_ids.size());

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rgw_read_remote_bilog_info() returns rgw_bucket_index_marker_info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: adding generation number to async notification

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: add custom json encode/decode for the v1 notify API

this adds wrapper structs rgw_data_notify_v1_encoder and
rgw_data_notify_v1_decoder that can encode/decode the v1 json format
directly on the v2 data structure

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: fix python error on test failure

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix handling of bucket layout in metadata sync

clear the bucket layout we get from the metadata master, and overwrite it
with our zone's defaults

without clearing the layout, init_default_bucket_layout() was adding another
log layout in addition to the one from the master. this caused the bilog
list API to provide a 'next_log' when only gen=0 exists

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: fix bucket shard state init function

* make sure src/dest shard ids are the same in sync pair
* copy sync pair by value in coroutine loop

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw: update bucket sync status after bucket shards finishes current gen

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: reshard preserves old index in multisite

if the old index is still referenced by an InIndex log layout, we can't
call clean_index() to remove the index objects yet. log trimming will do
that later, once the bilogs are no longer needed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BILog_List handles requests for generation=0

previous logic treated requests for generation=0 as the latest gen

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: zero-initialize rgw_bucket_sync_status::incremental_gen

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: handle older/newer generations after reading bucket sync status

wait until we've read the bucket sync status and found that we're in
incremental sync before we start using incremental_gen for comparison

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Handle entries of the wrong generation

Drop entries from past generations.

Send entries of future generations to the error repo for retry.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: rgw_data_change can decode v1 format if gen was 0

but if gen>0, require decoders to understand the v2 format. this way,
old clients can't decode entries with gen>0, so they won't be able to
serve them to other zones

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename rgw_data_change::gen_id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add gen parameter to RGWDataChangesLog::add_entry

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add gen_id to rgw_data_change

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: vector<rgw_data_change_log_entry> not list

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: add json encoding of bucket layout types

adds a "layout" section to RGWBucketInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

json: add encode_json() overload for string_view

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor per-entry reshard logic into separate function

this cuts down on nesting and avoids the need for goto

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard adds a log layout for the new index

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: try reshard even if bucket is resharding

allow reshard in case a previous reshard failed. if the reshard is
actually still in progress, we'll fail to get the reshard lock

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: fix threading for FaultInjector death tests

addresses test timeout and warning message:

[WARNING] /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-death-test.cc:1121:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads. See https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests-and-threads for more explanation and suggested solutions, especially if this is the last message you see before your test times out.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove fault injection options from usage

these are only used for testing, not administration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBucketReshard doesn't need a friend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix decode of cls_rgw reshard types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager takes layouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up uses of BucketShard::init() without info

the rgw_bucket overload of BucketShard::init() has to look up the bucket
info. use the RGWBucketInfo overload when we have one

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable coredumps for reshard fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add test_bucket_reshard() for fault injection testing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_rgw_reshard.py exec_cmd() can return error code

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket reshard' returns positive error codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard first cleans up old-style reshards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor reshard init/cleanup with fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add typedef for ReshardFaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager stores BucketReshardShards by value

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused attrs from RGWBucketReshard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard tests with fault injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

common: add generic FaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Better error handling

  - function update_bucket() handles updating bucket state

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: use a helper function to handle repetitive num_shards check

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: code organization fixes and error handling

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: clean up fixes for extra spaces and indentation.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: helper function to set reshard status from rgw::BucketReshardState.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: minor fixes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: failing to reshard, restore old indexes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: allow clean_index to take const bucket_index_layout_generation&

fix bi_get() to get objects after being resharded

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: - take non-optional gen_id in svc_bi_rados functions

     - call init_index() on target layout during reshard process.
       Takes const rgw::bucket_index_layout_generation&

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>