rgw: save bucket instance xattrs when resharding cancelled
There appears to be a long-standing bug in RGW such that when
resharding is cancelled and the bucket instance is updated to reflect
the new resharding status, the xattrs were lost. The xattrs are used
to store metadata such as ACLs and LifeCycle policies.
This commit makes sure that all call paths that lead to a cancelled
reshard provide the xattrs, so they can be included when the bucket
instance info is updated.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
rgw: resharding causes bucket attributes to be lost
With the new resharding code, some bucket metadata that is stored as
xattrs (e.g., ACLs, life-cycle policies) were not sent with the
updated bucket instance data when resharding completed. As a result,
resharding has a regression where that metadata is lost after a
successful reshard.
This commit restores the variable in the RGWBucketReshard class that
maintains the bucket attributes, so they can be saved when the bucket
instance object is updated.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
J. Eric Ivancich [Mon, 17 Jan 2022 21:14:16 +0000 (16:14 -0500)]
rgw: add indexless bucket logic to "bucket radoslist"
The "bucket radoslist" sub-command of radosgw-admin is supposed to
list all rados objects tied to one or all directories and thereby
provide a way to determine orphaned rados objects.
But indexless buckets don't provide an index to employ for this
purpose. So warnings or errors should be provided depending on the
circumstances.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
J. Eric Ivancich [Mon, 24 Jan 2022 21:10:57 +0000 (16:10 -0500)]
rgw: update indexless bucket check for bucket stats
The code for bucket stats was recently updated to check for an
indexless bucket before proceeding. The interface on RGWBucketInfo was
recently expanded to support these types of checks, so it is now used.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
J. Eric Ivancich [Mon, 24 Jan 2022 21:08:01 +0000 (16:08 -0500)]
rgw: add streamlined ways to handle indexless buckets correctly
Determining whether a bucket is indexless starting with an
RGWBucketInfo object requires traversing multiple data structures and
"inside knowledge" blurring the line between interface and
implementation. The same applies for retrieving the current index for
non-indexless buckets.
This commit adds to the RGWBucketInfo interface to make this
information readily accessible.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
J. Eric Ivancich [Tue, 21 Dec 2021 22:27:37 +0000 (17:27 -0500)]
rgw: "bucket check --fix" should delete damaged multipart uploads from bi
As one of the steps in `radosgw-admin bucket check --fix ...` it looks
for bucket index entries for incomplete multipart uploads that do not
have a corresponding ".meta" entry in the same bucket index. It then
intends to delete those entries, however the function that it calls
to perform the bucket index deletions was flawed and did not direct
the removals to the appropriate shard(s), but instead a non-existant
oid.
This commit determines the appropriate shard for each of the entries
to be removed and asynchronously issues "dir suggest changes" to each
of the shards to remove the entries.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
J. Eric Ivancich [Thu, 23 Dec 2021 21:25:26 +0000 (16:25 -0500)]
rgw: `radosgw-admin bucket stats` on indexless bucket crashes
The new bucket layout code didn't check whether the bucket is
indexless prior to asking for the last entry in the layout log. The
layout log appears to be empty for an indexless bucket, thereby
putting the runtime in an undefined state that later may cause a
failed assertion.
This commit adds two safety checks and returns -EINVAL along with
putting useful information on stderr when either stats are requested
on an indexless bucket or when the layout log is empty.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Casey Bodley [Mon, 22 Nov 2021 18:05:40 +0000 (13:05 -0500)]
rgw/multisite: RunBucketSourcesSync no longer takes optional target
RGWDataSyncSingleEntryCR is the only caller of RGWRunBucketSourcesSyncCR
it always provides a source_bs, and never provides a target_bs. so remove
all the complexity related to target_bs, and the idea that we'd need to
sync several source bucket shards related to the target bucket
we now just have the single loop over the target buckets that use the
given bucket as a source
Casey Bodley [Mon, 18 Oct 2021 18:15:33 +0000 (14:15 -0400)]
rgw: fix for uninitialized oldest_gen/latest_gen
when data sync queries RGWOp_BILog_Info from an un-upgraded gateway, it
doesn't include the oldest_gen/latest_gen fields. so initialize these
variables to 0 by default
Casey Bodley [Thu, 19 Aug 2021 21:21:37 +0000 (17:21 -0400)]
radosgw-admin: bucket sync status handles missing full status
if the full sync status object is missing, it's possible that we just
haven't started syncing it again after upgrading from just the per-shard
status objects
in this case, as long as we have a log generation 0, assume that we just
haven't initialized the full status object and try to read the gen=0
per-shard incremental status for comparison
Casey Bodley [Thu, 19 Aug 2021 20:12:21 +0000 (16:12 -0400)]
rgw: rgw_read_bucket_inc_sync_status doesn't need bucket info
all we need to construct the per-shard bucket sync status object names
are the bucket names themselves, which we already have from
rgw_sync_bucket_pipe
Casey Bodley [Thu, 19 Aug 2021 20:21:55 +0000 (16:21 -0400)]
rgw: RGWOp_BILog_Status reads full status unconditionally
the calls to rgw_read_bucket_inc_sync_status() depend on
sync_status.incremental_gen, which we need to read via
rgw_read_bucket_full_sync_status() regardless of whether
we're returning it to the client (version > 1)
Casey Bodley [Wed, 9 Jun 2021 16:26:54 +0000 (12:26 -0400)]
rgw: remove destination shard id from rgw_bucket_sync_pair_info
the sync_pair is used as input to RGWBucketPipeSyncStatusManager::status_oid()
to generate the per-shard sync status object names
this sync status tracks incremental bucket sync, which reads changes
from a source bucket's bilog shard, and copies objects from the remote
source bucket to the local destination bucket
this doesn't require sync to know anything about the destination bucket
shards, so rgw_bucket_sync_pair_info and status_oid() now only track the
the destination's rgw_bucket instead of rgw_bucket_shard
Casey Bodley [Tue, 23 Feb 2021 19:55:50 +0000 (14:55 -0500)]
rgw: add custom json encode/decode for the v1 notify API
this adds wrapper structs rgw_data_notify_v1_encoder and
rgw_data_notify_v1_decoder that can encode/decode the v1 json format
directly on the v2 data structure
Casey Bodley [Fri, 14 May 2021 16:30:21 +0000 (12:30 -0400)]
rgw: fix handling of bucket layout in metadata sync
clear the bucket layout we get from the metadata master, and overwrite it
with our zone's defaults
without clearing the layout, init_default_bucket_layout() was adding another
log layout in addition to the one from the master. this caused the bilog
list API to provide a 'next_log' when only gen=0 exists
Casey Bodley [Fri, 26 Mar 2021 15:00:57 +0000 (11:00 -0400)]
rgw: reshard preserves old index in multisite
if the old index is still referenced by an InIndex log layout, we can't
call clean_index() to remove the index objects yet. log trimming will do
that later, once the bilogs are no longer needed
Casey Bodley [Wed, 10 Feb 2021 00:04:19 +0000 (19:04 -0500)]
rgw: rgw_data_change can decode v1 format if gen was 0
but if gen>0, require decoders to understand the v2 format. this way,
old clients can't decode entries with gen>0, so they won't be able to
serve them to other zones
Casey Bodley [Tue, 19 Jan 2021 14:27:13 +0000 (09:27 -0500)]
test: fix threading for FaultInjector death tests
addresses test timeout and warning message:
[WARNING] /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-death-test.cc:1121:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads. See https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests-and-threads for more explanation and suggested solutions, especially if this is the last message you see before your test times out.
projects / ceph.git / log