rgw: vector<rgw_data_change_log_entry> not list

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: add json encoding of bucket layout types

adds a "layout" section to RGWBucketInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

json: add encode_json() overload for string_view

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor per-entry reshard logic into separate function

this cuts down on nesting and avoids the need for goto

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard adds a log layout for the new index

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: try reshard even if bucket is resharding

allow reshard in case a previous reshard failed. if the reshard is
actually still in progress, we'll fail to get the reshard lock

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: fix threading for FaultInjector death tests

addresses test timeout and warning message:

[WARNING] /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-death-test.cc:1121:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads. See https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests-and-threads for more explanation and suggested solutions, especially if this is the last message you see before your test times out.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: remove fault injection options from usage

these are only used for testing, not administration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWBucketReshard doesn't need a friend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix decode of cls_rgw reshard types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager takes layouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clean up uses of BucketShard::init() without info

the rgw_bucket overload of BucketShard::init() has to look up the bucket
info. use the RGWBucketInfo overload when we have one

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable coredumps for reshard fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add test_bucket_reshard() for fault injection testing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: test_rgw_reshard.py exec_cmd() can return error code

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket reshard' returns positive error codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard first cleans up old-style reshards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: refactor reshard init/cleanup with fault injection

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add typedef for ReshardFaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketReshardManager stores BucketReshardShards by value

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused attrs from RGWBucketReshard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reshard tests with fault injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

common: add generic FaultInjector

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Better error handling

  - function update_bucket() handles updating bucket state

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: use a helper function to handle repetitive num_shards check

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: code organization fixes and error handling

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: clean up fixes for extra spaces and indentation.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: helper function to set reshard status from rgw::BucketReshardState.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: minor fixes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: failing to reshard, restore old indexes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: allow clean_index to take const bucket_index_layout_generation&

fix bi_get() to get objects after being resharded

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: - take non-optional gen_id in svc_bi_rados functions

     - call init_index() on target layout during reshard process.
       Takes const rgw::bucket_index_layout_generation&

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: make RGWBucketInfo non-const in functions using guard_reshard()

 - remove unused 'instance_id' from cls_rgw_reshard_entry
 - other minor fixes

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>

rgw: target_layout to be optional

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>

rgw: use rgw::BucketReshardState for reshard state change

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: fetch updated bucketinfo rather than bucket_id in guard_reshard

To be able to read the updated layout.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: use BucketReshardState states in appropriate places

This replaces cls_rgw_reshard_status.

Remove unused parameters in cls_rgw_bucket_instance_entry.
Other minor cleanup fixes.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: Allow clean_index to take generation number

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: reshard bucket using the same instance id

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/dynamic-resharding: remove creation of new bucket instance

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

test/rgw: add --gen to radosgw-admin/help.t

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generations to error repo with binary format

adds a backward-compatible binary encoding for error repo keys that can
contain a generation number along with the bucket and shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generation support to BILog admin APIs

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: remove rgw_bucket_shard_sync_info::full_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: disable backoff on data sync error_retry_time

this backoff is triggered often by the per-bucket lease for full sync,
and causes tests to fail with checkpoint timeouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw_multisite_tests task loads default plugins

the ability to filter tests by attribute is provided by the
nose.plugins.attrib plugin, which wasn't being loaded by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable multisite tests for 'bucket sync disable'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add missing meta checkpoint to test_multipart_object_sync

if metadata sync hasn't finished, the 'bucket checkpoint' commands may
not find its bucket info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSyncBucketCR holds lease over Init state too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix up BucketShardIncrementalSync log message

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move full sync from SyncBucketShard to SyncBucket

renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id
parameter

renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack,
which now updates the bucket-level rgw_bucket_sync_status

renamed BucketShardFullSyncCR to BucketFullSyncCR

BucketSyncCR now takes a bucket-wide lease during full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync checkpoint' waits for full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync status' displays new full sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_read_bucket_full_sync_status()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: InitBucketShardSyncStatus always sets state to Incremental

full sync happens as the bucket level, so the shards will always start
in StateIncrementalSync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split SyncBucket from SyncBucketShard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add InitBucketFullSyncStatusCR

a coroutine to initialize a bucket for full sync using a new bucket-wide
sync status object

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosWriteAttrsCR supports exclusive create

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: system objects can set exclusive on set_attrs()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWSimpleRadosWriteCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: generalize error handling in RGWShardCollectCR

RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a
'failed to fetch log status' error message. this moves that logic into a
handle_result() virtual function. it also exposes the member variables
'status' and 'max_concurrent' as protected, so they can be consulted or
modified by overrides of handle_result() and spawn_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_bucket_shard_cr() factory function

allows other code to spawn this coroutine without having the class
definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to RGWSyncBucketShardCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add full_status_oid() for buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use const for string constants in rgw_data_sync.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to inc_status_oid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data structures for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to rgw_read_bucket_inc_sync_status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRunBucketsSyncBySourceCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosReadCR copies out objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46404 from phlogistonjohn/jjm-obj-format-pkg-fix

debian: include the new object_format.py file

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46356 from xxhdx1985126/wip-backref-gc-opt

crimson/os/seastore/segment_cleaner: add dedicated backref trimming process

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #45544 from wxypro/fix-typos-cephfs

common/options: fix typo

Reviewed-by: Venky Shankar <vshankar@redhat.com>

debian: include the new object_format.py file

This was missed when adding object_format.py to the RPM packaging.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

crimson/os/seastore/segment_cleaner: add dedicated backref trimming process

Space reclamation needs to merge backrefs up to the point where the latest
release of extents within the scope of the reclamation process happened.
When the journal size is large, that merge may generate a transaction
record with size exceeds the max record size threshold. So we need have a
backref trimming process that merge most of the backrefs before the space
reclamation happens.

This commit also fixes issue: https://tracker.ceph.com/issues/55692, by
repeating the inflight backrefs trimming transaction when it's
invalidated by other trans on the ROOT block

Fixes: https://tracker.ceph.com/issues/55692
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #46271 from xxhdx1985126/wip-backref-new_mapping-opt

crimson/os/seastore: backref related clean ups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge commit 'b76b6ea16899861d6c266797b3c11d9f68f924d7' into main

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #46390 from gabriellasroman/inclusive-lang

teuthology-cronjobs: master->main

teuthology-cronjobs: master->main

Signed-off-by: Gabriella Roman <gsroman@bu.edu>

Merge pull request #44486 from batrick/fs-workload-kclient-switches

qa: reduce fs:workload use of random selection

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46385 from ceph/inclusive-rename-yaml

Rename/re-symlink whitelist_*.yaml

Rename/re-symlink whitelist_*.yaml

whitelist_health.yaml -> ignorelist_health.yaml
whitelist_wrongly_marked_down.yaml -> ignore_wrongly_marked_down.yaml

This was mostly addressed in
https://github.com/ceph/ceph/commit/2ee9365d0bc220523632753f02c61eb5c2f4fbdd,
but the rename wasn't done there.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #44967 from batrick/nested-subset-doc

doc/dev: document nested subsets in teuthology

Merge pull request #46099 from NitzanMordhai/wip-nitzan-rados-test-delete-pools-parallel

systest: fix race with pool delete after socket injection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #45825 from NitzanMordhai/wip-nitzan-watch-notify-reconnect-107

librados/watch_notify: reconnect after socket injection

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #45805 from ifed01/wip-ifed-more-stats

os/bluestore: more debug information from BlueStore

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #45409 from NUABO/tancz

mon: add proxy to cache tier options

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #46324 from ceph/dgalloway-nodockerio

qa: Pull from quay.io not docker.io

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46366 from kalebskeithley/rocksdb

rocksdb: build with rocksdb-7.y.z

Merge pull request #46185 from dparmar18/fix_rm_error_output

cephfs-shell: fix rm cmd's error output

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #46160 from dparmar18/fix_tracker_40860

cephfs-shell: fix rmdir cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #46249 from rhcs-dashboard/rbd-mirroring-commands

mgr/dashboard: enable/disable/promote/demote/resync mirroring images

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore/segment_cleaner: downgrade the level of unnecessary logs to DEBUG

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/backref: rename BackrefManager::batch_insert_from_cache to BackrefManager::merge_cached_backrefs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/segment_cleaner: don't repeat backref mapping retrieval when gc transactions get invalidated

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: mandate all access to backrefs to go through backref manager

this would avoid other components' unnecessary dependency on Cache

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #45728 from lxbsz/wip-55144

client: stop retrying the request when exceeding 256 times

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46034 from mchangir/mgr-snap_schedule-remove-subvolume-support

mgr/snap_schedule: remove subvol interface

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46351 from cyx1231st/wip-seastore-cleaner-cleanup

crimson/os/seastore/segment_cleaner: minor cleanups

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #46314 from rishabh-d-dave/qa-fs-fsid

qa/vstart_runner: store FSID in LocalContext instances

Reviewed-by: Venky Shankar <vshankar@redhat.com>