test/rgw: add --gen to radosgw-admin/help.t

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generations to error repo with binary format

adds a backward-compatible binary encoding for error repo keys that can
contain a generation number along with the bucket and shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generation support to BILog admin APIs

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: remove rgw_bucket_shard_sync_info::full_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: disable backoff on data sync error_retry_time

this backoff is triggered often by the per-bucket lease for full sync,
and causes tests to fail with checkpoint timeouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw_multisite_tests task loads default plugins

the ability to filter tests by attribute is provided by the
nose.plugins.attrib plugin, which wasn't being loaded by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable multisite tests for 'bucket sync disable'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add missing meta checkpoint to test_multipart_object_sync

if metadata sync hasn't finished, the 'bucket checkpoint' commands may
not find its bucket info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSyncBucketCR holds lease over Init state too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix up BucketShardIncrementalSync log message

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move full sync from SyncBucketShard to SyncBucket

renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id
parameter

renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack,
which now updates the bucket-level rgw_bucket_sync_status

renamed BucketShardFullSyncCR to BucketFullSyncCR

BucketSyncCR now takes a bucket-wide lease during full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync checkpoint' waits for full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync status' displays new full sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_read_bucket_full_sync_status()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: InitBucketShardSyncStatus always sets state to Incremental

full sync happens as the bucket level, so the shards will always start
in StateIncrementalSync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split SyncBucket from SyncBucketShard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add InitBucketFullSyncStatusCR

a coroutine to initialize a bucket for full sync using a new bucket-wide
sync status object

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosWriteAttrsCR supports exclusive create

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: system objects can set exclusive on set_attrs()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWSimpleRadosWriteCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: generalize error handling in RGWShardCollectCR

RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a
'failed to fetch log status' error message. this moves that logic into a
handle_result() virtual function. it also exposes the member variables
'status' and 'max_concurrent' as protected, so they can be consulted or
modified by overrides of handle_result() and spawn_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_bucket_shard_cr() factory function

allows other code to spawn this coroutine without having the class
definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to RGWSyncBucketShardCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add full_status_oid() for buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use const for string constants in rgw_data_sync.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to inc_status_oid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data structures for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to rgw_read_bucket_inc_sync_status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRunBucketsSyncBySourceCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosReadCR copies out objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45598 from rkachach/fix_issue_55008

mgr/cephadm: Adding image tag and date to cephadm startup messages

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #45855 from rkachach/fix_issue_55265

mgr/cephadm: fixing call method timeout handling

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45877 from rkachach/fix_issue_55185

mgr/cephadm: Adding support to store ceph conf per cluster fsid

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45899 from rkachach/fix_issue_53527

mgr/cephadm: fixing upgrade candidate versions listing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45623 from dang/wip-dang-zipper-objctx

RGW: Zipper - zonegroup, placement tier

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46010 from gregsfortytwo/wip-55419-blocklist-test

test: Fix a blocklist API test

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44958 from rishabh-d-dave/qa-cephfs-timeout

qa/cephfs: change default timeout from 900 secs to 300

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45737 from cbodley/wip-rgw-ops-log-defaults

rgw: change default ops log backend from rados to file

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #46007 from cbodley/wip-49302

rgw: RGWCoroutine::set_sleeping() checks for null stack

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #46027 from cbodley/wip-55432

rgw: OpsLogFile::stop() signals under mutex

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #45784 from cbodley/wip-rgw-multisite-configurables

rgw multisite: configurable polling intervals and concurrency windows

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

qa/cephfs: change default timeout from 900 secs to 300

15 minutes is unnecessarily large as a default value for timeout for a
command. Not having to wait unnecessarily on a crash of a command will
reduce teuthology's testing queue and will save individual developer's
time while running tests locally.

Whatever lines are modified for this purpose are also modified to follow
the stlye guideline, specfically wrapping at 80 characters.

Fixes: https://tracker.ceph.com/issues/54236
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #45830 from rhcs-dashboard/cephadm-logs-save

mgr/dashboard: save mgr logs inside a folder

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #45370 from lxbsz/wip-54411

client: add option to disable collecting and sending metrics

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45904 from cfsnyder/fix_rocksdb_iter_perf

os/bluestore: set upper and lower bounds on rocksdb omap iterators

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #46028 from ceph/revert-43413-bluestore-bufferbloat-mitigation

Revert "os/bluestore: Add CoDel to BlueStore for Bufferbloat mitigation"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

rgw: OpsLogFile drops lock before flush()

flush() acquires the same lock, so lockdep complains about recursive
locking here

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWCoroutine::set_sleeping() checks for null stack

users of the RGWOmapAppend coroutine don't manage the lifetime of its
underlying coroutine stack, so end up making calls on RGWOmapAppend
after its stack goes away. this null check is a band-aid, and there are
still several other calls in RGWCoroutine that don't check for null
stack

Fixes: https://tracker.ceph.com/issues/49302
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

bluestore: Revert "os/bluestore: Add CoDel to BlueStore for Bufferbloat mitigation"

Merge caused: https://tracker.ceph.com/issues/55433

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #44989 from Ejdesgaard/patch-1

radosgw-admin zone deletion uses zone delete

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unnecessary OpsLogFile::flush_mutex

this mutex was only held by one function, OpsLogFile::flush(). this
private member function is only ever called from the background thread,
so doesn't need to be protected by a mutex

as a further cleanup, i renamed 'cond' and 'mutex' now that we don't
need to differentiate between different locks

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: OpsLogFile::stop() signals under mutex

this shuts up ceph::debug_condition_variable's assertion that the
associated mutex is held during notify_one(). this is not strictly
required for correct use, but is a common source of bugs

Fixes: https://tracker.ceph.com/issues/55432
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45407 from vshankar/wip-54046

mds: include encoded stray inode when sending dentry unlink message to replicas

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45243 from lxbsz/mdslog_bz

mds: flush mdlog if locked and still has wanted caps not satisfied

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45307 from lxbsz/wip54461

mds: notify clients if the session has already opened

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45887 from bluikko/patch-3

doc: replace spaces with underscores in config option names

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc: remove spaces at line ends and doubles, fix wrapping

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc: replace spaces with underscores in config option names

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #45948 from cyx1231st/wip-seastore-cleaner-work

crimson/os/seastore/segment_cleaner: cleanup segment_info_set_t

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

test: use the same address input format as output will generate

Otherwise, our grep fails!

Fixes: https://tracker.ceph.com/issues/55419
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #43413 from esmaeil-mirvakili/bluestore-bufferbloat-mitigation

os/bluestore: Add CoDel to BlueStore for Bufferbloat mitigation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/cephadm: fixing call method timeout handling
Fixes: https://tracker.ceph.com/issues/55265
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/cephadm: Adding support to store ceph conf per cluster fsid
Fixes: https://tracker.ceph.com/issues/55185
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/cephadm: fixing upgrade candidate verions listing
Fixes: https://tracker.ceph.com/issues/53527
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/cephadm: Using '.ID' instead of '.Id' which is only supported by podman

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45920 from adk3798/stop-mgr-nfs-upgrade-fail

qa/suites/orch/cephadm: stop upgrade tests if failures are seen

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45931 from rkachach/fix_issue_51671

mgr/cephadm: fixing logging output for rm-cluster operation

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45982 from rkachach/fix_issue_52727

mgr/cephadm: do not add _admin label when no-minimize-config is provided

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45934 from ceph/fix-github_triage-master

.github/pr-triage: reorder milestone step

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #45631 from Rethan/feat-mds-slow-ops

mds: add a perf counter to record slow replies

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45978 from tchaikov/wip-cmake-python3-exact

cmake/modules: use exact version of python3 when finding cython

Reviewed-by: Casey Bodley <cbodley@redhat.com>

kv/RocksDBStore: simplify RocksDBStore::get_cf_handle(string, IteratorBounds)

Adds a precondition to RocksDBStore::get_cf_handle(string, IteratorBounds)
to avoid duplicating logic of the only caller (RocksDBStore::get_iterator).
Assertions will fail if preconditions are not met.

Signed-off-by: Cory Snyder <csnyder@iland.com>

bluestore: add config option to allow rocksdb iterator bounds to be disabled

Add osd_rocksdb_iterator_bounds_enabled config option to allow rocksdb iterator bounds to be disabled.
Also includes minor refactoring to shorten code associated with IteratorBounds initialization in bluestore.

Signed-off-by: Cory Snyder <csnyder@iland.com>

Merge pull request #44151 from gregsfortytwo/wip-cidr-blocklist

Implement CIDR blocklisting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #45971 from rishabh-d-dave/cephfs-shell-exclam-marks

qa/cephfs: bug fix and some cleanup for test_cephfs_shell.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45977 from tchaikov/wip-cmake-dml-cleanup

cmake: try to find dml only if PMEM is enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/cephfs: cleanup some code in test_cephfs_shell.py

Also delete a comment since it's incorrect.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: don't use exclam mark in tests in test_cephfs_shell.py

Exclamation mark is a special character for bash as well as
cephfs-shell. For bash, it substitutes current command with matching
command from command history and for cephfs-shell it runs the command
as OS-level command and not inside the cephfs-shell.

And evey command executed in tests (say "ls") is run by passing it as a
parameter to cephfs-shell command (that is "cephfs-shell -c <conf> --
ls"). So, exclamation mark, when used in tests, is consumed by bash
instead of cephfs-shell.

To avoid these complications it's best (and even simpler!) to issue the
command meant for bash on bash without going through cephfs-shell.

Fixes: https://tracker.ceph.com/issues/55394
Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/cephadm: do not add _admin label when no-minimize-config is provided
Fixes: https://tracker.ceph.com/issues/52727
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45979 from lxbsz/doc_noatime

doc: cephfs: update with atime not supporting explaination

mgr/cephadm: Adding image tag and date to cephadm startup messages
Fixes: https://tracker.ceph.com/issues/55008
Fixes: https://tracker.ceph.com/issues/54373
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

doc: cephfs: update with atime not supporting explaination

Signed-off-by: Xiubo Li <xiubli@redhat.com>

test: osd: add a /0 cidr test for 32-bit...

...and make the OSDMap handle it.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: add a /0 cidr test that blocklists EVERYTHING

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: add a 128-bit range blocklist test

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #45975 from adamemerson/wip-mrun-bashism

mrun: Change mrun to use bash

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cmake/modules: use exact version of python3 when finding cython

* CMakeLists.txt:
    always pass "EXACT" to find_package(Python3).
    because per cmake document, "EXACT" only takes effect when
    <Package>_FIND_VERSION_COUNT is greater than 1, where <Package>
    is "Python3". see also cmake/modules/FindPython/Support.cmake
* cmake/modules/AddCephTest.cmake:
    drop redundant find_package(Python3) calls. since Python3 is
    a mandatory requirement for building Ceph, we only need a
    single call of find_package(Python3..) in the top of the source
    tree. the only possible case to repeat it is to ensure that we
    have the correct version of Python3 used in following CMake
    script. but there is no need to repeat it if we just want to
    ensure that we have a python3 interpretor in place.
* cmake/modules/Distutils.cmake:
    always pass "EXACT" to find_package(Python3).
    we should always pass EXACT to find_package() when finding python3,
    this is a follow-up of e2babdfae8c99f39f99a7c8a8f966299b2e62b19

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: try to find dml only if PMEM is enabled

as the feature dependent on DML library requires PMEM backend.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tooling: Change mrun to use bash

Since mrun contains some bashisms, have it use bash explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #45973 from ionutbalutoiu/add-windows-tests

qa/crontab/teuthology-cronjobs: add windows suite

qa/crontab/teuthology-cronjobs: add windows suite

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

RGW: Zipper - zonegroup, placement tier

Add ZoneGroup and PlacementTier to the Zipper API.  This cleaned up a
lot of issues in the Module branch.

In addition, add cloud transition.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

test: add 32-bit ipv4 and 64-bit ipv6 blocklist testing

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: add a 30-bit range mask to ipv6 blocklisting tests

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test: split up ip- and range-based blocklisting tests

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #44230 from optimistyzy/122_add_dml

blk/pmem: use DML library to offload read/write operations in pmem

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #45872 from idryomov/wip-make-check-enable-rbd-caches

run-make-check.sh: enable RBD persistent caches

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore/segment_cleaner: cleanup SegmentProvider interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: cleanup segment_info_set_t

* better encapsulated interfaces to SegmentCleaner;
* drop unused device related implementations;
* improve related metrics and logs;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #45609 from optimistyzy/wip_pmem_devdax

blk/pmem: Add the devdax support.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>