mgr/cephadm/nvmeof: Add OMAP read lock parameters to NVMEoF configuration

Fixes https://tracker.ceph.com/issues/70861

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #57206 from clwluvw/restrict-public-buckets

rgw: implement RestrictPublicBuckets from PublicAccessBlock

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62538 from cbodley/wip-rgw-control-watch-par

rgw: send concurrent watch/unwatch operations

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #62159 from cbodley/wip-qa-23264

qa/rgw: use storage classes for rgw/crypt

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #62706 from cbodley/wip-70729

test/rgw/multisite: fix parsing of 'topic list' output

Reviewed-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #62686 from Cybertinus/main

documentation:remove deprecated usage apt-key

documentation: Update get-packages.rst

The usage of apt-key is deprecated, thus updated the documentation to reference the new way of installing a repo key.

Signed-off-by: Cybertinus <github@cybertinus.nl>

Merge pull request #62727 from idryomov/wip-70831

test/librbd/test_notify.py: force line-buffered output

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Vinay Bhaskar Varada <vvarada@redhat.com>

Merge pull request #57448 from aclamk/wip-aclamk-bs-recompression-segmented-data

os/bluestore: Recompression, part 3. Segmented onode.

Merge pull request #62224 from aclamk/wip-aclamk-pere-wal-fsync

os/bluestore: Fast WAL for RocksDB

Merge pull request #62588 from aclamk/wip-aclamk-bluefs-remove-truncate

os/bluestore: Fix race in BlueFS truncate / remove

Merge pull request #61909 from bazaah/chore/deprecated-iterator-impl

include,dpdk: fix two usages of std::iterator (deprecated)

qa/rgw: use storage classes for rgw/crypt

enable compression on one storage class so we can test copy to/from
compressed objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: storage classes specify extra args in yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60670 from cbodley/wip-cls-rgw-shard-io

rgw/rados: add async algorithms for concurrent index operations

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #62720 from tchaikov/wip-kefu-mailmap

mailmap: update affiliation for Kefu Chai

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62716 from afreen23/wip-fix-empty-ver

mgr/dashboard: Fix empty ceph version in GET api/hosts

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

cls/rgw: remove unused CLSRGWConcurrentIO and subclasses

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore/bluefs: Fix race condition between truncate() and unlink()

It was possible for unlink() to interrupt ongoing truncate().
As the result, unlink() finishes properly, but truncate() is not aware
of it and does:
1) updates file that is already removed
2) releases same allocations again

Now fixed by checking if file is deleted under FILE lock.

https://tracker.ceph.com/issues/70747

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

rgw/rados: index operations use async_reads/writes()

replace the classes derived from CLSRGWConcurrentIO with classes
derived from Reader/Writer/RevertibleWriter and use the async algorithms

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: add concurrent io algorithms for sharded data

cls/rgw provides the base class CLSRGWConcurrentIO as a swiss army knife
for bucket index operations that visit every shard object. while it uses
asynchronous librados requests to perform the io, it blocks on a
condition variable when waiting for the AioCompletions.

for use in coroutines, we need a version of this that suspends instead
of blocking. and to support both stackful and stackless coroutines, we
want a fully-generic async inferface templated on CompletionToken.

while the CLSRGWConcurrentIO algorithm works for all current uses
(reads and writes, with/without retries, with/without cleanup), i chose
to break this into 3 algorithms with well-defined semantics:

1. reads: to produce a successful result, all shard operations must
   succeed. so any shard's failure causes the rest to be cancelled or
   skipped. supports retries for ListBucket (RGWBIAdvanceAndRetryError).
2. writes: even if some shards fail, we still want to visit every shard
   before returning the error. supports retries for log trimming
   operations (repeat until ENODATA).
3. revertible writes: similar to reads, requires all shard operations to
   succeed. on any failure, the side effects of any successful writes
   must be reverted before returning. only used by IndexInit (any created
   shards are removed on failure).

each algorithm provides a pure virtual base class that must be
implemented for each type of operation, similar to how existing
operations inherit from CLSRGWConcurrentIO.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: pass optional_yield to all index ops

prepare for coroutine support by making the yield context available to
all index ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62571 from clwluvw/trim-log-cleanup

rgw: BucketTrimShardCollectCR expect ENODATA as empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61966 from mkogan1/wip-qat-logs-crypto

rgw: qat: improve QAT logs when HW is miss-configured for QAT encryption

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mailmap: update affiliation for Kefu Chai

he's not affiliated to Red Hat anymore, so map his contribution to
his private email address.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr/dashboard: Fix empty ceph version in GET api/hosts

Fixes https://tracker.ceph.com/issues/70821

Due to the pagination the host list is being fetched from orchestrator which caused a regression as via orchestrator list ceph version is always marked empty.
Caused by https://github.com/ceph/ceph/pull/52154

Also fixed tests , as the new version addition causing whole json object mock to fail in tests

Signed-off-by: Afreen Misbah <afreen@ibm.com>

test/librbd/test_notify.py: drop RBD_DISABLE_UPDATE_FEATURES

This was put in place in commit 9c0b239d70cd ("qa/upgrade:
conditionally disable update_features tests") to paper over a backwards
compatibility issue that arose from commit 01ff1530544c ("librbd: make
all maintenance op notifications async").  It's not needed in squid or
later because upgrades from octopus are tested only until reef.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/test_notify.py: force line-buffered output

"master" and "slave" invocations are intended to run in parallel and
coordinate between themselves.  Ensure that their respective output is
properly timestamped and ordered in teuthology.log file.

Fixes: https://tracker.ceph.com/issues/70831
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #62635 from Matan-B/wip-matanb-crimson-ignore-abort

test/crimson/test_errorator: showcase assert_all/handle/pass_furhter

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #62691 from Matan-B/wip-matanb-crimson-cronjobs

qa/crontab: Update Crimson's Cron Jobs

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #62535 from Matan-B/wip-matanb-crimson-replicated-missing

osd/../client_request: Fix clone replicated reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #62640 from ivancich/wip-admin-robust-rm-obj

rgw: add force option to `radosgw-admin object rm ...`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62679 from rhcs-dashboard/add-nvmeof-grafana-dashboards

mgr/dashboard: add nvmeof performance and overview grafana dashboard to Block -> Nvme/TCP tab in dashboard

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #62563 from tchaikov/osd-atomic

osd: replace deprecated atomic_store with std::atomic<shared_ptr>

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62038 from phlogistonjohn/jjm-cephadm-infer-69278

cephadm: update and improve container image infer function

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60890 from rzarzynski/wip-os-fastomapiter-killomapiterator

osd, os: drop the OmapIterator in favor of omap_iterate. Remove ObjectMap

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

rgw/rados: remove duplicated RGWRados::cls_bucket_head()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: move trim_reshard_log() from RGWRados to RGWSI_BucketIndex_RADOS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: move RGWRados::bucket_set_reshard() to RGWSI_BucketIndex_RADOS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: remove unused RGWReshard::clear_bucket_resharding()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/multisite: fix parsing of 'topic list' output

adjust the parsing to match changes from 3f0428a017cdf35b7290cf7836eb4baa4eb0e774

Fixes: https://tracker.ceph.com/issues/70729
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62596 from rhcs-dashboard/rgw-url

mgr/dashboard: introduce dashboard setting to resolve rgw hostname

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

test/crimson/test_errorator: showcase assert_all/handle/pass_furhter

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62649 from Matan-B/wip-matanb-crimson-coroutines-errors

test/crimson/test_crimson_coroutines: showcase errorator examples

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #62080 from xxhdx1985126/wip-70180

crimson/osd/pg_recovery: use OperationThrottler to throttle object pushes/pulls

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: add nvmeof performance and overview grafana dashboards to Block -> Nvme/TCP tab in dashboard

Fixes: https://tracker.ceph.com/issues/70793
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #62687 from anthonyeleven/acronym

doc/cephadm/services: improve rgw.rst and snmp-gateway.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm/services: improve rgw.rst and snmp-gateway.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

qa/crontab: Run Crimson main branch twice a week.

Crimson's suite is reletavly limited and currently is run for
main only (not for prior releases).
Changes to Crimson are more delicate and having more main runs
to compare to might help with (git-)bisecting issues

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/crontab: Don't run squid branch for Crimson

Crimson is only supported in main currently (and T in once relevant).

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53799 from smanjara/wip-shilpa-cleanup-bi

rgw/multisite: bucket instance cleanup after deletion

tools: refactor std::tie with structured binding in COT

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: use std::string::starts_with() in PGLog::read_log_and_missing()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch two complex debugs to fmt::format in SnapMapper::run()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os: drop ObjectMap

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os: drop DBObjectMap

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tools: drop ceph-osdomap-tool

Please don't confuse with osdmaptool.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd, os: drop ObjectStore::get_omap_iterator in favor of omap_iterate

Please note this also drops a BlueStore's perf counter:
`l_bluestore_omap_seek_to_first_lat`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tools: switch COT to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch rest of PrimaryLogPG to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test: switch StoreTest out of get_omap_iterator

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tools/cot: don't print tail_key during list-slow-omap

`tail_key`, and even presence of some key designated to
be a gaaurd, seems to an impementation detail of
particular `ObjectStore`.

I doubt this piece of information is useful enough to
introduce a getter for it to the `ObjectStore` interface.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch SnapMapper to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os, os/memstore: make the omap_iterate reentrant

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: internalize the SnapMapper's dependency on OmapIterator

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch OSDriver of SnapMapper to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd, test: drop the pre-octopus keys conversion of SnapMapper

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch PGLog::read_log_and_missing to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os, os/bluestore: omap_iterate informs whether it has been stopped

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch OMAP scrubbing to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: ReplicatedBackend::build_push_op switch to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: prepare ScrubMapBuilder::omap_hash for optimization

When `get_omap_iterator()` is replaced with `omap_iterate()`,
the CRC calculations will be happening in-place, with 0copy.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw/multisite: account for zones opted out of sync and object filters
during bucket deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: remove unused objv tracker

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: check for both zonegroup and bucket level sync policies
when returning enotempty during bucket deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/tests: test for bucket enotempty check during delete

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/trim: restore update_mapping_update in RGWBucketInstanceMetadataHandler::remove
- fix testcase

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/trim: fix ENOENT return response from bucket sync status query.
only handle them when the bucket metadata is deleted. there is a case
when we get enoent when status objects have not been created yet,
for example when bucket metadata is created and synced but no data
exists yet and bucket sync status won't be initialized. these don't
need special handling.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: test trimming bilog on all zonegroups

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: test with checkpoint delays

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: don't sync metadata entry for bucket instance removal.
removal should happen locally

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: thread svc_datalog through RGWBucketCtl and BucketInstanceMetadataHandler

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multiiste: fix bucket enotempty check

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: move the remote bucket listing logic into a separate
function.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: in a multisite env with bucket sync policies configured,
we may end up orphaning objects on remote zones when a delete bucket
is issued on metadata master. to avoid this, list the buckets on remote
zones and delete bucket only when empty. if a zone is unreachable we
drop that zone and continue with bucket deletion. such zones might have
orphaned objects that will have to be cleaned up using radosgw-admin tool

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: update test for bucket instance deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: - add deleted layout on master zone
- handle the case when StatusShards returns -ENOENT

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: add a bilog trim testcase with bucket deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: add datalog entry when transitioning to 'DELETED' log layout

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: helper function to store bucket info flag

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: retry on detecting bucket instance metadata racing writes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: - don't overwrite bucket attrs while storing bucket instance info.
- get rid of the helper function that adds the 'deleted' log layout.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: add 'deleted' as a RGWBucketInfo flag

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: handle the 'deleted' index log addition in RGWBucketInstanceMetadataHandler.
create an async cr for removing bucket instance info in bilog trimming logic

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: draft of bucket index cleanup after deletion

tracker: https://tracker.ceph.com/issues/20802
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #62637 from smanjara/wip-rgw-revert-62323

rgw/multisite: revert pr 62323

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62680 from tchaikov/cmake-deprecated-warning

cmake: Fix googletest deprecated warnings by using target_compile_options()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62511 from tchaikov/test-remove-unused-var

test/ObjectMap: Remove unused CONTROL_PREFIX variable

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: drop dead code that depends on OmapIterator::status()

For `BlueStore`, `MemStore` and `KStore` the `status()`
unconditionally returns `0`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

DNB, os/bluestore: minimize size of critical section in omap_iterate()

This commit takes out both seeing and iterating over onode's
OMAP out of the `Collection::lock`.

It bases on work by Adam Kupczyk on the behavior of RocksDB::DBIter
maintaning consistent view despite ongoing writes. Kudos to Adam.

This is experimental commit; I'm adding it for testing early.
DNB -- Do Not Backport.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>