mgr/dashboard: Add RGW topic listing in dashboard

Fixes: https://tracker.ceph.com/issues/69143
Signed-off-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #62716 from afreen23/wip-fix-empty-ver

mgr/dashboard: Fix empty ceph version in GET api/hosts

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #62571 from clwluvw/trim-log-cleanup

rgw: BucketTrimShardCollectCR expect ENODATA as empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61966 from mkogan1/wip-qat-logs-crypto

rgw: qat: improve QAT logs when HW is miss-configured for QAT encryption

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Fix empty ceph version in GET api/hosts

Fixes https://tracker.ceph.com/issues/70821

Due to the pagination the host list is being fetched from orchestrator which caused a regression as via orchestrator list ceph version is always marked empty.
Caused by https://github.com/ceph/ceph/pull/52154

Also fixed tests , as the new version addition causing whole json object mock to fail in tests

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #62635 from Matan-B/wip-matanb-crimson-ignore-abort

test/crimson/test_errorator: showcase assert_all/handle/pass_furhter

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #62691 from Matan-B/wip-matanb-crimson-cronjobs

qa/crontab: Update Crimson's Cron Jobs

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #62535 from Matan-B/wip-matanb-crimson-replicated-missing

osd/../client_request: Fix clone replicated reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #62640 from ivancich/wip-admin-robust-rm-obj

rgw: add force option to `radosgw-admin object rm ...`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62679 from rhcs-dashboard/add-nvmeof-grafana-dashboards

mgr/dashboard: add nvmeof performance and overview grafana dashboard to Block -> Nvme/TCP tab in dashboard

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #62563 from tchaikov/osd-atomic

osd: replace deprecated atomic_store with std::atomic<shared_ptr>

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62038 from phlogistonjohn/jjm-cephadm-infer-69278

cephadm: update and improve container image infer function

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60890 from rzarzynski/wip-os-fastomapiter-killomapiterator

osd, os: drop the OmapIterator in favor of omap_iterate. Remove ObjectMap

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #62596 from rhcs-dashboard/rgw-url

mgr/dashboard: introduce dashboard setting to resolve rgw hostname

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

test/crimson/test_errorator: showcase assert_all/handle/pass_furhter

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62649 from Matan-B/wip-matanb-crimson-coroutines-errors

test/crimson/test_crimson_coroutines: showcase errorator examples

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #62080 from xxhdx1985126/wip-70180

crimson/osd/pg_recovery: use OperationThrottler to throttle object pushes/pulls

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: add nvmeof performance and overview grafana dashboards to Block -> Nvme/TCP tab in dashboard

Fixes: https://tracker.ceph.com/issues/70793
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #62687 from anthonyeleven/acronym

doc/cephadm/services: improve rgw.rst and snmp-gateway.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm/services: improve rgw.rst and snmp-gateway.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

qa/crontab: Run Crimson main branch twice a week.

Crimson's suite is reletavly limited and currently is run for
main only (not for prior releases).
Changes to Crimson are more delicate and having more main runs
to compare to might help with (git-)bisecting issues

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/crontab: Don't run squid branch for Crimson

Crimson is only supported in main currently (and T in once relevant).

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53799 from smanjara/wip-shilpa-cleanup-bi

rgw/multisite: bucket instance cleanup after deletion

tools: refactor std::tie with structured binding in COT

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: use std::string::starts_with() in PGLog::read_log_and_missing()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch two complex debugs to fmt::format in SnapMapper::run()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os: drop ObjectMap

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os: drop DBObjectMap

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tools: drop ceph-osdomap-tool

Please don't confuse with osdmaptool.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd, os: drop ObjectStore::get_omap_iterator in favor of omap_iterate

Please note this also drops a BlueStore's perf counter:
`l_bluestore_omap_seek_to_first_lat`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tools: switch COT to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch rest of PrimaryLogPG to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test: switch StoreTest out of get_omap_iterator

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tools/cot: don't print tail_key during list-slow-omap

`tail_key`, and even presence of some key designated to
be a gaaurd, seems to an impementation detail of
particular `ObjectStore`.

I doubt this piece of information is useful enough to
introduce a getter for it to the `ObjectStore` interface.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch SnapMapper to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os, os/memstore: make the omap_iterate reentrant

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: internalize the SnapMapper's dependency on OmapIterator

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch OSDriver of SnapMapper to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd, test: drop the pre-octopus keys conversion of SnapMapper

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch PGLog::read_log_and_missing to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os, os/bluestore: omap_iterate informs whether it has been stopped

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch OMAP scrubbing to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: ReplicatedBackend::build_push_op switch to omap_iterate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: prepare ScrubMapBuilder::omap_hash for optimization

When `get_omap_iterator()` is replaced with `omap_iterate()`,
the CRC calculations will be happening in-place, with 0copy.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw/multisite: account for zones opted out of sync and object filters
during bucket deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: remove unused objv tracker

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: check for both zonegroup and bucket level sync policies
when returning enotempty during bucket deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/tests: test for bucket enotempty check during delete

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/trim: restore update_mapping_update in RGWBucketInstanceMetadataHandler::remove
- fix testcase

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/trim: fix ENOENT return response from bucket sync status query.
only handle them when the bucket metadata is deleted. there is a case
when we get enoent when status objects have not been created yet,
for example when bucket metadata is created and synced but no data
exists yet and bucket sync status won't be initialized. these don't
need special handling.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: test trimming bilog on all zonegroups

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: test with checkpoint delays

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: don't sync metadata entry for bucket instance removal.
removal should happen locally

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: thread svc_datalog through RGWBucketCtl and BucketInstanceMetadataHandler

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multiiste: fix bucket enotempty check

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: move the remote bucket listing logic into a separate
function.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: in a multisite env with bucket sync policies configured,
we may end up orphaning objects on remote zones when a delete bucket
is issued on metadata master. to avoid this, list the buckets on remote
zones and delete bucket only when empty. if a zone is unreachable we
drop that zone and continue with bucket deletion. such zones might have
orphaned objects that will have to be cleaned up using radosgw-admin tool

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: update test for bucket instance deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: - add deleted layout on master zone
- handle the case when StatusShards returns -ENOENT

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: add a bilog trim testcase with bucket deletion

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: add datalog entry when transitioning to 'DELETED' log layout

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: helper function to store bucket info flag

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: retry on detecting bucket instance metadata racing writes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: - don't overwrite bucket attrs while storing bucket instance info.
- get rid of the helper function that adds the 'deleted' log layout.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: add 'deleted' as a RGWBucketInfo flag

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: handle the 'deleted' index log addition in RGWBucketInstanceMetadataHandler.
create an async cr for removing bucket instance info in bilog trimming logic

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: draft of bucket index cleanup after deletion

tracker: https://tracker.ceph.com/issues/20802
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #62637 from smanjara/wip-rgw-revert-62323

rgw/multisite: revert pr 62323

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62680 from tchaikov/cmake-deprecated-warning

cmake: Fix googletest deprecated warnings by using target_compile_options()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62511 from tchaikov/test-remove-unused-var

test/ObjectMap: Remove unused CONTROL_PREFIX variable

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: drop dead code that depends on OmapIterator::status()

For `BlueStore`, `MemStore` and `KStore` the `status()`
unconditionally returns `0`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

DNB, os/bluestore: minimize size of critical section in omap_iterate()

This commit takes out both seeing and iterating over onode's
OMAP out of the `Collection::lock`.

It bases on work by Adam Kupczyk on the behavior of RocksDB::DBIter
maintaning consistent view despite ongoing writes. Kudos to Adam.

This is experimental commit; I'm adding it for testing early.
DNB -- Do Not Backport.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os: deprecate ObjectStore::get_omap_iterator()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: Fix googletest deprecated warnings by using target_compile_options()

Previously, we attempted to disable deprecated declarations warnings when
building gtest by adding `-Wno-deprecated-declarations` to the COMPILE_OPTIONS
property of the googletest directory. However, this approach failed to apply
the option when actually building gtest.

This change applies the compile option directly to the `gtest` target using
target_compile_options() instead. Verified by forcing the condition to TRUE
and confirming the option is included when building `gtest-all.cc` through
`cmake --build ~/dev/ceph/build --target gtest --verbose`.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/osd/pg_recovery: use OperationThrottler to throttle object
pushes/pulls

Instead of throttling recovery/backfill operations

Fixes: https://tracker.ceph.com/issues/70180
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #61751 from VallariAg/wip-nvmeof-grafana-config

monitoring: add grafana nvmeof dashboard

Merge pull request #62661 from anthonyeleven/improve-62597

doc/radosgw: Improve cloud-restore and cloud-transition

doc/radosgw: Improve cloud-restore and cloud-transition

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Revert "rgw: reject PutBucketReplication on mismatched versioning and lock"

This reverts commit 7e47deede0ba672dab8d2a26c807d8a32d46a73a.
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Revert "rgw: skip versioned entries to non-versioned buckets and vice-versa"

This reverts commit 2c4b718162198e6db679113ce3196b391d8571e5.
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Revert "rgw: skip replication from lock enabled buckets to lock disbaled buckets"

This reverts commit 826d49368961677f05bc53fd1e4b6cc999057cdd.
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: clean-up of some code

Comments added and formatting adjusted.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add doc & doc testing for force option on `radosgw-admin object rm ...`

Subject says it all.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add force option to `radosgw-admin object rm ...`

The `radosgw-admin object rm ...` sub-command will give up if it
determines that there's an issue with the head object. This can make
it difficult for an admin to clean up a bucket index when there's a
damaged or missing head object.

When the user adds the "--yes-i-really-mean-it" command-line option,
it enables the "force mode". The bucket index entry(ies) will be
removed. If the object being removed is the current version in a
versioned bucket, the appropriate changes to the OLH will take place.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #62597 from soumyakoduri/wip-skoduri-restore-doc

rgw: Updating doc for cloud-restore feature

Merge pull request #62524 from ivancich/wip-fix-gap-list-tool

rgw: fix bug with rgw-gap-list

Reviewed-by: Jesse Williamson <jfw@ibm.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56499 from adamemerson/wip-rgw-close-datalog-write-hole

rgw: Close datalog write hole

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62583 from phlogistonjohn/jjm-refix-setsebool

qa/cephadm/smb: run setsebool with sudo

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #62639 from phlogistonjohn/jjm-doc-note-clustering-opt

doc/mgr: add a warning about the smb clustering option & placement

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Sachin Prabhu <sp@spui.uk>

Merge pull request #62604 from jzhu116-bloomberg/wip-70746

rgw: keep the tails when copying object to itself

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62325 from Matan-B/wip-matanb-with-crimson

src: sed -i 's/WITH_SEASTAR/WITH_CRIMSON/'

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/crimson/test_crimson_coroutines: showcase errorator examples

This came up in https://github.com/ceph/ceph/pull/62530.
Showcase an awaited throwing coroutine and a passed_further error.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

monitoring: Add nvmeof performance grafana dashboard

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

monitoring: Add nvmeof overview grafana dashboard

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

monitoring: make timeSeriesPanel default fillOpacity=8

Change default value of fillOpacity from 0 to 8 for
timeseries graphs.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

monitoring: add 'description' to timeSeriesPanel

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

monitoring: add 'noValue' config to garfana panels

Add noValue param to garfana panel functions:
- timeSeriesPanel
- addStatPanel

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #61804 from aainscow/erasure_code_chunks

osd: Modify erasure code plugins ready for new EC

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62233 from kamoltat/wip-ksirivad-ceph-s-leader-mon

mon: Add leader indicator && stretch mode to ceph -s output

Merge pull request #57806 from dongbula/wip-speed-oc-flush2

osdc/ObjectCacher: avoid io blocking for bufferheads exceed limit