RGW: Add observability over the persistent topics queue

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #51283 from rhcs-dashboard/disable-delete

mgr/dashboard: disable delete on multisite

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #51928 from samarahu/cmake-cleanup

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #52149 into main

* refs/pull/52149/head:
cephfs-journal-tool: disambiguate usage of all keyword (in tool help).

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52155 from nbalacha/poolreplayer-logging

rbd-mirror: add additional logs to PoolReplayer

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #51417 into main

* refs/pull/51417/head:
client: only wait for write MDS OPs when unmounting
client: flush the mdlog before waiting for unsafe requests

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #50908 into main

* refs/pull/50908/head:
mon/MDSMonitor: plug paxos when maybe manipulating the osdmap
mon/MDSMonitor: force immediate propose when evicting an MDS

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #49295 into main

* refs/pull/49295/head:
mgr/snap_schedule: catch all exceptions for cli

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51845 from ssdohammer-sl/doc-update-deduplication

doc:dev update how to use deduplication

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc:dev update how to use deduplication

Add prerequisite for deduplication, example outputs, and chunk-repair operation.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>

cephfs-journal-tool: disambiguate usage of all keyword (in tool help).

        The fs:all for rank option description was confusing. It seemd
like the fs was optional, but it is mandatory. This change modifies the
help message to reflect the correct way to use all in the --rank option.

Fixes: https://tracker.ceph.com/issues/61753
Signed-off-by: Manish M Yathnalli <myathnal@redhat.com>

Merge pull request #52181 from zdover23/wip-doc-2023-06-25-radosgw-s3select-formatting

doc/radosgw: remove pipes from s3select.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52110 from yuvalif/wip-yuval-fix-61639

rgw/amqp: remove possible race conditions with the amqp connections

doc/radosgw: remove pipes from s3select.rst

Remove strange formatting introduced in
61d6813726fc08980903c6a6c50b0a5797ba7193.

Note to the documentation team: This is not a line-edit. This commit
includes nothing but the removal of pipes added to the left of much of
the text in this file. Several future commits will line-edit this file
and correct its formatting.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/amqp: use "fmt" for converting connection_id to string

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51980 from aisakaki/wip-fix-check_node

crimson/os/seastore: fix in check_node

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: fix bug in check_node

EXIST_CLEAN and EXIST_MUTATION_PENDING shuold not be treated as
CLEAN in check_node because they are transaction private and the
leafnode has been duplicated for write.

fix: https://tracker.ceph.com/issues/61626

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>

Merge pull request #52167 from zdover23/wip-doc-2023-06-23-rados-operations-crush-map-edits-1-of-x

doc/rados: edit crush-map-edits.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: edit crush-map-edits.rst (1 of x)

Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

rbd-mirror: add logs to PoolReplayer

This commit adds logs to some PoolReplayer functions
in order to make debugging easier.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge PR #51974 into main

* refs/pull/51974/head:
doc: fix grammar in cephfs/standby

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #52164 from ljflores/wip-balancer-documentation-fix

doc/rados/operations: change file back to original name

doc/rados/operations: change file back to original name

The name of this file was changed in
4fcab2e7fc9f7ac170ede21cb07912b79926ccb9,
but on second thought, this could cause 404 situations.
Reverting the file name back to the original name.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #52156 from cbodley/wip-qa-rgw-s3tests-extra-attrs

qa/rgw/s3tests: make extra_attrs additive

Merge pull request #52085 from alimaredia/wip-rgw-tempest-keystone-fix

Tempest tests Keystone Fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: rgw/tempest suite only runs on ubuntu latest

Signed-off-by: Ali Maredia <amaredia@redhat.com>

qa: update rgw openstack versions

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #51738 from phlogistonjohn/jjm-cephadm-new-deploy

cephadm: create a new version of the deploy command

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51499 from ljflores/wip-balancer-documentation

doc/man/8: add documentation about read balancer

doc/man/8: add documentation about read balancer

Signed-off-by: Laura Flores <lflores@redhat.com>
Co-authored-by: Josh Salomon <josh.salomon@gmail.com>

Merge pull request #51819 from mkogan1/wip-d3n-obj-w-slash

rgw: D3N: cache objects which oid contains slash

qa/s3tests: remove stale comment about rfc2616

this comment was about the old fails_strict_rfc2616 attr that was
removed in 6238eaaf0bcbcaac3c8628f68c088481f96c199a

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: clean up extra_attrs and extra_args logic

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: make extra_attrs additive

the s3tests.py task is filtering out several attrs by default. but
when dbstore uses `extra_attrs` to add 'not fails_on_dbstore', it
overwrites those other filters

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52129 from yuvalif/wip-yuval-kafka-memleak

rgw/kafka: fix potential memory leak when push to queue fails

Merge pull request #51961 from Matan-B/wip-matanb-crimson-single-mosdmap-handling

Crimson/osd: Disable concurrent MOSDMap handling

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #52072 from NitzanMordhai/wip-nitzan-add-ec-clay-teuthology-tests

qa: adding clay test to thrash erasure code big

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #51667 from Matan-B/wip-matanb-share-osdmap

osd/OSD: cleanup OSDService::maybe_share_map()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51450 from batrick/i59716

pybind/rados: keep byte representation if decode fails

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #50206 from TRYTOBE8TME/wip-rgw-yield-work

rgw: replace null_yields with optional ones

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/cephadm: update script use use cephadm _orch deploy

cephadm _orch deploy will eventually be the only deploy method so
we need to update the test script.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: dont set ctx.image if json deploy image is unset

If no image has been provided in the "deploy from" json do not set
ctx.image to it (empty-string or None) as we may have had a valid
value passed on the --image CLI option.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52123 from zdover23/wip-doc-2023-06-20-rados-operations-control-1-of-x

doc/rados: edit control.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit control.rst (1 of x)

Edit doc/rados/operations/control.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52109 from idryomov/wip-61616

librbd: avoid object map corruption in snapshots taken under I/O

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw/amqp: remove possible race conditions with the amqp connections

* simplify memory management of the connection by not using a unique_ptr
* simplify the logic by handling all issues inside the amqp manager
* fix iterator invalidation issue with miltiple n/acks
* allow different connections with different exchanges
* modify the unit tests according to the new behavior

Fixes:
* https://tracker.ceph.com/issues/61639
* https://tracker.ceph.com/issues/46127

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

qa/workunits/rbd: make continuous export-diff test actually work

The current version is pretty useless:

- "rbd bench" writes the same byte (0xff) over and over again, so
  almost all checksumming is in vain
- snapshots are taken in a steady state (i.e. not under I/O), so no
  race conditions can get exposed
- even with these caveats, it's not wired up into the suite

Redo this workunit to be a reliable reproducer for the issue fixed
in the previous commit and wire it up for both krbd and rbd-nbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson/osd/osd_operations/pg_advance_map: 'from' epoch may vary between PGS explanation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osdmap_gate: got_map() assert increase of current epoch

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51827 from liu-chunmei/get_assert

crimson:  fix RefCountedObject::_get() assert

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/shard_services: get_or_create_pg() remove unused variable

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/shard_services: Refactor OSDSingletonState::load_map

Added logs
If case reconstructed - epoch is unsigned.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_meta: Abort instead of throwing runtime_error

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: Add logs around handling of osdmaps

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: Fix assert of osd_debug_verify_cached_snaps

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52104 from anthonyeleven/anthonyeleven/fixfft

doc/rados/operations: Improve formatting in crush-map.rst

doc/rados/operations: Improve formatting in crush-map.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

rgw/kafka: fix potential memory leak when push to queue fails

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #51500 into main

* refs/pull/51500/head:
test/libcephfs: add test case for revoking caps
client: issue a cap release immediately if no cap exists
mds: add the revoking caps back to _revokes list
mds: move confirm_receipt() to Capability.cc

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #51655 into main

* refs/pull/51655/head:
mds: display sane hex value (0x0) for empty feature bit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

src/rgw: Additional changes based on comments

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Changing null_yield in rgw_sal_rados.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Changing null_yield in rgw_reshard.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Passing optional_yield to rgw_rados.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rados: Changing null_yield to optional_yield

Passing optional_yield at all places in rgw_bucket.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Focused on rgw_rados_operate() and passing null_yield in
rgw_lc.cc, rgw_gc.cc, rgw_object_expirer_core.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw: pubsub: pass yield context

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rados: pass yield context in a few places

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #51681 from cbodley/wip-61349

librados: aio operate functions can set mtimes

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #51737 from cbodley/wip-rocksdb-44

bump rocksdb submodule to latest ceph-reef-v7.9.2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #51669 from ronen-fr/wip-rf-max-scrub-3

osd/scrub: increasing max_osd_scrubs to 3

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #48209 from kamoltat/wip-ksirivad-fix-tracker-57570

osd/OSDMap: Check for uneven weights & != 2 buckets post stretch mode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: D3N cache objects which oid contains slash

Fixes: https://tracker.ceph.com/issues/59739
Signed-off-by: Mark Kogan <mkogan@redhat.com>

mds: display sane hex value (0x0) for empty feature bit

Print a valid hex (0x0) during empty feature bit, so that the clients could recognize it.
When the _vec size becomes 0, print() function creates an invalid hex (0x) and 'perf stats'
crashes with the below error:
"
File "/opt/ceph/src/pybind/mgr/stats/fs/perf_stats.py", line 177, in notify_cmd
metric_features = int(metadata[CLIENT_METADATA_KEY]["metric_spec"]["metric_flags"]["feature_bits"], 16)
ValueError: invalid literal for int() with base 16: '0x'
"
This patch creates  a valid hex (0x0), when _vec size is 0.

Fixes: https://tracker.ceph.com/issues/59551
Signed-off-by: Jos Collin <jcollin@redhat.com>

mgr/dashboard: disable delete on multisite

Fixes: https://tracker.ceph.com/issues/59441
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

librbd: stop passing IOContext to image dispatch write methods

This is a major footgun since any value passed e.g. at the API layer
may be stale by the time we get to object dispatch.  All callers are
passing the IOContext returned by get_data_io_context() for their
ImageCtx anyway, highlighting that the parameter is fictitious.

Only the read method can meaningfully take IOContext.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: use an up-to-date snap context when owning the exclusive lock

By effectively moving capturing of the snap context to the API layer,
commit 1d0a3b17f590 ("librbd: pass IOContext to image-extent IO
dispatch methods") introduced a nasty regression.  The snap context can
be captured only after exclusive lock is safely held for the duration
of dealing with the image request and even then must be refreshed if
a snapshot creation request is accepted from a peer.  This is needed to
ensure correctness of the object map in general and fast-diff states in
particular (OBJECT_EXISTS vs OBJECT_EXISTS_CLEAN) and object deltas
computed based off of them.  Otherwise the object map that is forked
for the snapshot isn't guaranteed to accurately reflect the contents of
the snapshot when the snapshot is taken under I/O (as in disabling the
object map may lead to different results being returned for reads).

The regression affects mainly differential backup and snapshot-based
mirroring use cases with object-map and/or fast-diff enabled: since
some object deltas may be incomplete, the destination image may get
corrupted.

This commit represents a reasonable minimal fix: IOContext passed
through to ImageDispatch is effected only for reads and just gets
ignored for writes.  The next commit cleans up further by undoing the
passing of IOContext through the image dispatch layers for writes.

Fixes: https://tracker.ceph.com/issues/61616
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52105 from gsantella/patch-update-typo-on-dashboard

doc/mgr/dashboard: fix a typo

doc/mgr/dashboard: fix a typo

Signed-off-by: Guido Santella <gsantella@southhills.edu>

Merge pull request #52100 from zdover23/wip-doc-2023-06-17-rados-operations-crush-map-firefly-tunables

doc/rados: edit firefly tunables section

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: edit firefly tunables section

Edit the firefly tunables section in
/doc/rados/operations/crush-map.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52092 from zdover23/wip-doc-2023-06-16-rados-operations-crush-map-4-of-x

doc/rados: edit crush-map.rst (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit crush-map.rst (4 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50874 from rishabh-d-dave/mds-log-variable

mds: use variable g_ceph_context directly in MDSAuthCaps

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52095 from rhcs-dashboard/jsonschema-fix

mgr/dashboard: fix the jsonschema issue in install-deps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #51549 from vedanshbhartia/coverity_1517658

rgw: modify RGWRados::cls_bucket_list_ordered to store dirent key before move

Merge pull request #51719 from vedanshbhartia/coverity_1510555

rgw: Use a unique_ptr when creating a D3nCacheAioWriteRequest instance

Merge pull request #52076 from rhcs-dashboard/cephfs-no-mds-clients

mgr/dashboard: empty list of fs clients with no mds

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52089 from samarahu/d4n-cmake-fix

RGW: Place D4N files under cmake conditional

mds: use variable g_ceph_context directly in MDSAuthCaps

Variable g_ceph_context is global, therefore use it directly instead of
passing it as a parameter to method.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52010 from pkalever/fix-ImageMap

rbd-mirror: don't refer to (remote) mirror_uuid as peer_uuid

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51132 from lxbsz/wip-59349

qa: wait for 100 seconds to make sure the quota to be enforced

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard: fix the jsonschema issue in install-deps

Fixes: https://tracker.ceph.com/issues/61690
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52067 from zdover23/wip-doc-2023-06-15-rados-operations-crush-map-3-of-x

doc/rados: edit crush-map.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit crush-map.rst (3 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: empty list of fs clients with no mds

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>
Fixes: https://tracker.ceph.com/issues/61708

qa: adding clay test to thrash erasure code big
Currently we don't have any clay test in the erasure code big tests
adding also clay tests.

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

rbd-mirror: don't refer to (remote) mirror_uuid as peer_uuid

peer_uuid and mirror_uuid are different identifiers, we should not use
peer_uuid naming for remote mirror_uuid.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

RGW: Place D4N files under cmake conditional

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

cephadm: remove call to get_parm from fetch_configs

Stop using get_parm in fetch_configs. Doing so makes clear that
the two if-branches in fetch_configs are symmetric, in the
handling of custom_config_files and symmetric with the behavior
of fetch_custom_config_files.
It also reveals that get_parm only has one remaining caller making it
simpler to remove get_parm in the future.

Signed-off-by: John Mulligan <jmulligan@redhat.com>