qa/suites/rados/singleton/all: remove test_envlibrados_for_rocksdb

In rocksdb 7.0, all envlibrados files were moved to a separate repository (ref: https://github.com/facebook/rocksdb/pull/9206).
The new repo is temporary and serves as an example before it is finalized where and who to host RADOS support.

Since this new repo is outside of the rocksdb repo and in an unceratin state, we should remove support for it in main
and Reef test suites. Quincy and below still use rocksdb 6.0, so the same does not apply.

Fixes: https://tracker.ceph.com/issues/59057
Signed-off-by: Laura Flores <lflores@redhat.com>
(cherry picked from commit 2fca433c71965ffb13d89043d72029a33b563b06)

Merge pull request #52318 from mkogan1/wip-59739-reef

reef: rgw: D3N cache objects which oid contains slash

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52316 from mkogan1/wip-61661-reef

reef: rgw: d3n: fix valgrind reported leak related to libaio ops

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #52297 from cbodley/wip-61888-reef

reef: rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51910 from mkogan1/wip-58692-reef

reef: rgw: set bulk pool flag when creating data pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50936 from ifed01/wip-ifed-report-min-alloc-size-ree

reef: os/bluestore: report min_alloc_size through "ceph osd metadata"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52329 from soumyakoduri/wip-skoduri-reef

[reef]rgw: Do not update mtime in `set_attrs`

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #52321 from batrick/wip-61899-reef

reef: pybind: drop GIL during library callouts

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

PendingReleaseNotes: note rgw's compress-encrypted zonegroup feature flag

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fd2ec27f538bceadec412a848e2d29b187ed04bb)

doc/rgw: document compress-encrypted zonegroup feature flag

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d85011a7fc5d3df42a2c046ec7a43fe29f1db08f)

rgw: require zonegroup feature to compress encrypted objects

Reef adds support for combining server-side encryption and compression,
but previous releases would not replicate such objects correctly. gate
the feature behind a zonegroup feature flag to require that all zones
upgrade to reef before enabling. this feature is not enabled by default
in new zonegroups

Fixes: https://tracker.ceph.com/issues/57905
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4bc52954ab7b38a8eac50c116196d6db728cee00)

Conflicts:
src/rgw/rgw_op.cc: SiteConfig -> sal::ZoneGroup::supports

rgw/sal: add ZoneGroup::supports()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: don't capitalize zonegroup feature flags

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 39be699629df5c6a6b82fa956ed4c22fc4a38523)

rgw: don't enable all zone features by default

on zonegroup creation, all `supported` features get enabled by default.
however, some features should remain opt-in. add a separate list of
`enabled` features for the subset of features we want enabled by default
on zonegroup creation

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cdb1be836cc1b8c2e8f178bc00bad24d2b8413bf)

Conflicts:
src/rgw/driver/rados/rgw_zone.cc no create_default_zonegroup()

Merge pull request #52383 from zdover23/wip-doc-2023-07-11-backport-52366-to-reef

reef: doc/rados: edit add-or-rm-osds (1 of x)

doc/rados: edit add-or-rm-osds (1 of x)

Edit docs/rados/operations/add-or-rm-osds.

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a3ab5a7ad8a44c702089294b107d09dca3a98139)

Merge pull request #52381 from zdover23/wip-doc-2023-07-11-backport-52371-to-reef

reef: doc/rados: introduce emdash

doc/rados: introduce emdash

Introduce the emdash character in
docs/rados/troubleshooting/troubleshooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a971a3a66a93ce52898f201a41a9aa13e0dd8bfc)

Merge pull request #52333 from rhcs-dashboard/wip-61911-reef

reef: mgr/dashboard: fix a bug where data would plot wrongly

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52367 from zdover23/wip-doc-2023-07-09-backport-52327-to-reef

reef: doc: add information on expediting MDS recovery

doc: add information on expediting MDS recovery

Fixes: https://tracker.ceph.com/issues/61865
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0a15144c584d4445bd037340f053aa64b9c13f32)

Merge pull request #52361 from zdover23/wip-doc-2023-07-08-backport-52360-to-reef

reef: doc/radosgw: add Zonegroup policy explanation

Merge pull request #52226 from rzarzynski/wip-59102-reef

reef: msg/async: don't abort when public addrs mismatch bind addrs

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/radosgw: add Zonegroup policy explanation

Add revised Zonegroup policy for "multi-zonegroups". This commit
includes changes that Casey Bodley made in
https://github.com/ceph/ceph/pull/52324#discussion_r1253482258 and that
I have integrated into the docs only now.

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c6400ed79b1233a2238342e4931edf341f1c7b9f)

Merge pull request #52190 from ajarr/wip-61706-reef

reef: mgr/rbd_support: add user-friendly stderr message when module is not ready

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #52162 from alimaredia/wip-61773-reef

reef: Tempest tests Keystone Fix

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52347 from zdover23/wip-doc-2023-07-07-backport-52319-to-reef

reef: doc/foundation: Updating foundation members for July 2023

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52348 from zdover23/wip-doc-2023-07-07-backport-52324-to-reef

reef: doc/radosgw: add Zonegroup purpose

doc/radosgw: add Zonegroup purpose

Add a note about the telos of zonegroups. The information added in this
commit was originally sent to ceph-users by Casey Bodley on 30 Jun 2023
in response to a question from Yixin Jin. Alexander Patrakov then
suggested that Casey's response should be added to
doc/radsogw/multisite.rst.

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 5c46a1a00962918c885e434986d0f846c3ee4730)

doc/foundation: Updating foundation members for July 2023

Signed-off-by: Mike Perez <thingee@gmail.com>
(cherry picked from commit 640390906f76da4bfd66bcd8006e964d6e73c98e)

mgr/dashboard: fix a bug where data would plot on the graphs without converting to the data units of the graph

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
(cherry picked from commit 7aa70a16a54c3387b2e021763ca1d118d6785835)

rgw: Do not update mtime in `set_attrs`

As per https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html,
the only way for users to modify object metadata is to make a copy of the object and
set the metadata. Hence retain old mtime for any other attr changes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 31666d71aa5b58a9384938ba31892d48f2ac3f67)

pybind/rbd: drop GIL when calling into librbd

This was missing for rbd_mirror_peer_site_add() and
rbd_get_data_pool_id().

While at it, add a test for data_pool_id().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 1a60f66b0a13089c789e2e056b925ac6fef75021)

pybind/cephfs: drop gil during cephfs callouts

This has disastorous consequences including the possibility of deadlock.
In the best case, you have the rmdir holding the GIL until the MDS
responds!

Fixes: https://tracker.ceph.com/issues/61869
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5d249a4c978198f1e8974affac0297b71bebada5)

Merge pull request #52291 from zdover23/wip-doc-2023-07-03-backport-52280-to-reef-2nd-attempt

reef: doc/radosgw: refine "Zones" in multisite.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: refine "Zones" in multisite.rst

Describe in greater detail the function of Zones in serving S3 objects,
as requested by Alexander Patrakov here:
https://github.com/ceph/ceph/pull/49981#issuecomment-1615866374.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8dab6c066c4dd3699c7b7c80f7a30a0e18326bba)

reef: rgw: D3N cache objects which oid contains slash

Fixes: https://tracker.ceph.com/issues/59739
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit fcb91f3e2a6fd8066ac33950d421e0a9b9b4b490)

 Conflicts:
src/rgw/driver/rados/rgw_d3n_datacache.cc
src/rgw/rgw_aio.cc
src/rgw/rgw_d3n_cacherequest.h

reef: rgw: d3n: fix valgrind reported leak related to libaio ops

Fixes: https://tracker.ceph.com/issues/61661
The valgrind leak indication is a false positive in this case,
it is because the libaio internal thread have not timed out yet
when radosgw is terminated.

```
man aio_init
...
aio_idle_time
This field specifies the amount of time in seconds that a worker thread
should wait for further requests before terminating, after having
completed a previous request.   The
default value is 1.
...
```
for the sake of teuthology reducing the timeout

waiting for 2 minutes for example like below would also prevent the leak report
```
❯ env
LD_LIBRARY_PATH=/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/:$LD_LIBRARY_PATH
PYTHONPATH=$PYTHONPATH:/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/cython_modules/lib.3
RAGWEED_CONF=$(realpath ./ragweed.conf) RAGWEED_STAGES=prepare,check tox
-- -v |& ccze -Aonolookups ;   sleep 2m | pv -t ; pkill radosgw
```

Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit f834e10cfe97c7df8fc5ead682975fe60b435bd7)

Merge pull request #52298 from zdover23/wip-doc-2023-07-04-backport-51422-to-reef

reef: doc/rgw/lua: add info uploading a script in cephadm deployment

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw/lua: add info uploading a script in cephadm deployment

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 5cec4f719581e93760c22419e701c563a8688de9)

Merge pull request #52311 from zdover23/wip-doc-2023-07-05-backport-52207-to-reef

reef: doc/rados: edit crush-map-edits (2 of x)

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: edit crush-map-edits (2 of x)

Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ae04debe9e2fa18c1cb8745b0573ff56576fd3b0)

Merge pull request #52309 from zdover23/wip-doc-2023-07-05-backport-52308-to-reef

reef: doc/dev: remove seqdiag assets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: remove seqdiag assets

Remove seqdiag assets to determine whether the docs can be built if they
are absent. (Currently they cannot be built when they are present.) If
this works, then these diagrams will be replaced, probably with .png
files.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ce96ddd4c7da65351deaf14dc4f8b42f5148a352)

rgw: fetch_remote_obj() preserves RGW_ATTR_COMPRESSION of encrypted objects

if the source object was both compressed and encrypted, preserve its
original compression attribute so it can be decompressed the same way it
was originally compressed

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b91ff3fd86f52f7148daf7f9acc9864752186ce0)

rgw: fetch_remote_obj() will never verify etags of encrypted objects

fetch_remote_obj() transfers objects in their encrypted form, so does
not have access to the decrypted data for checksum verification

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ea45da747d5fa898b4aa4ec9223cd7b6983ca177)

rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

compression is applied before encryption. so if we skip decryption, we
can't decompress either

Fixes: https://tracker.ceph.com/issues/57905
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8d35759eb8edcc7cd15e6fd67b91da637fc477ae)

Merge pull request #52278 from zdover23/wip-doc-2023-07-01-backport-52277-to-reef

reef: doc/radosgw: s/s3select/S3 Select/

doc/radosgw: s/s3select/S3 Select/

Correct several misspelling of "S3 Select". Hat tip to Anthony D'Atri,
who caught this in an earlier PR.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 71c327ef4f56a6ec900a8c5ea071aaf8005a60a5)

Merge pull request #52187 from zdover23/wip-doc-2023-06-26-backport-52181-2nd-attempt-to-reef

reef: doc/radosgw: remove pipes from s3select.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52219 from zdover23/wip-doc-2023-06-28-backport-52194-to-reef

reef: doc/radosgw: edit "Overview" in s3select.rst

Merge pull request #52262 from zdover23/wip-doc-2023-06-30-backport-52250-to-reef

reef: doc/radosgw: edit "Basic Workflow" in s3select.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52260 from ceph/reef-release

v18.1.2

doc/radosgw: edit "Basic Workflow" in s3select.rst

Edit the "Basic Workflow" section in doc/radosgw/s3select.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4d2c09b683421552cfb4df7f467f2d9a6c9c7c26)

Merge pull request #52136 from ljflores/wip-61603-reef

reef: qa/suites/rados: whitelist POOL_APP_NOT_ENABLED for cls tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52135 from rzarzynski/wip-61742-reef

reef: mon/MonClient: before complete auth with error, reopen session

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge PR #50921 into reef

* refs/pull/50921/head:
qa: load file system info if not created

msg/async: don't abort when public addrs mismatch bind addrs

Before the 69b47c805fdd2ecd4f58547d58c9f019fc62d447 (PR #50153)
a mismatch (in number or types of stored `entity_addr_t`) between
public addrs and bind addrs vectors was ignored and the former
was taking over anything else -- it was possible to e.g. bind to
both v1 and v2 addresses but expose v2 only. Unfortunately, that's
exactly how Rook configures ceph-mon:

```
debug 2023-03-16T21:01:48.389+0000 7f99822bf8c0  0 starting mon.a rank 0 at public addrs v2:172.30.122.144:3300/0 at bind addrs [v2:10.129.2.21:3300/0,v1:10.129.2.21:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid acc14d1b-fb2b-4f01-8b61-6e7cb26e9200
```

The consequnece is the following abort:

```
ceph version 17.2.5-1338.el9cp (5adce3015143c7c2cc135a71368be194744f5761) quincy (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd3) [0x7ff05392c1a6]
2: /usr/lib64/ceph/libceph-common.so.2(+0x165ac1) [0x7ff05394eac1]
3: (AsyncMessenger::bindv(entity_addrvec_t const&, std::optional<entity_addrvec_t>)+0x1fe) [0x7ff053baa0ce]
4: main()
5: /lib64/libc.so.6(+0x3feb0) [0x7ff053048eb0]
6: __libc_start_main()
7: _start()
debug *** Caught signal (Aborted) **  in thread 7ff052ab18c0 thread_name:ceph-mon 2023-03-16T09:56:35.995+0000 7ff052ab18c0 -1 /builddir/build/BUILD/ceph-17.2.5-1338-g484e8dbb/src/msg/msg_types.h: In function 'void entity_addr_t::set_port(int)' thread 7ff052ab18c0 time 2023-03-16T09:56:35.996339+0000 /builddir/build/BUILD/ceph-17.2.5-1338-g484e8dbb/src/msg/msg_types.h: 359: ceph_abort_msg("abort() called")
```

This commit brings the original logic back but in a way that
preserves the port numbers figured out by. e.g. `Processor::bind`.

Fixes: https://tracker.ceph.com/issues/59100
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 68fbdcf9b0937e8102b86d0bf84815554b62328d)

doc/radosgw: edit "Overview" in s3select.rst

Edit the "Overview" section in doc/radosgw/s3select.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e051dd1a753614fc829f2054a69eda185f190db6)

18.1.2

Signed-off-by: Ceph Release Team <ceph-maintainers@ceph.io>

Merge pull request #51608 from vshankar/tr-61234

reef: mds: rdlock_path_xlock_dentry supports returning auth target inode

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #51606 from vshankar/tr-59723

reef: qa: run scrub post file system recovery

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #51515 from dparmar18/wip-61187-reef

reef: qa: ignore cluster warning when fs flag refuse_client_session is set

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #51041 from vshankar/tr-59412

reef: client, mds: update mtime and change attr for snapdir when snaps are created, deleted and renamed

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #51040 from vshankar/tr-59409

reef: mgr/volumes: avoid returning -ESHUTDOWN back to cli

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #50997 from lxbsz/wip-59404

reef: mds: force replay sessionmap version

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #50990 from lxbsz/wip-59399

reef: qa: enable kclient test for newop test

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #50987 from lxbsz/wip-59267

reef: client: clear the suid/sgid in fallocate path

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #50984 from lxbsz/wip-59384

reef: mds: wait for unlink operation to finish

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #52148 from idryomov/wip-61616-reef

reef: librbd: avoid object map corruption in snapshots taken under I/O

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #52116 from cbodley/wip-61729-reef

reef: librados: aio operate functions can set times

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #52041 from cbodley/wip-61665-reef

reef: qa: suppress SyscallParam error during startup on jammy

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/rbd_support: log number of images

... that have one snapshot request pending when the
mirror_snapshot_schedule handler is shutting down.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit edc3b0e80653c30d0fc318a24673e8a0568912ad)

mgr/rbd_support: add user-friendly stderr message

... when the rbd_support module is not ready.

Fixes: https://tracker.ceph.com/issues/61688
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 6351ef5c8e691e359b1bf913dde4dbc8a441be1d)

doc/radosgw: remove pipes from s3select.rst

Remove strange formatting introduced in
61d6813726fc08980903c6a6c50b0a5797ba7193.

Note to the documentation team: This is not a line-edit. This commit
includes nothing but the removal of pipes added to the left of much of
the text in this file. Several future commits will line-edit this file
and correct its formatting.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a4d302ee1a6e2f71263a898109ebed28e7e8eaca)

Merge pull request #52183 from zdover23/wip-doc-2023-06-25-backport-52181-to-reef

reef: doc/radosgw: remove pipes from s3select.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: remove pipes from s3select.rst

Remove strange formatting introduced in
61d6813726fc08980903c6a6c50b0a5797ba7193.

Note to the documentation team: This is not a line-edit. This commit
includes nothing but the removal of pipes added to the left of much of
the text in this file. Several future commits will line-edit this file
and correct its formatting.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a4d302ee1a6e2f71263a898109ebed28e7e8eaca)

Merge pull request #52179 from zdover23/wip-doc-2023-06-24-backport-52167-to-reef

reef: doc/rados: edit crush-map-edits.rst (1 of x)

doc/rados: edit crush-map-edits.rst (1 of x)

Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f447c290a877c84cb14f0141ff5330f7147b2f6a)

Merge pull request #52108 from mkogan1/wip-58911-reef

reef: RGW: Pass and check destination zone trace when fetching object

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52053 from trociny/wip-61432-reef

reef: rgw: multisite data log flag not used

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52119 from cbodley/wip-61380-reef

reef: Fix FTBFS on gcc 13

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #52168 from zdover23/wip-doc-2023-06-23-backport-45746-to-reef

reef: doc,test: clean up crush rule min/max_size leftovers

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc,test: clean up crush rule min/max_size leftovers

These fields were removed in quincy in commit f95eb04411cd ("crush:
eliminate min_size and max_size").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 1ab943cabb7677f9465e92a00a84d2bfca60beb7)

qa: rgw/tempest suite only runs on ubuntu latest

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 703effd45bf2c5d7cf8982e07e954a5f2cbb1902)

qa: update rgw openstack versions

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit fffa119424bad6e5c6b5ed436978d4319dd54a6b)

Merge pull request #52152 from zdover23/wip-doc-2023-06-22-backport-52123-to-reef

reef: doc/rados: edit control.rst (1 of x)

doc/rados: edit control.rst (1 of x)

Edit doc/rados/operations/control.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d14b828a1ac2f82525dce2cc0524ad83d2cdc80d)

qa/workunits/rbd: make continuous export-diff test actually work

The current version is pretty useless:

- "rbd bench" writes the same byte (0xff) over and over again, so
  almost all checksumming is in vain
- snapshots are taken in a steady state (i.e. not under I/O), so no
  race conditions can get exposed
- even with these caveats, it's not wired up into the suite

Redo this workunit to be a reliable reproducer for the issue fixed
in the previous commit and wire it up for both krbd and rbd-nbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit acb270a3dd07211f06aa11f8792a333f3a1b796d)

librbd: stop passing IOContext to image dispatch write methods

This is a major footgun since any value passed e.g. at the API layer
may be stale by the time we get to object dispatch.  All callers are
passing the IOContext returned by get_data_io_context() for their
ImageCtx anyway, highlighting that the parameter is fictitious.

Only the read method can meaningfully take IOContext.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 584f865ae116185087176715ced014b625f04e82)

librbd: use an up-to-date snap context when owning the exclusive lock

By effectively moving capturing of the snap context to the API layer,
commit 1d0a3b17f590 ("librbd: pass IOContext to image-extent IO
dispatch methods") introduced a nasty regression.  The snap context can
be captured only after exclusive lock is safely held for the duration
of dealing with the image request and even then must be refreshed if
a snapshot creation request is accepted from a peer.  This is needed to
ensure correctness of the object map in general and fast-diff states in
particular (OBJECT_EXISTS vs OBJECT_EXISTS_CLEAN) and object deltas
computed based off of them.  Otherwise the object map that is forked
for the snapshot isn't guaranteed to accurately reflect the contents of
the snapshot when the snapshot is taken under I/O (as in disabling the
object map may lead to different results being returned for reads).

The regression affects mainly differential backup and snapshot-based
mirroring use cases with object-map and/or fast-diff enabled: since
some object deltas may be incomplete, the destination image may get
corrupted.

This commit represents a reasonable minimal fix: IOContext passed
through to ImageDispatch is effected only for reads and just gets
ignored for writes.  The next commit cleans up further by undoing the
passing of IOContext through the image dispatch layers for writes.

Fixes: https://tracker.ceph.com/issues/61616
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit e4b1e0466354942c935e9eca2ab2858e75049415)

Merge pull request #52139 from zdover23/wip-doc-2023-06-21-backport-52104-to-reef

reef: doc/rados/operations: Improve formatting in crush-map.rst

Merge pull request #52141 from zdover23/wip-doc-2023-06-21-backport-52105-to-reef

reef: doc/mgr/dashboard: fix a typo

doc/mgr/dashboard: fix a typo

Signed-off-by: Guido Santella <gsantella@southhills.edu>
(cherry picked from commit d64c52150c48c993a214bb80b3802652d0685174)

doc/rados/operations: Improve formatting in crush-map.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 540d6ce3ee014a6ad1560e628c8d04e694a0c599)

qa/suites/rados: whitelist POOL_APP_NOT_ENABLED for rados cls tests

Fixes: https://tracker.ceph.com/issues/59192
Signed-off-by: Laura Flores <lflores@redhat.com>
(cherry picked from commit 66a6e7fdebe294f233bf34d5c833742bf15c9f03)

mon/MonClient: before complete auth with error, reopen session
When monClient try to authenticate and fail with -EAGAIN there is
a possibility that we no longer hunting and not have active_con.
that will result of disconnecting the monClient and ticks will continue
without having open session.
the solution is to check at the end of auth, that we don't have -EAGAIN
error, and if we do, reopen the session and on the next tick try auth again

Fixes: https://tracker.ceph.com/issues/58379
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>
(cherry picked from commit f8dad00f07f52d58d8ec67083e8bf568deec3277)

bump rocksdb submodule to latest ceph-reef-v7.9.2

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 87cd54281c88baca423c0e541f71a5e66d719fc4)

osd: use fmt::format_to(...) to fix FTBFS on gcc 13

Without this the build will fail on gcc 13 with errors like:

src/osd/scrubber/scrub_backend.cc:1309:14: error: call of overloaded 'format_to(std::back_insert_iterator<fmt::v9::basic_memory_buffer<char> >, const char [35], const char*, const uint64_t&, const uint64_t&, pg_shard_t&)' is ambiguous
  1309 |     format_to(std::back_inserter(out),
       |     ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
  1310 |               "{}size {} != size {} from shard {}",
       |               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1311 |               sep(error),
       |               ~~~~~~~~~~~
  1312 |               candidate.size,
       |               ~~~~~~~~~~~~~~~
  1313 |               auth.size,
       |               ~~~~~~~~~~
  1314 |               auth_shard);
       |               ~~~~~~~~~~~
 /usr/include/fmt/core.h:3233:17: note: candidate: 'OutputIt fmt::v9::format_to(OutputIt, format_string<T ...>, T&& ...) [with OutputIt = std::back_insert_iterator<basic_memory_buffer<char> >; T = {const char*, const long unsigned int&, const long unsigned int&, pg_shard_t&}; typename std::enable_if<detail::is_output_iterator<OutputIt, char>::value, int>::type <anonymous> = 0; format_string<T ...> = basic_format_string<char, const char*, const long unsigned int&, const long unsigned int&, pg_shard_t&>]'
 3233 | FMT_INLINE auto format_to(OutputIt out, format_string<T...> fmt, T&&... args)
      |                 ^~~~~~~~~
 /usr/include/c++/13/format:3761:5: note: candidate: '_Out std::format_to(_Out, format_string<_Args ...>, _Args&& ...) [with _Out = back_insert_iterator<fmt::v9::basic_memory_buffer<char> >; _Args = {const char*, const long unsigned int&, const long unsigned int&, pg_shard_t&}; format_string<_Args ...> = basic_format_string<char, const char*, const long unsigned int&, const long unsigned int&, pg_shard_t&>]'
  3761 |     format_to(_Out __out, format_string<_Args...> __fmt, _Args&&... __args)
       |     ^~~~~~~~~

Fixes: https://tracker.ceph.com/issues/58477
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 1ae8eb68c21b7df0bc7c38035176f2447dffaed2)

common, librbd, msg, test: fix FTBFS on gcc 13

This is based on 0024-gcc-13.patch from https://tracker.ceph.com/issues/58477
but with an extra #include <cstdint> to avoid dropping std:: prefixes on
integer types.

Fixes: https://tracker.ceph.com/issues/58477
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 796ce63667b460ba9b8d40891f30e45981ba237a)