qa/tests: retry the api call after making the request

based on the pointer from Bill in https://tracker.ceph.com/issues/62972#note-75

Fixes: https://tracker.ceph.com/issues/62972
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 1588712b558f229d54fdfab744f2480f15333067)

qa/dashboard: fix test_list_enabled_module failure

Check the ports availability and go for a new port if the current one is
not available

Fixes: https://tracker.ceph.com/issues/62972
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit a2a4a3415c0e5ceef2cb01d3bcdf5eb1fff23803)

qa/dashboard: fix tasks.mgr.dashboard.test_health.HealthTest

as per: https://tracker.ceph.com/issues/47612#note-14

Fixes: https://tracker.ceph.com/issues/47612
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 79d61bbb58cb34f9da678e37b4373fc84bd393f5)

qa/tests: fix test_list_enabled_modules timeout error

This test deals with enabling/disabling the modules. The assumption I
have is after enabling the
module test will wait for an active mgr but its not able to find it in
time and it fails. so taking inspiration from https://github.com/ceph/ceph/pull/58995/commits/6c7253be6f6fbfa6faed7a539cb78847fec04580 adding retries and logs to see if that's the case

Fixes: https://tracker.ceph.com/issues/62972
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit b2da7394ee02abd6525372d94cab090818cd6c8e)

qa/mgr/dashboard: fix test race condition

Fixes: https://tracker.ceph.com/issues/66844
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 6c7253be6f6fbfa6faed7a539cb78847fec04580)

Merge pull request #59734 from zdover23/wip-doc-2024-09-11-backport-59635-to-reef

reef: doc/rados: add "pgs not deep scrubbed in time" info

Merge pull request #59655 from zdover23/wip-doc-2024-09-08-backport-59628-to-reef

reef: doc/dev: add "activate latest release" RTD step

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: add "pgs not deep scrubbed in time" info

Add a procedure to doc/rados/operations/health-warnings.rst that
explains how to remedy the "X PGs not deep-scrubbed in time" health
warning.

This procedure was developed by Eugen Block, and is at the time of this
commit available on his blog at
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d620a51c306d1ebcdfc974dc0b85dc46cfde8d40)

Merge pull request #59683 from zdover23/wip-doc-2024-09-10-backport-59677-to-reef

reef: doc/man: edit ceph-bluestore-tool.rst

doc/man: edit ceph-bluestore-tool.rst

Edit the section "bluefs-bdev-migrate" in
doc/man/8/ceph-bluestore-tool.rst to add the information that this
operation expands the target storage by updating its size label, making
"bluefs-bdev-expand" unnecessary.

Improve the subject-verb agreement in this section, and supply some
absent definite articles.

Co-authored-by: Peter Gervai <grin@drop.grin.hu>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6b34707f827b2b197f53fe2e430d173b30b81401)

Merge pull request #59454 from adk3798/wip-65418-reef

reef: cephadm: fix host-maintenance command always exiting with a failure

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59664 from zdover23/wip-doc-2024-09-09-backport-59642-to-reef

reef: doc/cephadm: add malformed-JSON removal instructions

Merge pull request #59451 from adk3798/wip-65379-reef

reef: mgr/cephadm: make client-keyring deploying ceph.conf optional

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59453 from adk3798/wip-65416-reef

reef: cephadm: add idmap.conf to nfs sample file

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59458 from adk3798/wip-65949-reef

reef: mgr/pybind/object_format: fix json-pretty being marked invalid

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59459 from adk3798/wip-65952-reef

reef: mgr/cephadm: cleanup iscsi and nvmeof keyrings upon daemon removal

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

doc/cephadm: add malformed-JSON removal instructions

Add instructions for removing malformed JSON that causes the cephadm
module to fail to start, which in turn makes the orchestrator fail to
start.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 93d90ea358cf17b08e502caf8529061a10008fad)

Merge pull request #59601 from idryomov/wip-67752-reef

reef: rbd: amend "rbd {group,} rename" and "rbd mirror pool" command descriptions

Reviewed-by: Ramana Raja <rraja@redhat.com>

doc/dev: add "activate latest release" RTD step

Add a step to the section called "First Stable Release" that explains
how to activate the latest release in Read the Docs.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e7bd4bcdd074a3defae5866a6c035559bf210271)

Merge pull request #59262 from k0ste/wip-63920-reef

reef: ceph-volume: Fix unbound var in disk.get_devices()

Merge pull request #59201 from k0ste/wip-67528-reef

reef: debian pkg: record python3-packaging dependency for ceph-volume

Merge pull request #58687 from kamoltat/wip-ksirivad-backport-reef-57906

reef: mon/ElectionLogic: tie-breaker mon ignore proposal from marked down mon
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #59463 from adk3798/wip-67733-reef

reef: qa/distros: reinstall nvme-cli on centos 9 nodes

ceph-volume: add "packaging" to install_requires

in 0985e201, "packaging" was introduced as a runtime dependency of
ceph-volume, and `ceph.spec.in` was updated accordingly to note
this new dependency. but the debian packaging was not updated.

in 80edcd40, the missing dependency was added to debian/control as
one of ceph-volume's runtime dependency.

but dh_python3 is able to figure out the dependencies by reading
the egg's metadata of the ceph-volume python module. and as a
python project, ceph-volume is using its `setup.py` for
tracking its dependencies.

so in order to be more consistent, and keep all of its dependencies
in one place, let's move this dependency to setup.py . as the
packagings in both distros are able to figure the dependencies
from egg-info.

see also
- https://manpages.debian.org/testing/dh-python/dh_python3.1.en.html#dependencies
- https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/#_automatically_generated_dependencies

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 729fd8e25ff2bfbcf99790d6cd08489d1c4e2ede)

debian pkg: record python3-packaging dependency for ceph-volume

Since commit 0985e201342 ("ceph-volume: use 'no workqueue' options
with dmcrypt") the python "packaging" module is used to parse the
cryptsetup version output, but the debian packaging was not updated to
record that new dependency.

So simply record this in the d/control file, adding a <pkg>.requires
file seemed to not really winning us anything here.

Fixes: https://tracker.ceph.com/issues/67290
Fixes: 0985e201342fa53c014a811156aed661b4b8f994
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 80edcd40e8092d9fb3b45c1a6c7f9b7f4f37b69e)

rbd: mention namespace in "rbd mirror pool" command descriptions

Commit 5e64748927d0 ("doc/rbd: add namespace information for mirror
commands") did this for the man page, update the built-in help as well.

The "by default" bit in the description of "rbd mirror pool enable" and
"rbd mirror pool disable" commands is specific to pool mode which is in
turn specific to journal-based mirroring, so it's removed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 7e02946a3f60da0dcc99e0b4a4b6b3d359db1c15)

rbd: clarify "rbd {group,} rename" command description

Make it clearer that, despite a full image or group spec being taken
for source and destination, an image or a group can be renamed only
within its pool or namespace.

Rename across pools or namespaces within the same pool is unsupported.

Fixes: https://tracker.ceph.com/issues/67752
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5f064a52707af58fd6007734a768899aeb5a32fc)

Merge pull request #59583 from zdover23/wip-doc-2024-09-04-backport-59546-to-reef

reef: doc/rados/operations: Improve health-checks.rst

Merge pull request #59586 from zdover23/wip-doc-2024-09-04-backport-59577-to-reef

reef: doc/mds: improve wording

doc/mds: improve wording

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>
(cherry picked from commit 332804bad58c892d01d2d2da557e42104365ef8a)

doc/rados/operations: Improve health-checks.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 2aa82539ede8ade095ad0b22e973eac538354711)

Merge pull request #59560 from zdover23/wip-doc-2024-09-02-backport-59556-to-reef

reef: doc: Correct link to Prometheus docs

doc: Correct link to Prometheus docs

The link is to the `#http_sd_config` anchor in the prometheus config docs; that link only works without the trailing `/`.

This correction would ideally get backported to at least reef & squid.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
(cherry picked from commit 84a30ba6b94b34806faac8217ccaa299c9ee68d6)

Merge pull request #59549 from zdover23/wip-doc-2024-09-01-backport-59544-to-reef

reef: doc: update tests-integration-testing-teuthology-workflow.rst

doc: update tests-integration-testing-teuthology-workflow.rst

* add "Infrastructure" section.

* move "Naming the ceph-ci branch" section under
   "Getting binaries - Build Ceph". Also mention
   about centos9-only trick.

* in "Teuthology Archives", mention about developer
   playground machines and ceph log files.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
(cherry picked from commit 9bfcb8e17db8c61e523e10856d12b237433d831a)

Merge pull request #59541 from zdover23/wip-doc-2024-08-31-backport-59528-to-reef

reef: doc/ceph-volume: add spillover fix procedure

doc/ceph-volume: add spillover fix procedure

Add a procedure that explains how, after an upgrade, to move bytes that
have spilled over to a relatively slow device back to the faster device.

This procedure was developed by Chris Dunlop on the [ceph-users] mailing
list, here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/POPUFSZGXR3P2RPYPJ4WJ4HGHZ3QESF6/

Eugen Block requested the addition of this procedure to the
documentation on 30 Aug 2024.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 98618aaa1c8b786c7d240a210b62cc737fdb048d)

Merge pull request #59462 from adk3798/wip-66428-reef

reef: mgr/cephadm: make SMB and NVMEoF upgrade last in staggered upgrade

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59461 from adk3798/wip-66426-reef

reef: cephadm: CephExporter doesn't bind to IPv6 in dual stack

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59460 from adk3798/wip-65969-reef

reef: mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59455 from adk3798/wip-65723-reef

reef: cephadm: have agent check for errors before json loading mgr response

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #57519 from asm0deuz/backport_PR54158

reef: cephadm: added check for `--skip-firewalld` to section on adding explicit Ports to firewalld

Reviewed-by: Adam King <adking@redhat.com

Merge pull request #57234 from adk3798/wip-65763-reef

reef: mgr/cephadm: set OSD cap for NVMEoF daemon to "profile rbd"

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #56909 from adk3798/wip-65383-reef

reef: mgr/cephadm: Allows enabling NFS Ganesha NLM

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #56490 from adk3798/wip-64991-reef

reef: cephadm: fix `cephadm shell --name <daemon-name>` for stopped/failed daemon

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59268 from k0ste/wip-64671-reef

reef: qa/tasks/ceph_manager.py: Rewrite test_pool_min_size
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

qa/distros: reinstall nvme-cli on centos 9 nodes

To work around a potential linking issue between
nvme-cli ad libnvme that prevents nvme-cli from
correctly generating a hostnqn, causing

nvme_fabrics: found same hostid edb4e426-766f-44c6-b127-da2a5b7446ef but different hostnqn hostnqn

messages in dmesg and the inability to setup nvme
loop devices

Fixes: https://tracker.ceph.com/issues/67684
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 4e5f269c01dd94741fbddaa7f76ebe4ac155e650)

mgr/cephadm: make SMB and NVMEoF upgrade last in staggered upgrade

This needs to happen as some work on the NVMEoF side (still unmerged
as of writing this) will make the NVMEoF daemon dependent on the mon.
Prior to this patch, in a staggered upgrade, all daemons not using the
ceph image were upgraded after the mgr since we typically only care
about the default image changing or potential changes to how we handle
our systemd units which only needs the mgr to be upgraded to be applied.
This NVMEoF dependency on the mon changes this and we can no longer
upgrade it directly after the mgr. This patch changes it so the NVMEoF
daemon is instead upgraded after all ceph image daemons have been
upgraded in a staggered upgrade scenario. Non-staggered upgrades
are unaffected as the NVMEoF daemon was already upgraded near the
end in that scenario. The SMB dameon has no reason it needs to be
upgraded later, but it's in the (small) pool of daemons that don't
use the ceph image and aren't for monitoring, so it's been affected
by this as well.

NOTE: This is a bit of an ugly patch imo and shows that a refactoring
of the upgrade code is likely required. Hopefully this patch is more
of a stopgap until that larger effort can be made

Fixes: https://tracker.ceph.com/issues/65809
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 5e7a3c2147d87c1fc5be71acbadedefb70e024bf)

Fix CephExporter protocol bind logic

In a dual stack configuration ceph-exporter binds to ipv4 only and the metrics fail in ipv6

Signed-off-by: Mouratidis Theofilos <mtheofilos@gmail.com>
(cherry picked from commit 110bc665078fe19c31e3680c4197587e69e4e751)

Conflicts:
src/cephadm/cephadmlib/daemons/ceph.py

cephadm: Adding support to pass --no-cgroups-split flag when adopting legacy daemons

Signed-off-by: Gilad Sid <sid.gilad@gmail.com>
(cherry picked from commit 20ffd4d6e330095c8cf2816a36f61bd950e213a5)

mgr/cephadm: clean up nvmeof keyring when removing daemon

The parent class' (CephService) post_remove handles
keyring cleanup

Fixes: https://tracker.ceph.com/issues/65717
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 04065e34d74421fd1f85b52b87efaf357fe88505)

Conflicts:
src/pybind/mgr/cephadm/services/nvmeof.py

mgr/cephadm: clean up iscsi auth keyring when removing daemon

The parent class (CephService) handles the cleanup of
the auth keyring

Fixes: https://tracker.ceph.com/issues/65717
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit f7e8ee74d5bfb941a5f67d0e03c3845f5cdb1525)

mgr/pybind/object_format: fix json-pretty being marked invalid

without this patch you'd get

```
[ceph: root@vm-00 /]# ceph nfs cluster info --format json-pretty

Error EINVAL: Unknown format name: json-pretty
```

this seems to be because valid formats are checked using
the class

```
class Format(str, enum.Enum):
    plain = "plain"
    json = "json"
    json_pretty = "json-pretty"
    yaml = "yaml"
    xml_pretty = "xml-pretty"
    xml = "xml"
```

and then

```
set(str(v) for v in Format.__members__)
```

but that resolves to

```
{'yaml', 'json_pretty', 'plain', 'xml', 'json', 'xml_pretty'}
```

and so json-pretty is marked as invalid. Note that it's also
impossible to pass json_pretty as the format as core ceph
blocks it with

invalid choice: 'json_pretty' (choose from 'json', 'json-pretty', 'xml', 'xml-pretty', 'plain', 'yaml')

Fixes: https://tracker.ceph.com/issues/65554
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit cd988a01a0edf13882527f5526f1793d4dece437)

cephadm: have agent check for errors before json loading mgr response

Currently, since it tries to json.loads the response
payload before checking the return code, if there was
an error it fails with

Failed to send metadata to mgr: the JSON object must be str, bytes or bytearray, not ConnectionRefusedError

which is masking the actual failure.

Also adds more context to the RuntimeError raised

Fixes: https://tracker.ceph.com/issues/65553
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 287bd34eec09815602700747c62e0a709e6e5ff0)

cephadm: fix host-maintenance command always exiting with a failure

The host-maintenance command would always fail because
command_maintenance always returns a string. This string is passed to
sys.exit and thus always gets printed and causes a non-zero exit code.
Fix the command line behavior by renaming the original function and
adding a new command_maintenance that prints the string and returns an
int like other command_* functions do.

Fixes: https://tracker.ceph.com/issues/65122
Signed-off-by: John Mulligan <jmulligan@redhat.com>
(cherry picked from commit 0a137b140e47d02b03d133adb55e62aa0de7b984)

cephadm: add idmap.conf to nfs sample file

Otherwise deployment using this sample will fail with
"ERROR: required file missing from config-json: idmap.conf"

Fixes: https://tracker.ceph.com/issues/65155
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 79b1dd5b30f8556aacc9a7536366694ba901795e)

doc/cephadm: document new --no-ceph-conf flag for `orch client-keyring set`

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 2741d9471c959f07f9b55c52114ea9d71916345e)

mgr/cephadm: make client-keyring deploying ceph.conf optional

There are cases where users would like to manage their own
ceph.conf but still have cephadm deploy the client keyrings,
so this is being added to facilitate that.

Fixes: https://tracker.ceph.com/issues/65335
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 0dab95eb4fffb493edc3e542a6613bdb5332a670)

mgr/cephadm: rename ceph_admin_key -> client_key in _calc_client_files

I think this variable name is misleading, as we use this same
logic for all client-keyrings that can be setup, not just
the client.admin keyring

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 9e2327f0c5f0c103e2db8c7d86ceddd177179df9)

Merge pull request #59431 from zdover23/wip-doc-2024-08-26-backport-59428-to-reef

reef: doc/cephadm: how to get exact size_spec from device

doc/cephadm: how to get exact size_spec from device

Add instructions for retrieving the exact size of block devices.

Fixes: https://tracker.ceph.com/issues/66754
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d00d1b52d50b5575d918c3be7b7a8249ef31f0a8)

Merge pull request #59425 from zdover23/wip-doc-2024-08-25-backport-59418-to-reef

reef: doc/glossary: add "object storage"

doc/glossary: add "object storage"

Add a (very basic) definition of object storage.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 43057b88538e868b817acb04d5e6c4e95b4c716e)

Merge pull request #57625 from sajibreadd/wip-65938-reef

reef: os/bluestore: set rocksdb iterator bounds for Bluestore::_collection_list()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #57621 from sajibreadd/wip-66145-reef

reef: osd: CEPH_OSD_OP_FLAG_BYPASS_CLEAN_CACHE flag is passed from ECBackend

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55110 from k0ste/wip-63977-reef

reef: mgr/BaseMgrModule: Optimize CPython Call in Finish Function

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #53269 from YiteGu/backport-always-generate-random-nonce

reef: msg: always generate random nonce; don't try to reuse PID

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #59411 from adk3798/wip-67682-reef

reef: mgr/cephadm: add "original_weight" parameter to OSD class

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59381 from zdover23/wip-doc-2024-08-21-backport-59348-to-reef

reef: doc/rados: document unfound object cache-tiering scenario

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/cephadm: add "original_weight" parameter to OSD class

Fixes: https://tracker.ceph.com/issues/67329
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 04330f5df92994882efcd4879d5c37279138e97b)

Merge pull request #59075 from tobias-urdin/reef-keystone-admin-token

reef: rgw: invalidate and retry keystone admin token

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59018 from Svelar/wip-67072-reef

reef: rgw/amqp: lock erase and create connection before emplace

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #59056 from yuvalif/wip-67363-reef

reef: common/dout: fix FTBFS on GCC 14

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57197 from k0ste/wip-63315-reef

reef: os/bluestore: fix crash caused by dividing by 0

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #57194 from k0ste/wip-64590-reef

reef: os/bluestore: fix the problem of l_bluefs_log_compactions double recording

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56813 from Matan-B/wip-65305-reef

reef: osd/SnapMapper: fix _lookup_purged_snap

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #56431 from Matan-B/wip-65096-reef

reef: mon/OSDMonitor: fix rmsnap command

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55778 from ifed01/wip-ifed-fix-63795-reef

reef: test/store_test: fix deferred writing test cases

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #55220 from ifed01/wip-ifed-cache-ratios

reef: osd: make _set_cache_sizes ratio aware of cache_kv_onode_ratio

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #58312 from cbodley/wip-66710-reef

reef: rgw/notifications/test: fix rabbitmq and kafka issues in centos9

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #58846 from idryomov/wip-58120-reef

reef: osd: avoid watcher remains after "rados watch" is interrupted

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados: document unfound object cache-tiering scenario

Explain how to deal with "unfound objects" when restarting OSDs in a
cache-tiered environment.

Fixes: https://tracker.ceph.com/issues/44286
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f01d7a8d5b85170c034acb962b9833913853a1c5)

Merge pull request #58513 from k0ste/wip-66890-reef

reef: mgr/Mgr.cc: clear daemon health metrics instead of removing down/out osd from daemon state

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #57487 from ljflores/wip-65014-reef

reef: qa/suites/rados/singleton: add POOL_APP_NOT_ENABLED to ignorelist

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #57408 from k0ste/wip-62927-reef

reef: mon: stuck peering since warning is misleading

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #57402 from k0ste/wip-65916-reef

reef: kv/RocksDBStore: Configure compact-on-deletion for all CFs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #59295 from zdover23/wip-doc-2024-08-19-backport-59256-to-reef

doc/cephfs: s/mountpoint/mount point/

doc/cephfs: s/mountpoint/mount point/

Change the string "mountpoint" to "mount point" in English-language
strings (as opposed to in commands, where the string "mountpoint"
sometimes appears and is correct).

cf. https://github.com/ceph/ceph/pull/58908#discussion_r1697715486
in which page 345 of The IBM Style Guide is referenced to back up this
change.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 57ed73eaa6945c69059d299ea5310eaac57e2299)

Merge pull request #59287 from zdover23/wip-doc-2024-08-18-backport-59257-to-reef

reef: doc/cephfs: s/mountpoint/mount point/

doc/cephfs: s/mountpoint/mount point/

Change the string "mountpoint" to "mount point" in English-language
strings (as opposed to in commands, where the string "mountpoint"
sometimes appears and is correct).

cf. https://github.com/ceph/ceph/pull/58908#discussion_r1697715486 in
which page 345 of The IBM Style Guide is referenced to back up this
change.

This commit alters only English-language text and example commands in
which the string "{mount point}" is meant to be replaced. No commands
meant for cutting-and-pasting have been altered in this commit.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a0f81cfb5094164630f55a717efbbcdce45bce58)

qa/tasks/ceph_manager.py: Added more loggings for all_active_or_peered()

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 9762656160c9ae12d06b29a3e8a8d0dd13847328)

qa/tasks/ceph_manager.py: Rewrite test_pool_min_size

Problem:

Failed the test in EC Pool configuration because PGs are
not going into active+clean (our fault for over thrashing and checking the wrong thing).
Also, PG would not go into active because we thrash below min_size
in an EC pool config, not enough shards in the acting set.
Therefore, failed the wait_for_recovery check.
Moreover, When we revive osds, we didn't add the osd back in the cluster,
this messes up true count for live_osds in the test.

Solution:

Instead of randomly choosing OSDs to thrash,
we randomly select a PG from each pool and
thrash the OSDs in the PG's acting set until
we reach min_size, then we check to see if the
PG is still active. After that we revive all
the OSDs to see if the PG recovered cleanly.

We removed some of the unnecessary part such
as `min_dead`, `min_live`, `min_out` and etc.

Also, we refractored the part of where we are
assigning k,m for the EC pools so that we get
better code readablility.

Fixes: Fixes: https://tracker.ceph.com/issues/59172
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 8c4768ecb3ec38c8fce209eae9fe931e974d0495)

qa/tasks/rados.py: Allow rados task to override config

Problem:

Currently, no option override the config in rados task.

Solution:

Enable override of the config file in rados task.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 92bf1a8aa8d0d208577c4076d4a86644c01548d5)

qa/tasks/ceph_manager.py: init test_min_size_duration

Added comment about test_min_size_duration
in qa/tasks/thrashosds.

But also use the variable in ceph_manager.py

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 9f19dffc93463513e03908f3506c62e65364c0cd)

qa/suites/rados: Added wait_for_all_active_clean_pgs flag

Added flag to not allow rados suite to delete
the pool unless all pgs are active+clean
and all OSDs are up in the thrashosds side
of the test.

Fixes: https://tracker.ceph.com/issues/59172
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 3ccd10f266cfd7ec6dd1ad930598bfe4ca422a90)

ceph-volume: Fix unbound var in disk.get_devices()

00ba00fdfab8 looks to have regressed.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 0e95b27402e46c34586f460d2140af48d03fa305)

Merge pull request #59251 from zdover23/wip-doc-2024-08-16-backport-59167-to-reef

reef: doc/cephfs: improve "layout fields" text

doc/cephfs: improve "layout fields" text

Improve "layout fields" text in doc/cephfs/file-layouts.rst, as suggesed
by Anthony D'Atri in these comments:

https://github.com/ceph/ceph/pull/59021#discussion_r1704108581
https://github.com/ceph/ceph/pull/59021#discussion_r1704112320

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0949b410433837f0509fb73169fa7f22c8f6c256)

Merge pull request #59022 from zdover23/wip-doc-2024-08-05-backport-58891-to-reef

reef: doc/cephfs: edit "Layout Fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #59234 from zdover23/wip-doc-2024-08-15-backport-59219-to-reef

reef: doc/rgw/notification: persistent notification queue full behavior