Merge pull request #57981 from dparmar18/wip-66191-squid

squid: client: set LIBMOUNT_FORCE_MOUNT2=always

Merge pull request #58085 from rishabh-d-dave/wip-66500-squid

squid: cephfs-shell: prints warning, hangs and aborts when launched

Merge pull request #58389 from zdover23/wip-doc-2024-07-02-backport-58092-to-squid

squid: doc/start: remove mention of Centos 8 support

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/start: remove mention of Centos 8 support

Remove information so that it is clear that Centos 8 will no longer be
supported by Ceph.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6b149e4f6ab2ac716e73f42b0d6c67482a768da9)

Merge pull request #58266 from amathuria/wip-66694-squid

squid: osd/ECBackend.cc: Fix double increment of num_shards_repaired stat

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #58341 from guits/wip-66746-squid

squid: tests: make sure nvmetcli and nvme-cli are up to date

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58219 from guits/wip-66026-squid

squid: ceph-volume: create LVs when using partitions

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58151 from guits/wip-66464-squid

squid: ceph-volume: fix set_dmcrypt_no_workqueue()

Reviewed-by: Teoman ONAY <tonay@ibm.com>

Merge pull request #58370 from zdover23/wip-doc-2024-07-01-backport-58366-to-squid

squid: doc/cephfs: update cephfs-shell link

Merge pull request #58198 from ljflores/wip-66605-squid

squid: Telemetry upgrade test fixes

doc/cephfs: update cephfs-shell link

Repair the link to cephfs-shell.rst in doc/cephfs/cephfs-shell.rst that
was broken in https://github.com/ceph/ceph/pull/41165/ when
doc/cephfs/cephfs-shell.rst was moved to doc/man/8/cephfs-shell.rst.

This commit is made in response to a request by Lander Duncan that was
made on the [ceph-users] mailing list, and can be seen here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/F7V4CWLIYCAJ4JXI2JLNY6QPCFPR4SLA/

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4f8903cf80a94596a1b17b4a07affb874f21ae76)

Merge pull request #58359 from zdover23/wip-doc-2024-06-30-backport-58358-to-squid

squid: doc/cephfs: take Anthony's suggestion

doc/cephfs: take Anthony's suggestion

Incorporate Anthony's suggestion in
https://github.com/ceph/ceph/pull/57567#discussion_r1607016050 into
doc/cephfs/fs-volumes.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 7a78dda0a92339db27359f91f0cb081703e2e7f9)

Merge pull request #58264 from amathuria/wip-66690-squid

squid: mon, qa: suites override ec profiles with --yes_i_really_mean_it; monitors accept that

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #58262 from amathuria/wip-66688-squid

squid: qa/suites/rados/thrash/workloads: remove cache tiering workload

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #57589 from NitzanMordhai/wip-66142-squid

squid: common/pick_address: check if address in subnet all public address

Reviewed-by: Laura Flores <lflores@redhat.com>

tests: make sure nvmetcli and nvme-cli are up to date

Otherwise jobs end up with the following failure:

```
2024-06-25T14:22:18.659 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to write to /dev/nvme-fabrics: Invalid argument
```

Also, the output of nvme list has changed so we have to update
qa/tasks/nvme_loop.py accordingly.

Fixes: https://tracker.ceph.com/issues/66707
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit d707c41eacb513bdcb9fec17c823bfbfdc182700)

Merge pull request #58307 from zdover23/wip-doc-2024-06-27-backport-58287-to-squid

squid: doc/rados: update how to install c++ header files

Merge pull request #58304 from zdover23/wip-doc-2024-06-27-backport-58299-to-squid

squid: doc/dev: Fix typos in encoding.rst

Merge pull request #58090 from vshankar/wip-66519-squid

squid: qa: use centos9 for fs:upgrade

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: update how to install c++ header files

In this example librados2-devel only install C header files on fedora 40,
therefore I added libradospp-devel to the command to include C++ header files.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>
(cherry picked from commit 7e87441601db2379488fab62e900cd834086f26d)

doc/dev: Fix typos in encoding.rst

Signed-off-by: N Balachandran <nibalach@redhat.com>
(cherry picked from commit 9f43c67a639958a13fef198ffe62cf5241f1f53c)

Merge pull request #58271 from zdover23/wip-doc-2024-06-26-backport-58226-to-squid

squid: doc/rados: edit troubleshooting-osd.rst

doc/rados: edit troubleshooting-osd.rst

Make minor changes to the "Debugging Slow Requests" section of
doc/rados/troubleshooting/troubleshooting-osd.rst in preparation
for an expansion of this section in response to a reqeust from Joel
Davidow.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8b211b9c7f9b173bb12028eb7108d13867883d6e)

Merge pull request #58257 from zdover23/wip-doc-2024-06-25-backport-58243-to-squid

squid: doc/rados: credit Prashant for a procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

osd/ECBackend.cc: Fix double increment of num_shards_repaired stat

Commit https://github.com/ceph/ceph/commit/deffa8209f9c0bd300cfdb54d358402bfc6e41c6 refactored
ECBackend::handle_recovery_push for Crimson but accidentally duplicated the code that increments
the num_shards_repaired OSD statistic.

This caused one of the QA tests to fail because the stat reported twice as much repair work
had been completed than expected:

qa/standalone/scrub/osd-scrub-repair.sh: TEST_repair_stats_ec: test 26 = 13

Fixes: https://tracker.ceph.com/issues/64437
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
(cherry picked from commit e618dc01a7a1bdfaa3e1a6fa2a9a9ac13eee11b8)

mon: settle erasure-code-profile --yes-i-really-mean-it in asok's schema

Fixes: https://tracker.ceph.com/issues/65183
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 0e70f679d0f32fafd7db39229325cf7db7f6eb05)

qa: tests override ec profiles with --yes-i-really-mean-it

This fixes a fallout from 629ba7bd349d48cdaa6d094751e7cfce651ba2bc.
The problem has been nailed down by Laura Flores.

Fixes: https://tracker.ceph.com/issues/65183
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 6f813236f8c617b3ff9517cf73ed256b5fede90e)

qa/suites/rados/thrash/workloads: remove cache tiering workload

Fixes: https://tracker.ceph.com/issues/65852
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 35505a7f1fd8bcad2047426a4d641b4c016180af)

doc/rados: credit Prashant for a procedure

Credit Prashant D for creating the stretch-mode workaround procedure for
retrieving the correct size of datacenters.

Follows: https://github.com/ceph/ceph/pull/58109

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2e777cb4f8bd682b807b97074f930826aa902db9)

Merge pull request #57970 from adk3798/squid-replace-pkg_resources-with-importlib

squid: cephadm: use importlib.metadata for querying ceph_iscsi's version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57400 from adk3798/squid-cephadm-timeout-ignore-refresh-failed

squid: qa/cephadm: ignore CEPHADM_REFRESH_FAILED on timeout test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57768 from adk3798/wip-65722-squid

squid: cephadm: have agent check for errors before json loading mgr response

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57398 from adk3798/squid-cephadm-ignore-orch-paused

squid: qa/cephadm: ignore CEPHADM_PAUSED on test_orch_cli test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57187 from adk3798/squid-stray-laundry

squid: qa/cephadm: ignore stray daemon warning during rados_api_tests

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57770 from adk3798/wip-65948-squid

squid: mgr/pybind/object_format: fix json-pretty being marked invalid

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57516 from adk3798/wip-65958-squid

squid: mgr/cephadm:  update default NVMEoF container image version

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

ceph-volume: create LVs when using partitions

commit [1] introduced a behavior change.
`ceph-volume lvm prepare` used to create VGs/LVs when it was passed partitions
for db and/or wal devices. Since commit [1] has been introduced, it made ceph-volume
consume the partition directly, it doesn't create LV anymore. Although this
doesn't prevent from creating OSDs, this is a behavior change.

Fixes: https://tracker.ceph.com/issues/65477
[1] 1e7223281fa044c9653633e305c0b344e4c9b3a4

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 11f0173f1299ed6d137159dcf62c4a7a9fd20f82)

Merge pull request #58202 from zdover23/wip-doc-2024-06-23-backport-58200-to-squid

squid: doc/start: add links to Beginner's Guide

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add links to Beginner's Guide

Add two links to Sage Weil's overview talks about Ceph.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 785020f212504cdbf374e0d3ca843fd92309de98)

Merge pull request #57936 from zdover23/wip-doc-2024-06-08-backport-57923-to-squid

squid: doc/src: add inline literals (``) to variables

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58181 from zdover23/wip-doc-2024-06-21-backport-58109-to-squid

squid: doc/rados: add stretch_rule workaround

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58161 from zdover23/wip-doc-2024-06-20-backport-57156-to-squid

squid: doc/rados: followup to PR#58057

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58189 from zdover23/wip-doc-2024-06-21-backport-55675-to-squid

squid: doc/src: invadvisable is not a word

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/suites/upgrade/telemetry-upgrade/quincy-x: update image reference

`quay.io/ceph/daemon-base:latest-quincy` points to the latest quincy tag (v17.2.7),
not the tip of quincy, which is actually what we want to test.

Pulling the daemon-base:latest-quincy image shows it is running v17.2.7:
```
lflores:ceph$ podman run --rm -it daemon-base:latest-quincy
[root@e4eb265e7fa0 /]# ceph -v
ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)
```

It is important that we test the tip of quincy in case there is a fix there that
we want to pick up.

I referenced this commit to make the change: https://github.com/ceph/ceph/commit/d0381ed7e68a6de7c9ff1e4220698ed8eab28ac6

Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 914149df5f3d7189ea6bf96bbfaf7d948ca4ed01)

qa/suites/upgrade/telemetry-upgrade/reef-x: update how cephadm is pulled and change image reference

Update how cephadm is pulled:

`cephadm_git_url` and `cephadm_branch` are used in releases older than reef
to install cephadm. Both of these keys are needed to install it from the github
repo.

However, in reef and on, the compiled zipapp cephadm needs to be pulled differently
than the old single python script `cephadm` from earlier releases.

This commit is what I referenced for this change: https://github.com/ceph/ceph/commit/967716f2e6c8cd3f7d810a5f8e5e49448717e513

-------

Change image reference:

`quay.io/ceph/daemon-base:latest-reef` points to the latest reef tag (v18.2.2),
not the tip of reef, which is actually what we want to test.

Pulling the daemon-base:latest-reef image shows it is running v18.2.2:
```
lflores:ceph$ podman run --rm -it daemon-base:latest-reef
[root@d4d9a0eb04be /]# ceph -v
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)
```

It is important that we test the tip of reef in case there is a fix there that we
want to pick up.

I referenced this commit to make the change: https://github.com/ceph/ceph/commit/d0381ed7e68a6de7c9ff1e4220698ed8eab28ac6

Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit fcb22a022038ce64dcc6e95f9be56cf75a3be4b0)

qa/suites/upgrade/telemetry-upgrade: add more ignorelist items and require_osd_release=squid

The warnings added to the ignorelist show up in the cluster log, but they are
expected during upgrades and should thus be ignored.

We also need to set require_osd_release=squid to avoid this warning:
```
cluster [WRN] Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED)
```

Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit dc65e55d6e88d1dc24b92837f5ae0f4f9aa5169f)

qa/workunits: Opt in to new telemetry collections right away after upgrading

If this does not happen, the test may fail due to the "TELEMETRY_CHANGED" warning
appearing for too long in the cluster log.

Fixes: https://tracker.ceph.com/issues/64458
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 769d9ccffbd0edb30d65f3e8e0aeb7f2070145ef)

qa/suites/upgrade/telemetry-upgrade: upgrade from reef instead of pacific

With cephadm upgrades, we are only allowed to upgrade from as far back as N-2
releases. On the main branch, that means we can only upgrade from quincy and reef, and
we can no longer upgrade from pacific.

This test was trying to upgrade from pacific, which isn't allowed, which led to an
`UPGRADE_BAD_TARGET_VERSION` cluster error.

Fixes: https://tracker.ceph.com/issues/65189
Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit 0b39555f9fcbb093ca03c7bad15acb4e368b99f0)

Merge pull request #58187 from zdover23/wip-doc-2024-06-21-backport-58112-to-squid

squid: doc/rbd: fix typos in NVMe-oF docs.

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58171 from idryomov/wip-qemu-task-version-squid

squid: qa/tasks/qemu: Fix OS version comparison

Reviewed-by: Ramana Raja <rraja@redhat.com>

doc/src: invadvisable is not a word

changed to inadvisable on line 4997

Signed-off-by: Doug Whitfield <douglasawh@gmail.com>
(cherry picked from commit fb4a5d09e11b75a2f38c42c489af7e6f148916a7)

doc/rbd: fix typos in NVMe-oF docs.

Signed-off-by: N Balachandran <nibalach@redhat.com>
(cherry picked from commit 0cd415c53b49b2e9168df488340dab91d4c6badc)

doc/rados: add stretch_rule workaround

Add a method for defining a CRUSH rule that returns the actual value of
the total available size.

Fixes: https://tracker.ceph.com/issues/56650
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 007385a3ef05bd92e006fc7d6aba3fbb51792ef7)

Merge pull request #57990 from idryomov/wip-66359-squid

squid: pybind/rbd: parse access and modify timestamps in UTC

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58098 from adk3798/wip-66452-squid

squid: cephadm: _extract_host_info_from_*() refactor

Reviewed-by: John Mulligan <jmulligan@redhat.com

Merge pull request #58096 from adk3798/wip-66429-squid

squid: mgr/cephadm: make SMB and NVMEoF upgrade last in staggered upgrade

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58095 from adk3798/wip-66427-squid

squid: cephadm: CephExporter doesn't bind to IPv6 in dual stack

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57972 from adk3798/squid-keepalive-security-context

squid: cephadm/services/ingress: configure security user in keepalived template

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57773 from adk3798/wip-65968-squid

squid: mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57772 from adk3798/wip-65955-squid

squid: mgr/cephadm: Fix unfound progress events

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57771 from adk3798/wip-65953-squid

squid: qa/tasks/cephadm: fix pulling containers from private registries

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57769 from adk3798/wip-65724-squid

squid: python-common: handle "anonymous_access: false" in to_json of Grafana spec

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57517 from adk3798/wip-65951-squid

squid: mgr/cephadm: cleanup iscsi and nvmeof keyrings upon daemon removal

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57411 from adk3798/squid-cephadm-ignore-stray-on-upgrade

squid: qa/cephadm: ignore CEPHADM_STRAY_DAEMON in upgrade tests

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57397 from adk3798/squid-no-stray-recent-removal

squid: mgr/cephadm: don't mark daemons created/removed in the last minute as stray

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57396 from adk3798/squid-autotune-testing

squid: mgr/cephadm: additional debug logging for autotuner

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57233 from adk3798/wip-65762-squid

squid: mgr/cephadm: set OSD cap for NVMEoF daemon to "profile rbd"

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57186 from adk3798/squid-service-size-cleanup

squid: mgr/cephadm: cleanup service size logic block

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57185 from adk3798/squid-cephadm-apply-spec-file

squid: cephadm: check if file exists when passing `--apply_spec`

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #57184 from adk3798/squid-bootstrap-custom-alerts

squid: cephadm: allow passing custom prometheus alerts to bootstrap

Reviewed-by: John Mulligan <jmulligan@redhat.com>

qa/tasks/qemu: Fix OS version comparison

See: https://sentry.ceph.com/share/issue/21ed88d705854238bdafbf6711e795ee/
They're strings, not floats.
This surfaced as a result of https://github.com/ceph/teuthology/pull/1953

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 260a45a03148ba4d2bbfcac9485bcf9a8743cade)

Merge pull request #58130 from zdover23/wip-doc-2024-06-19-backport-58123-to-squid

squid: doc: clarify availability vs integrity

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: followup to PR#58057

Incorporate Anthony D'Atri's suggestions in
https://github.com/ceph/ceph/pull/58057

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2e999a26ef2d3c12d81bd54c187a6225a5fcd7a6)

pybind/rbd: parse access and modify timestamps in UTC

It appears that commits 08cee16d0a4b ("pybind/rbd: always parse
timestamps in UTC") and 809c5430c292 ("librbd: add image access/last
modified timestamps") raced with each other and we ended up with two
more timezone-dependent timestamps.

Fixes: https://tracker.ceph.com/issues/66359
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit fafa911b51199f61d0c97bc6dfc71d4f65134db4)

test/pybind/rbd: make timestamp tests meaningful

The existing asserts don't really test anything, with some of them
being for inequality against a literal of a mismatching type.  As
a result, a bug in access_timestamp() and modify_timestamp() went
unnoticed for years.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit a1b6ca863a384fe32e750b9ee8f10688f88dd1e3)

test/pybind/rbd: fix tests that compare strings with b''

assert_not_equal(b'', self.image.id()) is bogus because Image::id()
returns a string (str), not bytes.  If the types don't match, values
are guaranteed to not match.

The same goes for Image::block_name_prefix().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 05f7e4f2025c435dca8ae0b34f65425afbc212b5)

Merge pull request #57571 from idryomov/wip-65813-squid

squid: librbd: don't crash on a zero-length read if buffer is NULL

Merge pull request #57602 from ajarr/wip-66117-squid

squid: librbd: create rbd_trash object during pool initialization and namespace creation

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57766 from idryomov/wip-66232-squid

squid: qa/suites/rbd: override extra_system_packages directly on install task

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58044 from idryomov/wip-64662-squid

squid: librbd: allow cloning from non-user snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>

ceph-volume: fix regex usage in `set_dmcrypt_no_workqueue`

- Updated the regex pattern to `r'(\d+\.?)+'` to more accurately
  capture version numbers.

- Replaced `re.match` with `re.search` to properly match the cryptsetup
  version in the output.

- `re.match` only checks for a match at the beginning of the string,
   while `re.search` looks for a match anywhere in the string.

This fix ensures that the function correctly retrieves the
cryptsetup version from the output.

Fixes: https://tracker.ceph.com/issues/66393
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 607eb34b2c278566c386efcbf3018629cf08ccfd)

ceph-volume: fix set_dmcrypt_no_workqueue()

`set_dmcrypt_no_workqueue()` from `ceph_volume.util.encryption`

The function `set_dmcrypt_no_workqueue` in `encryption.py` now
dynamically retrieves the installed cryptsetup version using `cryptsetup
--version` command. It then parses the version string using a regular
expression to accommodate varying digit counts. If the retrieved version
is greater than or equal to the specified target version,
`conf.dmcrypt_no_workqueue` is set to True, allowing for flexible version
handling.

Fixes: https://tracker.ceph.com/issues/66393
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit dc28b77a6ea50b3390663ac02eeb80367650b7ed)

Merge pull request #56883 from guits/wip-65480-squid

squid: ceph-volume: osd objectstore refactor

Merge pull request #56931 from adk3798/wip-65515-squid

squid: mgr/dashboard: fix duplicate grafana panels when on mgr failover

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #58122 from ljflores/wip-66554-squid

squid: qa/rados: don't pass --max_attr_len to ceph_test_rados unconditionally

python-common/service_spec: fix some mypy complaints

The python/mypy combination on the jenkins nodes the CI
is running on don't seem to care, but locally I get

mypy: commands[0]> mypy --config-file=../mypy.ini -p ceph
ceph/deployment/service_spec.py: note: In member "validate" of class "NvmeofServiceSpec":
ceph/deployment/service_spec.py:1497: error: Unsupported operand types for > ("float" and "None")  [operator]
ceph/deployment/service_spec.py:1497: note: Left operand is of type "Optional[float]"
ceph/deployment/service_spec.py:1500: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1500: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1503: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1503: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1506: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1506: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1509: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1509: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1512: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1512: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1515: error: Unsupported operand types for > ("float" and "None")  [operator]
ceph/deployment/service_spec.py:1515: note: Left operand is of type "Optional[float]"
ceph/deployment/service_spec.py:1518: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1518: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1521: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1521: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1524: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1524: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1527: error: Unsupported operand types for > ("int" and "None")  [operator]
ceph/deployment/service_spec.py:1527: note: Left operand is of type "Optional[int]"
ceph/deployment/service_spec.py:1530: error: Unsupported operand types for > ("float" and "None")  [operator]
ceph/deployment/service_spec.py:1530: note: Left operand is of type "Optional[float]"
Found 12 errors in 1 file (checked 27 source files)

The errors make sense to me, so I think we should fix them

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 7520f65f261c61d01dd75b9722c84147229d9e87)

mgr/cephadm: update default NVMEoF container image version

Otherwise the nvmeof daemon fails to start up

i148-njpgei[69585]: Traceback (most recent call last):
i148-njpgei[69585]:   File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
i148-njpgei[69585]:     return _run_code(code, main_globals, None,
i148-njpgei[69585]:   File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
i148-njpgei[69585]:     exec(code, run_globals)
i148-njpgei[69585]:   File "/src/control/__main__.py", line 43, in <module>
i148-njpgei[69585]:     gateway.serve()
i148-njpgei[69585]:   File "/src/control/server.py", line 177, in serve
i148-njpgei[69585]:     omap_lock = OmapLock(omap_state, gateway_state)
i148-njpgei[69585]:   File "/src/control/state.py", line 201, in __init__
i148-njpgei[69585]:     self.omap_file_lock_retry_sleep_interval = self.omap_state.config.getint_with_default("gateway",
i148-njpgei[69585]:   File "/src/control/config.py", line 47, in getint_with_default
i148-njpgei[69585]:     return self.config.getint(section, param, fallback=value)
i148-njpgei[69585]:   File "/usr/lib64/python3.9/configparser.py", line 818, in getint
i148-njpgei[69585]:     return self._get_conv(section, option, int, raw=raw, vars=vars,
i148-njpgei[69585]:   File "/usr/lib64/python3.9/configparser.py", line 808, in _get_conv
i148-njpgei[69585]:     return self._get(section, conv, option, raw=raw, vars=vars,
i148-njpgei[69585]:   File "/usr/lib64/python3.9/configparser.py", line 803, in _get
i148-njpgei[69585]:     return conv(self.get(section, option, **kwargs))
i148-njpgei[69585]: ValueError: invalid literal for int() with base 10: '1.0'

I've been told 1.2.5 has a patch that allows this value to be a float

Fixes: https://tracker.ceph.com/issues/65718
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 0cebec2888d309e3c727ec2547692f680f3896af)

Merge pull request #58127 from petrutlucian94/wip-66547-squid

squid: ceph_test_rados_api_misc: adjust LibRadosMiscConnectFailure.ConnectTimeout timeout

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58116 from zdover23/wip-doc-2024-06-18-backport-57352-to-squid

squid: doc/dev/crimson: CPU cores allocation

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58105 from zdover23/wip-doc-2024-06-18-backport-58901-to-squid

squid: doc/dev: link to ceph.io leads list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58067 from Matan-B/wip-57788-squid

squid: crimson/os/seastore: implement disk and writer level stats reporting

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58066 from Matan-B/wip-57654-squid

squid: crimson/os/seastore: fix compile error in release build

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58032 from Matan-B/wip-57279-squid

squid: crimson/osd/pg: trigger wait_for_active_blocker on replica osds when the activate event is committed

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: clarify availability vs integrity

(cherry picked from commit 3888cf77075eceb079b1d5a8ce19092d12fd82bd)

ceph_test_rados_api_misc: adjust LibRadosMiscConnectFailure.ConnectTimeout timeout

The test sets a 2 second connection timeout and asserts that the
attempt takes less than 4 seconds.

Under load, the asserted timeout can be exceeded. In particular,
this affects the Windows CI job, which uses a VM with a reduced
amount of resources.

    [ RUN      ] LibRadosMiscConnectFailure.ConnectTimeout
    /home/ubuntu/ceph/src/test/librados/misc.cc:87: Failure
    Expected: (dur) < (utime_t(4, 0)), actual: 5.987712 vs 4.000000

We'll adjust the timeouts, using a 5s configured timeout, asserting
that the connection attempt doesn't take more than 15s.

Fixes: https://tracker.ceph.com/issues/66534
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 3a2b4a58898477ae9f62b506bd5be56ff9abacf0)

qa/rados: don't pass --max_attr_len to ceph_test_rados unconditionally

Fixes: https://tracker.ceph.com/issues/66321
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 8a6d56e62fc03062e60828a3bc9f37ed7232b3e7)

doc/dev/crimson: CPU cores allocation

CPU cores allocation and
add crimson-smp vstart flag

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 35d257bfaa5ed7cf91c8a8b523ed4494fdd27667)

Merge pull request #58093 from Matan-B/wip-55831-squid

squid: crimson/os/seastore: lba pointer consolidation

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58065 from Matan-B/wip-57383-squid

squid: crimson: fix user_version handling

Reviewed-by: Samuel Just <sjust@redhat.com>