squid: cephadm: update latest stable release and default image

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #59751 from rhcs-dashboard/wip-68041-squid

squid: mgr/dashboard: add gw_groups support to nvmeof api

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59722 from aaSharma14/wip-67990-squid

squid: mgr/dashboard: use grafana server instead of grafana-server in grafana 10.4.0

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59760 from rhcs-dashboard/squid-hosts-selection-show-only-10-entries-fix

Squid: mgr/dashboard: service form hosts selection only show up to 10 entries

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59798 from zdover23/wip-doc-2024-09-15-backport-59024-to-squid

squid: doc/README.md - add ordered list

Merge pull request #59802 from zdover23/wip-doc-2024-09-16-backport-59801-to-squid

squid: doc/rados: add osd_deep_scrub_interval setting operation

doc/rados: add osd_deep_scrub_interval setting operation

Add a second method of changing the value of osd_deep_scrub_interval to
remedy the condition indicated by the "PGs not deep-scrubbed in time"
warning.

This procedure was developed by Eugen Block, and is at the time of this
commit available on his blog at
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f57e99e1731af3affee1c299bb9a79852e31f5ea)

Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid

squid: doc/rados: add link to messenger v2 info in mon-lookup-dns.rst

doc/README.md - add ordered list

Add ordered-list formatting to the "Building Ceph" section of README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1f21057e4166f2ea9f51ee037eb8e234c1390cfb)

doc/rados: add link to messenger v2 info in mon-lookup-dns.rst

Add a link to the page about Messenger v2 to the end of
doc/rados/configuration/mon-lookup-dns.rst.

Fixes: https://tracker.ceph.com/issues/58752
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 81f9d064f714f42ff27bf620df526f87b393402b)

Merge pull request #59785 from zdover23/wip-doc-2024-09-14-backport-59732-to-squid

squid: doc/README.md: improve formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/README.md: improve formatting

Improve the formatting in the section "Building Ceph" in the file
README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit b9ca3957303989bbba9301cbbd18ba8faa0b8168)

Merge pull request #59767 from zdover23/wip-doc-2024-09-13-backport-59438-to-squid

squid: doc/rgw/d3n: pass cache dir volume to extra_container_args

Merge pull request #59718 from afreen23/wip-67994-squid

squid: mgr/dashboard: Update nvmeof microcopies

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

doc/rgw/d3n: pass cache dir volume to extra_container_args

Fixes: https://tracker.ceph.com/issues/66492
Signed-off-by: Mark Kogan <mkogan@ibm.com>
(cherry picked from commit 4594e38bf19a8f73191904f5f373a3707cddd2f3)

Merge pull request #59756 from zdover23/wip-doc-2024-09-12-backport-59753-to-squid

squid: doc/install: Keep the name field of the created user consistent with …

doc/install: Keep the name field of the created user consistent with the node name in the Start RADOSGW service command

If the user name does not match the name of the node that started the RADOSGW service, this will cause confusion for those who are new to ceph. Because they can't start the radosgw service normally as shown in the tutorial.

Signed-off-by: hejindong <wodeshijie3394@126.com>
(cherry picked from commit 128e6451d8c4764025d4a2565f15ec175c857bb2)

mgr/dashboard: service form hosts selection only show up to 10 entries

exposed new UIRouter API to get all hosts

Fixes: https://tracker.ceph.com/issues/67413
Signed-off-by: Naman Munet <nmunet@redhat.com>
(cherry picked from commit ec3e4abeeb7feebd996681a7d8d4e950de5cf518)

mgr/dashboard: expose gw_groups list api

also if the cephadm is available, make the api smart enough to
understand the running daemons and get its gateway_addr instead of
taking the first one in the list so that the HA will be there for the UI
and API as well

Fixes: https://tracker.ceph.com/issues/67774
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit c8434d67d08882ac88c579c6e199f59f05a31699)

mgr/dashboard: add gw_groups support to nvmeof api

Introduces a gw_group query-param to gateway and subsystem requests.

Modified the dashboard config to support adding more than one gateway to
the list.

Fixes: https://tracker.ceph.com/issues/67774
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 47f57d434d1690cc3c29dd048ec4cdedf618571f)

Merge pull request #59717 from afreen23/wip-67993-squid

squid: mgr/dashboard: Increase maximum namespace count to 1024

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59721 from aaSharma14/wip-67996-squid

squid: mgr/dashboard: Rephrase dedicated pool helper in rbd create form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #59733 from zdover23/wip-doc-2024-09-11-backport-59635-to-squid

squid: doc/rados: add "pgs not deep scrubbed in time" info

Merge pull request #59700 from zdover23/wip-doc-2024-09-10-backport-59675-to-squid

squid: doc/README.md: improve formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: add "pgs not deep scrubbed in time" info

Add a procedure to doc/rados/operations/health-warnings.rst that
explains how to remedy the "X PGs not deep-scrubbed in time" health
warning.

This procedure was developed by Eugen Block, and is at the time of this
commit available on his blog at
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d620a51c306d1ebcdfc974dc0b85dc46cfde8d40)

Merge pull request #59693 from rhcs-dashboard/squid-wrong-uid

Squid: mgr/dashboard: Wrong(half) uid is observed in dashboard when user created via cli contains $ in its name

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: use grafana server instead of grafana-server in grafana
10.4.0

The grafana-server command is deprecated in grafana v10.4.0. It is
advised to use grafan server in place of it

Fixes: https://tracker.ceph.com/issues/65479
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit ba7b6ec215701d17295b751835d29b18950be960)

mgr/dashboard: Rephrase dedicated pool helper in rbd create form

Fixes: https://tracker.ceph.com/issues/67893
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit a94a0ad634d3fdf637b74666a8a8758afc50b9e0)

Conflicts:
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-form/rbd-form.component.html (conflicts with carbon changes)

mgr/dashboard: Update nvmeof microcopies

- As per the review from the stakeholders , updating the microcopies in nvmeof.
- in task service file changed casing of `Service` to align with rest.

Fixes https://tracker.ceph.com/issues/67728

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>
(cherry picked from commit 5d1ee140d0514348faa1db323a18c69fcb8950e6)

mgr/dashboard: Increase maximum namespace count to 1024

Fixes https://tracker.ceph.com/issues/67871

The maximum namespace count will be 1024 for subsystems now.

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>
(cherry picked from commit 12c7c6522bac5a0e669ceabaa044e6a7cd479f88)

Merge pull request #59682 from zdover23/wip-doc-2024-09-10-backport-59677-to-squid

squid: doc/man: edit ceph-bluestore-tool.rst

doc/README.md: improve formatting

Improve the formatting of the section "Building Ceph" in the file
README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 7171f73c81d603f8ff62198fc1e5f50c20a0aa05)

mgr/dashboard: Wrong(half) uid is observed in dashboard

Fixes: https://tracker.ceph.com/issues/67850
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

doc/man: edit ceph-bluestore-tool.rst

Edit the section "bluefs-bdev-migrate" in
doc/man/8/ceph-bluestore-tool.rst to add the information that this
operation expands the target storage by updating its size label, making
"bluefs-bdev-expand" unnecessary.

Improve the subject-verb agreement in this section, and supply some
absent definite articles.

Co-authored-by: Peter Gervai <grin@drop.grin.hu>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 6b34707f827b2b197f53fe2e430d173b30b81401)

Merge pull request #58383 from adk3798/squid-fix-secure-monitoring-stack-check

squid: mgr/cephadm: redeploy when some dependency daemon is add/removed

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #59138 from k0ste/wip-67110-squid

squid: mgr/rgw: Adding a retry config while calling zone_create()

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59473 from adk3798/wip-67695-squid

squid: mgr/orchestrator: fix encrypted flag handling in orch daemon add osd

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59663 from zdover23/wip-doc-2024-09-09-backport-59642-to-squid

squid: doc/cephadm: add malformed-JSON removal instructions

Merge pull request #59596 from aaSharma14/wip-67792-squid

squid: mgr/dashboard: Zone details showing incorrect data for data pool values and compression info for Storage Classes

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59597 from aaSharma14/wip-66559-squid

squid: mgr/dashboard: Show which daemons failed in CEPHADM_FAILED_DAEMON healthcheck

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58912 from lxbsz/wip-67122

squid: mds/QuiesceDbManager: get requested state of members before iterating…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58984 from joscollin/wip-65223-squid

squid: cephfs_mirror: use snapdiff api for incremental syncing

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59068 from joscollin/wip-67382-squid

squid: qa/suites: drop --show-reachable=yes from fs:valgrind tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59070 from joscollin/wip-67387-squid

squid: cephfs_mirror: provide metrics for last successful snapshot sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephadm: add malformed-JSON removal instructions

Add instructions for removing malformed JSON that causes the cephadm
module to fail to start, which in turn makes the orchestrator fail to
start.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 93d90ea358cf17b08e502caf8529061a10008fad)

Merge pull request #59654 from zdover23/wip-doc-2024-09-08-backport-59628-to-squid

squid: doc/dev: add "activate latest release" RTD step

doc/dev: add "activate latest release" RTD step

Add a step to the section called "First Stable Release" that explains
how to activate the latest release in Read the Docs.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e7bd4bcdd074a3defae5866a6c035559bf210271)

Merge pull request #59637 from zdover23/wip-doc-2024-09-06-backport-59630-to-squid

squid: doc/README.md: edit "Build Prerequisites"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #59623 from guits/wip-67924-squid

squid: ceph-volume: switch over to new disk sorting behavior

doc/README.md: edit "Build Prerequisites"

Make the sentences in the "Build Prerequisites" section of README.md
more direct.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0b277fe3809c2eae7bfced002ac85f7e1629fa30)

Merge pull request #59598 from guits/wip-67892-squid

squid: ceph-volume: fix generic activation with raw osds

Merge pull request #59622 from guits/wip-67923-squid

squid: ceph-volume: pass self.osd_id to create_id() call

Merge pull request #59202 from k0ste/wip-67529-squid

squid: debian pkg: record python3-packaging dependency for ceph-volume

Merge pull request #59602 from idryomov/wip-67752-squid

squid: rbd: amend "rbd {group,} rename" and "rbd mirror pool" command descriptions

Reviewed-by: Ramana Raja <rraja@redhat.com>

ceph-volume: switch over to new disk sorting behavior by default

The automatic disk sorting behavior is deprecated.
The new behavior should be the default since Ceph Pacific but it
got pushed back multiple times.

Fixes: https://tracker.ceph.com/issues/67497
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit be725040bdff367c29262becdea269416b326b8f)

ceph-volume: pass self.osd_id to create_id() call

This fixes a regression
(most likely introduced with 04c93a1ed42)

Current (broken) behavior:
```
[root@devel1-1 /]# ceph osd tree
ID  CLASS  WEIGHT  TYPE NAME     STATUS     REWEIGHT  PRI-AFF
-1              0  root default
 0              0  osd.0         destroyed         0  1.00000
[root@devel1-1 /]# ceph-volume raw prepare --bluestore --data /dev/vdb --osd-id 0
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 543e0268-ea66-451f-a2f6-95ae60ab6146
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-1
Running command: /usr/bin/chown -R ceph:ceph /dev/vdb
Running command: /usr/bin/ln -s /dev/vdb /var/lib/ceph/osd/ceph-1/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap
 stderr: got monmap epoch 1
--> Creating keyring file for osd.1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid 543e0268-ea66-451f-a2f6-95ae60ab6146 --setuser ceph --setgroup ceph
 stderr: 2024-09-04T11:46:54.283+0000 7f0027d2b640 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
 stderr: 2024-09-04T11:46:54.283+0000 7f0027d2b640 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
 stderr: 2024-09-04T11:46:54.283+0000 7f0027d2b640 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
 stderr: 2024-09-04T11:46:54.283+0000 7f0027d2b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
--> ceph-volume raw dmcrypt prepare successful for: /dev/vdb
[root@devel1-1 /]# ceph osd tree
ID  CLASS  WEIGHT  TYPE NAME     STATUS     REWEIGHT  PRI-AFF
-1              0  root default
 0              0  osd.0         destroyed         0  1.00000
 1              0  osd.1              down   1.00000  1.00000
```

A new osd id has been generated while we want to reuse the existing one (0 in this case).

Expected (fixed) behavior:

```
[root@devel1-1 /]# ceph osd tree
ID  CLASS  WEIGHT  TYPE NAME     STATUS     REWEIGHT  PRI-AFF
-1              0  root default
 0              0  osd.0         destroyed         0  1.00000
[root@devel1-1 /]# ceph-volume raw prepare --bluestore --data /dev/vdb --osd-id 0
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c87841fb-f56f-40a2-a653-62bb092e8e95 0
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -R ceph:ceph /dev/vdb
Running command: /usr/bin/ln -s /dev/vdb /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
 stderr: got monmap epoch 1
--> Creating keyring file for osd.0
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid c87841fb-f56f-40a2-a653-62bb092e8e95 --setuser ceph --setgroup ceph
 stderr: 2024-09-04T13:07:39.125+0000 7ff02baaa640 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
 stderr: 2024-09-04T13:07:39.125+0000 7ff02baaa640 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
 stderr: 2024-09-04T13:07:39.126+0000 7ff02baaa640 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
 stderr: 2024-09-04T13:07:39.126+0000 7ff02baaa640 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
--> ceph-volume raw dmcrypt prepare successful for: /dev/vdb
[root@devel1-1 /]# ceph osd tree
ID  CLASS  WEIGHT  TYPE NAME     STATUS  REWEIGHT  PRI-AFF
-1              0  root default
 0              0  osd.0           down         0  1.00000
```

Fixes: https://tracker.ceph.com/issues/67898
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 5fc36f6463c247ca108b8f123135dd15aae5f13b)

Merge pull request #59492 from ljflores/wip-bz-2305110-squid

squid: osd: fix require_min_compat_client handling for msr rules

qa/tasks/ceph_manager: set-require-min-compat-client to squid for msr profiles

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 4f9289e11ae4118f7d26bb154d04288c8fd87a4b)

rbd: mention namespace in "rbd mirror pool" command descriptions

Commit 5e64748927d0 ("doc/rbd: add namespace information for mirror
commands") did this for the man page, update the built-in help as well.

The "by default" bit in the description of "rbd mirror pool enable" and
"rbd mirror pool disable" commands is specific to pool mode which is in
turn specific to journal-based mirroring, so it's removed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 7e02946a3f60da0dcc99e0b4a4b6b3d359db1c15)

rbd: clarify "rbd {group,} rename" command description

Make it clearer that, despite a full image or group spec being taken
for source and destination, an image or a group can be renamed only
within its pool or namespace.

Rename across pools or namespaces within the same pool is unsupported.

Fixes: https://tracker.ceph.com/issues/67752
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5f064a52707af58fd6007734a768899aeb5a32fc)

Merge pull request #59582 from zdover23/wip-doc-2024-09-04-backport-59546-to-squid

squid: doc/rados/operations: Improve health-checks.rst

Merge pull request #59585 from zdover23/wip-doc-2024-09-04-backport-59577-to-squid

squid: doc/mds: improve wording

ceph-volume: fix generic activation with raw osds

Typical failure:

```
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
--> Failed to activate via raw: activate() takes 1 positional argument but 5 were given
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
--> Failed to activate via LVM: could not find osd.0 with osd_fsid 1e764f4a-db4b-4b41-86eb-468efe4c3f44
--> Failed to activate via simple: 'Namespace' object has no attribute 'json_config'
--> Failed to activate any OSD(s)
```

04c93a1ed42 seems to have broken it.
This commit fixes it.

Fixes: https://tracker.ceph.com/issues/67873
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 710747031b78948d4a373036f5b8c41269b5a399)

mgr/dashboard: Show which daemons failed in CEPHADM_FAILED_DAEMON
healthcheck

Fixes: https://tracker.ceph.com/issues/63792
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 5a44b395d4fd86f51fe7431a8f8f539ee57eeee7)

mgr/dashboard: Zone details showing incorrect data for data pool values and compression info for Storage Classes

Fixes: https://tracker.ceph.com/issues/67103
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 76af1c0b1be562d38f3d6323351b7b7d0462af48)

ceph-volume: add "packaging" to install_requires

in 0985e201, "packaging" was introduced as a runtime dependency of
ceph-volume, and `ceph.spec.in` was updated accordingly to note
this new dependency. but the debian packaging was not updated.

in 80edcd40, the missing dependency was added to debian/control as
one of ceph-volume's runtime dependency.

but dh_python3 is able to figure out the dependencies by reading
the egg's metadata of the ceph-volume python module. and as a
python project, ceph-volume is using its `setup.py` for
tracking its dependencies.

so in order to be more consistent, and keep all of its dependencies
in one place, let's move this dependency to setup.py . as the
packagings in both distros are able to figure the dependencies
from egg-info.

see also
- https://manpages.debian.org/testing/dh-python/dh_python3.1.en.html#dependencies
- https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/#_automatically_generated_dependencies

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 729fd8e25ff2bfbcf99790d6cd08489d1c4e2ede)

debian pkg: record python3-packaging dependency for ceph-volume

Since commit 0985e201342 ("ceph-volume: use 'no workqueue' options
with dmcrypt") the python "packaging" module is used to parse the
cryptsetup version output, but the debian packaging was not updated to
record that new dependency.

So simply record this in the d/control file, adding a <pkg>.requires
file seemed to not really winning us anything here.

Fixes: https://tracker.ceph.com/issues/67290
Fixes: 0985e201342fa53c014a811156aed661b4b8f994
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 80edcd40e8092d9fb3b45c1a6c7f9b7f4f37b69e)

Merge pull request #59446 from afreen23/wip-67724-squid

squid: mgr/dashboard: Add group field in nvmeof service form

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/mds: improve wording

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>
(cherry picked from commit 332804bad58c892d01d2d2da557e42104365ef8a)

doc/rados/operations: Improve health-checks.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 2aa82539ede8ade095ad0b22e973eac538354711)

Merge pull request #59578 from zdover23/wip-doc-2024-09-03-backport-59545-to-squid

squid: doc/radosgw: Improve config-ref.rst

doc/radosgw: Improve config-ref.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 8e743fc603310a802b8553af5e73266a9caf6554)

mgr/dashboard: Fixed bug in service form

The commit in main went through this PR https://github.com/ceph/ceph/pull/59496/commits/6c25635be8c2ec0c2f637c4fe6952f781c1ad12c

This PR wont be backported to squid hence adding the change in separate commit.

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59559 from zdover23/wip-doc-2024-09-02-backport-59556-to-squid

squid: doc: Correct link to Prometheus docs

doc: Correct link to Prometheus docs

The link is to the `#http_sd_config` anchor in the prometheus config docs; that link only works without the trailing `/`.

This correction would ideally get backported to at least reef & squid.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
(cherry picked from commit 84a30ba6b94b34806faac8217ccaa299c9ee68d6)

Merge pull request #59548 from zdover23/wip-doc-2024-09-01-backport-59544-to-squid

squid: doc: update tests-integration-testing-teuthology-workflow.rst

Reviewed-by: Vallari Agrawal <val.agl002@gmail.com>

doc: update tests-integration-testing-teuthology-workflow.rst

* add "Infrastructure" section.

* move "Naming the ceph-ci branch" section under
   "Getting binaries - Build Ceph". Also mention
   about centos9-only trick.

* in "Teuthology Archives", mention about developer
   playground machines and ceph log files.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
(cherry picked from commit 9bfcb8e17db8c61e523e10856d12b237433d831a)

Merge pull request #59464 from sajibreadd/wip-67037-squid

squid: os/bluestore: Warning added for slow operations and stalled read

Merge pull request #59540 from zdover23/wip-doc-2024-08-31-backport-59528-to-squid

squid: doc/ceph-volume: add spillover fix procedure

doc/ceph-volume: add spillover fix procedure

Add a procedure that explains how, after an upgrade, to move bytes that
have spilled over to a relatively slow device back to the faster device.

This procedure was developed by Chris Dunlop on the [ceph-users] mailing
list, here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/POPUFSZGXR3P2RPYPJ4WJ4HGHZ3QESF6/

Eugen Block requested the addition of this procedure to the
documentation on 30 Aug 2024.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 98618aaa1c8b786c7d240a210b62cc737fdb048d)

Merge pull request #59537 from neha-ojha/wip-squid-stable

squid: src/ceph_release, doc: mark squid stable

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

src/ceph_release, doc: mark squid stable

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #59491 from cbodley/wip-67782-squid

squid: doc/rgw/account: Handling notification topics when migrating an existing user into an account

Reviewed-by: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #59084 from kamoltat/wip-ksirivad-backport-squid-56233

squid: RADOS: Generalize stretch mode pg temp handling to be usable without stretch mode
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #59221 from cbodley/wip-67554-squid

squid: rgw: revert account-related changes to get_iam_policy_from_attr()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mon/OSDMonitor: check crush features in crush_rule_create_erasure

Also adjusts validate_crush_against_features to use an ostream&
rather than a stringstream&.

Fixes: https://tracker.ceph.com/issues/67755
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1d6b4d4a507a1241a5e03016231e8160ef4b8f92)

osd: use CEPH_FEATUREMASK_CRUSH_MSR, not CEPH_FEATURE_CRUSH_MSR

This was simply a mistake:

uint64_t features = 0;
features |= CEPH_FEATURE_CRUSH_MSR;
ceph_assert(HAVE_FEATURE(features, CRUSH_MSR));

will fail the assert as CEPH_FEATURE_CRUSH_MSR lacks the mask.  In
basically all cases, CEPH_FEATUREMASK_* is the correct choice.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit cb157b45c713e76ec95d8b80548998587e02a758)

osd/OSDMap: require CRUSH_MSR if any rule is msr, even if used by no pool

OSDMap::get_features is used by
OSDMonitor::validate_crush_against_features via
OSDMap::get_min_compat_client() to check whether changes to the crushmap
will require newer features than the existing require_min_compat_client
field.

Monitor commands which create rules from ec profiles may result in msr
rules.  While it might be harmless to allow msr rules to exist as long
as there aren't any pools actually using the rule, it's probably simpler
to disallow their creation in the first place until
require_min_compat_client is updated.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 2130115ec2252e37403a377c482e9599a5c81a66)

mon: validate everybody understands MSR on set-require-min-compat-client

Unit testing
------------
```
[rzarzynski@o06 build]$ bin/unittest_features
...
[ RUN      ] features.release_features
1 argonaut features 0x40000 looks like argonaut
2 bobtail features 0x40000 looks like argonaut
3 cuttlefish features 0x40000 looks like argonaut
4 dumpling features 0x42040000 looks like dumpling
5 emperor features 0x42040000 looks like dumpling
6 firefly features 0x20842040000 looks like firefly
7 giant features 0x20842040000 looks like firefly
8 hammer features 0x1020842040000 looks like hammer
9 infernalis features 0x1020842040000 looks like hammer
10 jewel features 0x401020842040000 looks like jewel
11 kraken features 0xc01020842040000 looks like kraken
12 luminous features 0xe01020842240000 looks like luminous
13 mimic features 0xe01020842240000 looks like luminous
14 nautilus features 0xe01020842240000 looks like luminous
15 octopus features 0xe01020842240000 looks like luminous
16 pacific features 0xe01020842240000 looks like luminous
17 quincy features 0xe01020842240000 looks like luminous
18 reef features 0xe010208d2240000 looks like reef
19 squid features 0xe010248d2240000 looks like squid
[       OK ] features.release_features (0 ms)
```

Manual testing
--------------
\### `reef` client present in `squid` cluster
```
[rzarzynski@o06 build]$ bin/ceph daemon mon.a sessions | jq  -jr '.[] | .name, "\t", .con_features, "\t", .con_features_hex, "\n"' | grep client
client.?        4540701547738038271     3f03cffffffdffff
client.?        4540138322906710015     3f01cfbffffdffff
[rzarzynski@o06 build]$ bin/ceph osd get-require-min-compat-client
luminous
[rzarzynski@o06 build]$ bin/ceph osd set-require-min-compat-client squid
Error EPERM: cannot set require_min_compat_client to squid: 1 connected client(s) look like reef (missing 0x4000000000); add --yes-i-really-mean-it to do it anyway
```

\### only `squid` clients and `squid` cluster
```
[rzarzynski@o06 build]$ bin/ceph daemon mon.a sessions | jq  -jr '.[] | .name, "\t", .con_features, "\t", .con_features_hex, "\n"' | grep client
client.?        4540701547738038271     3f03cffffffdffff
client.?        4540701547738038271     3f03cffffffdffff
[rzarzynski@o06 build]$ bin/ceph osd set-require-min-compat-client squid
set require_min_compat_client to squid
```

Fixes: https://tracker.ceph.com/issues/66297
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 4b54b07640b56e387716f2e73d9004fa7757f4e3)

doc/rgw/account: Handling notification topics when migrating an existing user into an account

Add a subsection under "Migrate an existing User into an Account" to
describe how a client can seamlessly migrate the notification topics
after account migration.

Fixes https://tracker.ceph.com/issues/67656

Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
(cherry picked from commit 574d3f8e73570d06f259cdb9c36429effa6fb426)

qa/suites/rados: 3-az-stretch-cluster-netsplit test

Test the case where 2 DC loses connection with each other
for a 3 AZ stretch cluster with stretch pool enabled.
Check if cluster is accessible and PGs are active+clean
after reconnected.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 7b41aff3f001d247e4ea5d73b774b5f7709eda49)

Conflicts:
- qa/tasks/ceph_test_case.py (trivial fix)

doc/rados/operations/pools.rst: Added docs for stretch pool set|unset|show

Fixes: https://tracker.ceph.com/issues/64802
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit fb0011a692c656db0c64bbeb7d47d79cc62b16c9)

qa/suites/rados/singleton/all: init mon-stretch-pool.yaml

Test the following new Ceph CLI commands:

`ceph osd pool stretch set`
`ceph osd pool stretch unset`
`ceph osd pool stretch show`

`qa/workunits/mon/mon-stretch-pool.sh`

will create the stretch cluster
while performing input validation for the CLI
Commands mentioned above.

`qa/tasks/stretch_cluster.py`

is in charge of
setting a pool to stretch cluster
and checks whether it prevents PGs
from the going active when there is not
enough buckets available in the acting
set of PGs to go active.

Also, test different MON fail over scenarios
after setting pool as stretch

`qa/suites/rados/singleton/all/mon-stretch-pool.yaml`

brings the scripts together.

Fixes: https://tracker.ceph.com/issues/64802
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 4ca1320727d1f7dca2ddfe7ad1a0b76f95b80ab0)

Conflicts:
- qa/tasks/ceph_test_case.py (trivial fix)

Allow setting peering_crush_bucket_{count|target|barrier}

In the command `ceph osd pool stretch set`

<pool> <peering_crush_bucket_count>
<peering_crush_bucket_target> <peering_crush_bucket_barrier>
<crush_rule> <size> <min_size>

user has the option of setting the value of `peering_crush_bucket_{count|target|barrier}`.
This will then allow the utilization `calc_replicated_acting_stretch`,
since with `peering_crush_bucket_count != 0`
the pool is now a stretch_pool and we can handle pg_temp
better by settubg barriers and limits to how much OSDs
should be in a pg_temp.

This will enable the specify pool to
handle pg_temp properly during create_acting, as a stretch pool
should.

User can also use the command:
`osd pool stretch show <pool> `

to show all the stretch related information for the pool

pool: cephfs.a.data
pool_id: 3
is_stretch_pool: 1
peering_crush_bucket_count: 3
peering_crush_bucket_target: 3
peering_crush_bucket_barrier: 8
crush_rule: replicated_rule_custom
size: 3
min_size: 2

User can also unset the stretch pool wiith the commnad:
`osd pool stretch unset <pool>`
However, the pool must be a stretch pool.

Fixes: https://tracker.ceph.com/issues/64802
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 661e8e22cff8763cf8775f204dda4490e464db2c)

Merge pull request #59083 from kamoltat/wip-ksirivad-backport-squid-57381

squid: PeeringState.cc: Only populate want_acting when num_osds < bucket_max
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58669 from kamoltat/wip-ksirivad-backport-squid-57906

squid: mon/ElectionLogic: tie-breaker mon ignore proposal from marked down mon
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #59471 from adk3798/wip-67732-squid

squid: qa/distros: reinstall nvme-cli on centos 9 nodes

Merge pull request #59038 from aaSharma14/wip-67359-squid

squid: mgr/dashboard: Dashboard not showing Object/Overview correctly.

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59380 from zdover23/wip-doc-2024-08-21-backport-59348-to-squid

squid: doc/rados: document unfound object cache-tiering scenario

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

mgr/orchestrator: fix encrypted flag handling in orch daemon add osd

The current implementation incorrectly parses this `encrypted` flag as a string rather than a boolean value.
This leads to unintended behavior causing an LVM encryption layer to be created regardless of whether `encrypted=True` or `encrypted=False` is passed.
The only way to prevent this behavior is by omitting the `encrypted` flag entirely.
This change prevents potential errors, aligning the behavior with user expectations.

Fixes: https://tracker.ceph.com/issues/67372
Signed-off-by: Yonatan Zaken <yonatan.zaken@gmail.com>
(cherry picked from commit 42721c03ee6f2c47a20dfb4d40af4f7f7afe6113)