qa/tasks/mgr/test_insights: Remove test for persistent checks

This test makes no sense if we are no longer persisting the store.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 32d1cca2d9b606915c590f52d61856ee401fb4fc)

pybind/mgr/insights: Don't persist report data

Don't store health reports in rocksdb.

Fixes: https://tracker.ceph.com/issues/48269
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit de66522517edd6f7baf19cc0660478502d3c25e8)

mgr/tests: Fix typo in function name

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 11268b332f582d658553774f0a9b9662da6c02ef)

Merge pull request #42868 from sebastian-philipp/pacific-backport-42852-42863

pacific: qa/cephadm: centos_8.x_container_tools_3.0.yaml

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #42850 from ceph/wip-81356-pacific

pacific: run-make-check.sh: Increase failure output log size

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

qa/distro: Add centos_8.2_container_tools_3.0.yaml

Let's avoid latest kubic stable

Fixes: https://tracker.ceph.com/issues/52279
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit e436483c77adf7f876533c25a11c14d19a2d32a4)

Conflicts:
qa/suites/orch/cephadm/dashboard/0-distro/centos_8.2_kubic_stable.yaml
qa/suites/orch/cephadm/smoke/distro/centos_8.2_kubic_stable.yaml

qa/cephadm: centos_8.3_container_tools_3.0.yaml

Let's avoid latest stable

Fixes: https://tracker.ceph.com/issues/52279
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit c6bd968eff96666da7e321fb5787c82dc66ddbb7)

Merge pull request #42211 from ideepika/wip-51549-pacific

pacific: monitoring/grafana/build/Makefile: revamp for arm64 builds, pushes to docker and quay, jenkins

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

run-make-check.sh: Increase failure output log size

Fixes: https://tracker.ceph.com/issues/43566
Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit e1b55f6878cfe847112959885d2fc44e91f3c44f)

Merge pull request #42736 from sebastian-philipp/pacific-backport-41854-42108-42134-42017-41906-42032-41829-41938-42177-42092-

pacific: cephadm: August batch 1

Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #42753 from guits/wip-52133-pacific

pacific: ceph-volume: work around phantom atari partitions

Merge pull request #42829 from tchaikov/pacific-pr-42811

pacific: mgr/dashboard/api: set a UTF-8 locale when running pip

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard/api: set a UTF-8 locale when running pip

ansible-core started to include files whose filenames are encoded in
non-ascii characters, so we have to use a more capable encoding for the
locale in order to install this package. otherwise we'd have following
error:

Collecting ansible-core<2.12,>=2.11.3
  Using cached ansible-core-2.11.4.tar.gz (6.8 MB)
ERROR: Exception:

Traceback (most recent call last):
  File "/tmp/tmp.fX76ASIrch/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
    status = self.run(options, args)
...
  File "/tmp/tmp.fX76ASIrch/venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 226, in untar_file
    with open(path, "wb") as destfp:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 137-140: ordinal not in range(256)

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 05e4145856bb5ed19ecc879f2e50b5a88cb2045e)

Merge pull request #42819 from rhcs-dashboard/wip-52293-pacific

pacific: mgr/dashboard: disable create snapshot with subvolumes

Merge pull request #42817 from rhcs-dashboard/wip-52294-pacific

pacific: mgr/dashboard: Refresh button on the iscsi targets page

Merge pull request #42812 from aaSharma14/wip-52135-pacific

pacific: mgr/dashboard: monitoring: replace Grafana JSON with Grafonnet based code

Merge pull request #42702 from joscollin/wip-52084-pacific

pacific: pybind/mgr/stats: validate cmdtag

Reviewed-by: Ramana Raja <rraja@redhat.com>

mgr/dashboard: disable create snapshot with subvolumes

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit b38259b62950ed2b15175935af7de00059926f3f)

mgr/dashboard: Refresh button on the iscsi targets page

Added a refresh button on the iscsi targets page. I've noticed that the
auto reload causes some load on the backend. So I disabled the auto
reload and went for the same approach as we have on rgw. A yellow
warning color on refresh btn to let the user know to manually refresh
that whenever needed.

Fixes: https://tracker.ceph.com/issues/52102
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 8f23a587a8ccc686b1e3acbfa587fcab76bee45d)

mgr/dashboard: fix grafonnet build error

This PR tends to fix the issue caused by #42194

Fixes:https://tracker.ceph.com/issues/52238
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 4907c78bb732b613ae223d3a7d1a046da457d1cc)

Conflicts:
ceph.spec.in (added jsonnet package)
monitoring/grafana/dashboards/CMakeLists.txt (added grafana jsons code)
(cherry picked from commit 570160e291b9029c7998d4dd4fee33327da91312)

Merge pull request #42788 from rhcs-dashboard/wip-52248-pacific

pacific: mgr/dashboard: tox.ini: delete useless env. 'apidocs'

Merge pull request #42789 from rhcs-dashboard/wip-52250-pacific

pacific: mgr/dashboard: cephadm e2e start script: add --expanded option

Merge pull request #42810 from rhcs-dashboard/wip-52291-pacific

pacific: mgr/dashboard: fix rename inventory to disks

mgr/dashboard: monitoring: replace Grafana JSON with Grafonnet based Code

This PR intends to add grafonnet to generate grafana JSON files

Fixes: https://tracker.ceph.com/issues/45184
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit e9bd94515f865b6c8865210b6329c11f3ebe9d23)

Conflicts:
debian/control (added jsonnet package)
run-make-check.sh (switched ON the WITH_GRAFANA bool for make check)

mgr/dashboard: fix rename inventory to disks

Fix Inventory breadcrumbs

Fixes: https://tracker.ceph.com/issues/50314
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>
(cherry picked from commit 23f44e894462e5658f8bcb85899df009fe379379)

mgr/dashboard: fix rename inventory to disks

Fix Orchestrator e2e tests

Fixes: https://tracker.ceph.com/issues/50314
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>
(cherry picked from commit 15b3665f27d7ceb2de74796a365d4bd19d716ff6)

mgr/dashboard: fix rename inventory to disks
Fix failure check in Host detail component

Fixes: https://tracker.ceph.com/issues/50314
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>
(cherry picked from commit 356c3c3bbf18a1dbce05d5b7b190df5ef69112e0)

mgr/dashboard: fix rename inventory to disks
Inventory -> Physical Disks changed in total 3 locations

Fixes: https://tracker.ceph.com/issues/50314
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>
(cherry picked from commit f2aa58a5faaa8aa47c4245fc557a584a97496e67)

mgr/dashboard: fix rename inventory to disks

Fixes: https://tracker.ceph.com/issues/50314
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>
(cherry picked from commit 6c609ace4998a087f21fa55c576b1f3cf1a0e054)

mgr/dashboard: fix rename inventory to disks

Fixes: https://tracker.ceph.com/issues/50314
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>
(cherry picked from commit fe0bbdaf6a1da13b76a261f8aebe181fd66bee91)

Merge pull request #42659 from cfsnyder/wip-50386-pacific

pacific: SimpleRADOSStriper: use debug_cephsqlite

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42693 from tchaikov/pacific-pr-41215

pacific: cmake: Replace boost download url

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #42692 from tchaikov/pacific-pr-42681

pacific: cmake: link bundled fmt statically

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #42561 from smithfarm/wip-51969-pacific

pacific: rpm/luarocks: simplify conditional and support Leap 15.3

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #42544 from cybozu/pacific-kv-rocksdbstore-enrich-debug-message

pacific: kv/RocksDBStore: enrich debug message

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42542 from smithfarm/wip-51955-pacific

pacific: compression/snappy: use uint32_t to be compatible with 1.1.9

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42224 from cfsnyder/wip-51568-pacific

pacific: mon/OSDMonitor: resize oversized Lec::epoch_by_pg, after PG merging, preventing osdmap trimming

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42536 from batrick/i51940

pacific: MDSMonitor: monitor crash after upgrade from ceph 15.2.13 to 16.2.4

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42773 from neha-ojha/wip-52244-pacific

pacific: os/bluestore: make deferred writes less aggressive for large writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42135 from dang/wip-dang-41991-backport

pacific: RGW - Bucket Remove Op: Pass in user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40696 from singuliere/wip-50156-pacific

pacific: rgw/notification: fix persistent notification hang when ack-levl=none

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #40695 from singuliere/wip-50157-pacific

pacific: rgw/notification: fixing the "persistent=false" flag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40693 from singuliere/wip-50214-pacific

pacific: rgw: objectlock: improve client error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: run-cephadm-e2e-tests.sh improvements

- Jenkins env.: make sure the cluster is always started.
- PR template: add trigger phrase to the jenkins commands list.
- Cypress: add --no-install flag; clean previous reports.

Fixes: https://tracker.ceph.com/issues/52082
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit cadf8c75ac0266e42c1ea0b4d6733590ea3dac21)

mgr/dashboard: cephadm e2e start script: add --expanded option

- Additional improvements: clean npm cache in jenkins env.; display mgr logs on error.

Fixes: https://tracker.ceph.com/issues/52082
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 973a6f534383e84bd92b690eaf2862a5a30ba70c)

mgr/dashboard: tox.ini: delete useless env. 'apidocs'

Fixes: https://tracker.ceph.com/issues/52130
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit d7346ec1486d7ca24177c3064f457f9722c0c8cb)

test/store_test: more testing for deferred writes

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 324ea6b425e5898eca6e86264fa4d34cb868a6ae)

os/bluestore: enforce one more non-inclusive comparision against prefer_deferred_size

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit cb30c9965854e7dbfe851ed0cfaf113b16bcab5e)

os/bluestore: Better handling of deferred write trigger

Now deferred write in _do_alloc_write does not depend on blob size,
but on size of extent allocated on disk.
It is now possible to set bluestore_prefer_deferred_size way larger than
bluestore_max_blob_size and still get desired behavior.
Example: for deferred=256K, blob=64K : when op write is 128K both blobs will be
written as deferred. When op write is 256K then all will go as regular write.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 06ac561a72a93bd24d05b6c2266d66fee49228b8)

os/bluestore: account for alignment with max_blob_size when splitting
write I/O into chunks.

Without the fix the following write seq:
0~4M
4096~4M

produces tons of deferred writes at the second stage.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit b06bcf99f50cf7db5856eb1f6cd56100a0d0bbbc)

os/bluestore: introduce l_bluestore_write_deferred_bytes perf counter.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f38f18c149b9347a26d845539eaeef29979e9006)

os/bluestore: use non-inclusive comparision against prefer_deferred_size

Fixes: https://tracker.ceph.com/issues/52089
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f1c4448fc549ed58bb58f920dfa16886e4910315)

os/bluestore: improve logging around deferred writes

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 89a20d63ec7e255d36363bd8977c3fca1d277877)

os/bluestore: fix missing 'l_bluestore_write_deferred' perf counter increment

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dc367c4bbc701b04a5bde562e30ab5b9c1705657)

os/bluestore: cosmetic cleanups

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 080531d6813b868eda562cf27babe18d8a974413)

Merge pull request #42679 from callithea/wip-51992-pacific

pacific: monitoring/grafana/cluster: use per-unit max and limit values

Merge pull request #42755 from guits/wip-50896-pacific

pacific: ceph-volume: Consider /dev/root as mounted

ceph-volume: work around phantom atari partitions

See Rook issue https://github.com/rook/rook/issues/7940 for full
information.

Ceph bluestore disks can sometimes appear as though they have "phantom"
Atari (AHDI) partitions created on them when they don't in reality. This
is due to a series of bugs in the Linux kernel when it is built with
Atari support enabled. This behavior does not appear for raw mode OSDs on
partitions, only on disks.

Changing the on-disk format of Bluestore OSDs comes with
backwards-compatibility challenges, and fixing the issue in the Kernel
could be years before users get a fix. Working around the Kernel issue
in ceph-volume is therefore the best place to fix the issue for Ceph.

To work around the issue in Ceph volume, there are two behaviors that need
adjusted:
1. `ceph-volume inventory` should not report that a partition is
   available if the parent device is a BlueStore OSD.
2. `ceph-volume raw list` should report parent disks if the disk is a
   BlueStore OSD and not report the disk's children, BUT it should still
   report children if the parent disk is not a BlueStore OSD.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
(cherry picked from commit 9212420bbc88e0caaf0b2302dd1f379f563f7d26)

ceph-volume: use safer check for bluestore label

Using only the exit status of `ceph-bluestore-tool show-label` to
determine if a device is a bluestore OSD could report a false negative
if there is a system error when `ceph-bluestore-tool` opens the device.

A better check is to open the device and read the bluestore device
label (the first 22 bytes of the device) to look for the bluestore
device signature ("bluestore block device"). If ceph-volume fails to
open the device due to a system error, it is safest to assume the device
is BlueStore so that an existing OSD isn't overwritten.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
(cherry picked from commit 651b28f2e3cb39dbe9c7038cd677a01523f08821)

ceph-volume: consider /dev/root mounted

Fixes: https://tracker.ceph.com/issues/50604
Signed-off-by: David Caro <david@dcaro.es>
(cherry picked from commit c7b959c35ff463cfbffa2abf0f56a1c1a5f594c8)

Merge pull request #42493 from dsavineau/wip-51852-pacific

pacific: ceph-volume: lvm batch: fast_allocations(): avoid ZeroDivisionError

Merge pull request #42546 from dsavineau/wip-51958-pacific

pacific: ceph-volume/tests: retry when destroying osd

cephadm: fix container name detection

'enter' was broken because we weren't correctly identifying the container
name.  Strip the newline from the inspect result so that we can reliably
match against the 'running' state.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit fb5e4b3740241d5c41b348e0fc5f59ebde5ee558)

cephadm: add AlmaLinux to supported distro

AlmaLinux [1] is an open-source 1:1 binary compatible fork of RHEL.

[1] https://almalinux.org/

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit f450f25bcf2ddd3322efca8f0c21eba90178840d)

doc/cephadm: update documentation

This commit fixes a couple of typos in the cephadm documentation.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 32b2c57bf66ccb030e24f78e3a5902babc1bbb38)

cephadm: fix unit tests.

Fixes AttributeError: 'FakePipeWrapper' object has no attribute 'readable'

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 61cfaec7002caadb3b4e09d672919ed159853311)

doc/cephadm: Offline hosts will pause upgrades

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 1eb48e3876d0b1fab7d1fac2574639112d39a8bf)

cephadm: Introduce unit.stop

Reason is we now have to stop two containers named differently. This is possible
with `bash -c ... echo %i | tr . -`. But we're not gainig anything readability
compared to putting it into a unit.stop script.

As not all daemons have this stop script, we still have to call podman for old daemons.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 0f66a5198e38c59aba1be11d586d49ad01c9afce)

cephadm: don't log on exit.

At this point, the garbage collector already colleceted the `open` function

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 71162c6fea7aba2e30cf50d8a0f1a5a0cf3b2462)

cephadm: `stats` might return `--` for containers

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit d183661be552acf7f30fce2e038173c5bdfa8421)

cephadm: exec: also search for old cname

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit a57ab6f50968921b26b87695e704f84a25101597)

cephadm: dashes: use both cnames for `inspect`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit a1f9959df474e671e979849450f8a00d0e24ea72)

cephadm: Add CephContainer.for_daemon()

To avoid calling get_container()

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit bb0587b8e0bde24d9bbfed7c65e0d351c5f7446b)

cephadm: use dashes for container names

podman adds the current container name to the /etc/hosts
file. Turns out, python's `socket.getfqdn()` differs from
`hostname -f`, when we have the container names containing
dots in it.:

[root@sebastians-laptop /]# cat /etc/hosts
127.0.0.1   localhost
::1         localhost
127.0.1.1   sebastians-laptop foo.bar.baz.com
[root@sebastians-laptop /]# hostname -f
sebastians-laptop
[root@sebastians-laptop /]# python3 -c 'import socket; print(socket.getfqdn())'
foo.bar.baz.com

Fascinatingly, this doesn't happen when using dashes.

Fixes: https://tracker.ceph.com/issues/51590
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 67abea15b12c1a60a9da2db35e3470d8d8a128f7)

doc/cephadm: MON IP change

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 9df20490b210cf0e5fcf17920e91d390989cc4bf)

doc/cephadm: Move some sections from mon.rst to serivce-management.rst

Avoid duplication and instead only reference the corresponding sections.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 24b753f2c8111305cf73de9b40a393c0c1a9bae7)

cephadm: Clarify no container engine message

Multiple people didn't understand the message. Let's try to impove it.

(Also make mypy happy)

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 240b2435bad903b811a3279b0e7a5d0806ee0b93)

cephadm: Add missing type annotations

Turnd out `command_maintenance` missing an if-branch that returned
None instead of a str

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 0c78d239a1933338fea14e21fde2638a418bc660)

Conflicts:
src/cephadm/cephadm

cephadm: set global default container for ingress

This was partially done in b94c8de but only for haproxy in the cephadm
mgr module not in the cephadm binary.
This adds the same change for keepalived container image.
Now both haproxy and keepalived container images are fully qualified
(registry + namespace + image).

Fixes: https://tracker.ceph.com/issues/51973
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit f9eb7d826563ab9b995b4123c03f6d1b89423e01)

doc/cephadm: update about disabling logging to journald for quincy

Signed-off-by: Varsha Rao <varao@redhat.com>
(cherry picked from commit bdb2241ca5a9758e8c52d47320d8b5ea0766aea9)

doc/cephadm: remove repeated section on disabling logging to stderr

Signed-off-by: Varsha Rao <varao@redhat.com>
(cherry picked from commit 9691f76d46cd2d2a0fa4f7ec776e0689c3c22413)

doc/cephadm: linking to log material

This PR rewrites a section in the Troubleshooting
chapter of the Cephadm Guide. The material that this
section discusses has been covered already in the
Cephadm Guide in the Cephadm Operations chapter.
There's no reason to repeat this information twice,
unless adding technical debt to the documentation
is our goal (which of course it is not, and the
opposite of adding technical debt to the documentation
has been the aim that has guided my work these past
six months).

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit d0a6903274a9f9335b1d4dc31504fac03943521b)

mgr/cephadm: make return message for entering maintenance mode more explicit

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 8b3493e5513b98ae4480323e2b6ac6e1ce8d257a)

mgr/cephadm: don't return hosts in offline_hosts set as schedulable

we are only checking for host status here but we
should also be checking the offline_hosts set

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit d87f1ec841f63a9d7bd52e03461b04d8c6273a94)

mgr/cephadm: fix unit test for don't touch offline hosts

We use an offline_hosts set for marking offline hosts
rather than the host status so changing this unit test
to reflect that

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 141072bc71b23d10554cb3b3414451a7eda76ce4)

mgr/cephadm: stop removal of daemons from offline hosts

This check was only looking for the status of the
host and not looking at the offline_hosts set so
it wasn't actually stopping daemons from being removed
from offline hosts

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 7c138ed52d806726f5e0a1548d958af0f95813e8)

mgr/cephadm: fix exceptions causing stuck progress indicators

Added a try block to ensure that progress of applying a service spec is updated as failed in the case of exceptions.

Fixes: https://tracker.ceph.com/issues/51961
Signed-off-by: Cory Snyder <csnyder@iland.com>
(cherry picked from commit f247a7bed24756ba8a6f02579c98813931497421)

cephadm: validate fsid during ceph-volume command

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit f0609429c05a70b6e6e7f131f56468e704be9640)

cephadm: validate fsid during shell command

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 25febfd9e081eaa275faa786102d59522ddba405)

cephadm: `ceph-volume` should raise fsid mismatch

raise an fsid mismatch error when passed differing fsids via `--fsid` and `--config`:

```
self = <tests.test_cephadm.TestCephVolume object at 0x7f1c711961f0>, cephadm_fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f1c713addc0>

    def test_fsid(self, cephadm_fs):
        cv_cmd = ['--', 'inventory', '--format', 'json']
        fsid = '00000000-0000-0000-0000-0000deadbeef'

        cmd = ['ceph-volume', '--fsid', fsid] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            cd.command_ceph_volume(ctx)
            assert ctx.fsid == fsid

        s = get_ceph_conf(fsid=fsid)
        f = cephadm_fs.create_file('ceph.conf', contents=s)

        cmd = ['ceph-volume', '--fsid', fsid, '--config', f.path] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            cd.command_ceph_volume(ctx)
            assert ctx.fsid == fsid

        cmd = ['ceph-volume', '--fsid', '10000000-0000-0000-0000-0000deadbeef', '--config', f.path] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            err = 'fsid does not match ceph.conf'
            with pytest.raises(cd.Error, match=err):
                cd.command_ceph_volume(ctx)
>               assert ctx.fsid == None
E               AssertionError: assert '10000000-0000-0000-0000-0000deadbeef' == None
E                +  where '10000000-0000-0000-0000-0000deadbeef' = <cephadm.CephadmContext object at 0x7f1c7121c1c0>.fsid
```

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit d9198d8668055fdc9e5c3c58668a6aeeda61df4a)

cephadm: add `ceph-volume` tests

add basic ceph-volume tests for `--fsid`, `--config`, and `--keyring`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 2d2bb9f96d7c34fab3e8d25c77986ab1a319fa1b)

cephadm: remove `get_parm` mock

fixture does not need to patch the `get_parm` func

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit b589ae68fa63ddb303a60232fe14200443bca0cc)

cephadm: don't use ctx.fsid for clean_cgroup

The clean_cgroup method assumes that the ctx.fsid is set while this is
true for the bootstrap command, it isn't set for adopt or deploy commands
(and maybe others).

This ends up to the adopt command to fails:

Traceback (most recent call last):
  File "/sbin/cephadm", line 8301, in <module>
    main()
  File "/sbin/cephadm", line 8289, in main
    r = ctx.func(ctx)
  File "/sbin/cephadm", line 1764, in _default_image
    return func(ctx)
  File "/sbin/cephadm", line 5091, in command_adopt
    command_adopt_ceph(ctx, daemon_type, daemon_id, fsid)
  File "/sbin/cephadm", line 5299, in command_adopt_ceph
    osd_fsid=osd_fsid)
  File "/sbin/cephadm", line 2884, in deploy_daemon_units
    clean_cgroup(ctx, unit_name)
  File "/sbin/cephadm", line 2724, in clean_cgroup
    if not ctx.fsid:
  File "/sbin/cephadm", line 155, in __getattr__
    return super().__getattribute__(name)
AttributeError: 'CephadmContext' object has no attribute 'fsid'

Since we already have the fsid value in deploy_daemon_units (which calls
clean_cgroup) then we can pass the fsid value directly.

This fixes a regression introduced by 1fee255

Fixes: https://tracker.ceph.com/issues/51902
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 3907ce7d6e091f87c3bd4437d13951ee838dc02b)

cephadm: don't fail hard on SameFileError during shutil.copy

Fixes: https://tracker.ceph.com/issues/51829
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 3909be178addb6f692d97651bfdaf9ae742497f1)

mgr/cephadm: ingress: fix typo in spec.virtual_interface_networks reference

When using virtual_inteface_networks to identify the interface to have the
virtual ip on, it referenced spec.networks instead of
spec.virtual_interface_networks.

Fixes: https://tracker.ceph.com/issues/51721
Signed-off-by: Asbjørn Sannes <asbjorn.sannes@interhost.no>
(cherry picked from commit 024c6aba01362e18ff02d88004b501663dbfdeed)

mgr/cephadm: Don't allow stopping full mgr, mon or osd services

I can't think of any case where we would want to allow this

Fixes: https://tracker.ceph.com/issues/51298
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 6402c587d006f8e179f01ec2007828bb5fbf5489)

mgr/cephadm/grafana: check if dashboard is enabled

When deploying the grafana service but the mgr dashboard isn't enabled then
dashboard set-grafana-api-ssl-verify command fails.

Closes: https://tracker.ceph.com/issues/51796
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 16bb5b8076a6df30b1e6323b406fee0ba6cc2b39)

mgr/cephadm/iscsi: simplify the dashboard check

We don't need to run an extra command (mgr module ls) to obtain the mgr
modules list since we already have this information in the mgr_map.
This workflow is already done for the monitoring stack or for configuring
the iscsi integration within the dashboard (during creation) via the
config_dashboard method.

The mgr_map is mocked in the tests with the dashboard module enabled so we
don't need _mon_command_mock_mgr_module_ls anymore.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit a6808efca4535f10c5639ee0a6a517c110da3f44)

mgr/cephadm: Fix haproxy not being recognized as a proper daemon

Turns out daemon types != service types:

    cephadm [WRN] Found unknown service type haproxy on host smithi019
    cephadm [WRN] Found unknown service type keepalived on host smithi019

leading to `self.mgr.cache.get_daemons_by_service(spec.service_name())`
not returning any daemons.

Fixes: https://tracker.ceph.com/issues/51311
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit a8f1cf2edb0ef5b48632af8da9577c8a42a6ff60)

mgr/cephadm/templates: add jinja2 lint

This adds a jinja2 lint environment in tox for testing the cephadm jinja2
templates.

This patch fixes some minor jinja2 syntax for ganesha and keepalived even if
the current templates work perfectly.

Tags should have one (and only one) space

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 9eac2fba90a3b179087455100e494d7da6b0910e)