pybind/mgr/devicehealth/module.py: use NotifyType enum

master/quincy use a precreated .mgr pool and does not
need this commit

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr: only queue notify events that modules ask for

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit ee4e3ecd6e0754854c00f3dfa0eaaa17bbf3603d)

pybind/mgr: annotate which events modules consume

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 95ca3a444505b5bd38f4961ed63760ac45abcf64)

 Conflicts:
src/pybind/mgr/mds_autoscaler/module.py - trivial resolution

pybind/mgr: introduce NotifyType enum

Note that we don't annotate the dashboard NotificationQueue because it is
used internally by the dashboard with other events.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 1ac480df45058e103a2f4954950b8a4edb23610a)

 Conflicts:
src/pybind/mgr/cephadm/module.py - trivial resolution
src/pybind/mgr/dashboard/module.py - trivial resolution
src/pybind/mgr/localpool/module.py - trivial resolution
src/pybind/mgr/mds_autoscaler/module.py - trivial resolution

mgr: stop issuing events that no modules consume

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit d3c8f171e9aaa9cbf970b6dadda5837ca9b030c7)

mgr: limit changes to pg_num

We need to avoid making drastic changes to pg_num that outpace pgp_num or
else we will may hit the per-osd pg limits.

Fixes: https://tracker.ceph.com/issues/53442
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 3b2a11249aff6ee608efc95212d6723df180cd07)

 Conflicts:
src/common/options/mgr.yaml.in - old way of specifying config settings

mgr/progress: avoid inefficient dump of all pg stats

We only use a handful of fields, and the pg dump includes a gazillion
fields that we waste CPU copying to python-land.  This tends to lead to
long ClusterState::lock hold times, leading to long ms_dispatch delays
and generally gumming up the works.

Instead, create a new "pg_progress" item that dumps only the fields that
mgr/progress needs.

Fixes: https://tracker.ceph.com/issues/53475
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit f5973ccef415571c72560a04968592e8a6daf93a)

pybind/mgr/mgr_module: cache mgr_ip

This does not change for the lifetime of an active mgr module.  No need to
keep calling back into Mgr to re-fetch it.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 994832e8e50c3566e21928f6be61e45a4411e556)

 Conflicts:
src/pybind/mgr/mgr_module.py - self._db_lock = threading.Lock() DNE in pacific

Merge pull request #45028 from rhcs-dashboard/wip-54269-pacific

pacific: mgr/dashboard: Directories Menu Can't Use on Ceph File System Dashboard

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #45052 from rhcs-dashboard/wip-54295-pacific

pacific: mgr/dashboard: change privacy protocol field from required to optional

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44747 from cybozu/wip-53070-pacific

pacific: rgw: fix lock scope in ObjectCache::get()

Merge pull request #44750 from rhcs-dashboard/wip-53987-pacific

pacific: mgr: TTL Cache in mgr module

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: add validation for snmp v3 engine id

Fixes: https://tracker.ceph.com/issues/54270
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 2866db1eac7d726201f5bb34abdb32981c783f0e)

mgr/dashboard: change privacy protocol field from required to optional

Fixes: https://tracker.ceph.com/issues/54270
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Privacy protocol field shouldn't be a required field.

(cherry picked from commit 2d8f2b8195a0f0c7a21d4ec5061b1b51a3aade2c)

Merge pull request #44543 from cfsnyder/wip-53659-pacific

pacific: mon: prevent new sessions during shutdown

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44756 from votdev/wip-53929-pacific

pacific: mgr/dashboard: Notification banners at the top of the UI have fixed height

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #44938 from rhcs-dashboard/wip-54091-pacific

pacific: mgr/dashboard: cephadm e2e job improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44935 from rhcs-dashboard/pacific-test-applitools

pacific: mgr/dashboard: set appropriate baseline branch for applitools

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44876 from aaSharma14/wip-53930-pacific

pacific: mgr/dashboard: Improve notifications for osd nearfull, full

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44789 from guits/wip-54021-pacific

pacific: ceph-volume: improve mpath devices support

Merge pull request #44884 from guits/wip-54127-pacific

pacific: ceph-volume: fix error 'KeyError' with inventory

mgr/dashboard:Directories Menu Can't Use on Ceph File System Dashboard

Added exception handling to opendir() in cephfs.py for directories with no execute permission.

Fixes: https://tracker.ceph.com/issues/51611
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>
(cherry picked from commit ea1af5438d380eb2160de635ffc7b08a69baf04c)

Merge pull request #44767 from guits/wip-54007-pacific

pacific: ceph-volume: fix tags dict output in `lvm list`

ceph-volume: fix error 'KeyError' with inventory

The tag ceph.cluster_name is always set at the end.
The only way it could be absent was if the osd prepare
has been interrupted between [1] and [2].

[1] https://github.com/ceph/ceph/blob/v14.2.11/src/ceph-volume/ceph_volume/devices/lvm/strategies/bluestore.py#L355-L387
[2] https://github.com/ceph/ceph/blob/v14.2.11/src/ceph-volume/ceph_volume/devices/lvm/prepare.py

Although the code received tremendous changes meantime
and this error shouldn't show up again, we need to handle
the case where this tag wouldn't have been set.

Fixes: https://tracker.ceph.com/issues/44356
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 691660c42eaa568a754670e093c512aa041d1479)

ceph-volume: fix typo in tests

This fixes 2 typo in ceph-volume tests.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit b07bd3e0e17021e0cf9773f916fad954f12254ed)

doc/ceph-volume: fix a typo

This fixes a typo in ceph-volume documentation.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5d0a3cee5d7021dafd1e166e17946689b4bb90b7)

ceph-volume: add a test `test_mpath_device_is_device`

This test checks that Device.is_device() returns True for a mpath device.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 0280ff6df09bc26107bc97446e9d5c18fbc582e9)

ceph-volume: improve mpath devices support

ee8887f4c0ff4f91117f31b621b95c8d08019130 was intended for adding
mpath devices support in ceph-volume but it has missed the lvm batch scenario.
This also fixes the zapping of mpath devices prepared with `ceph-volume raw`

Fixes: https://tracker.ceph.com/issues/52908
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 601ff7ed0a3ba5172b6bd886ca8ba2bd4d9e655a)

Merge pull request #44973 from guits/wip-54244-pacific

pacific: ceph-volume: honour osd_dmcrypt_key_size option

Merge pull request #44977 from rhcs-dashboard/wip-54240-pacific

pacific: mgr/dashboard: support snmp-gateway service creation from UI

mgr/dashboard: add snmp-gateway service e2e tests

Fixes: https://tracker.ceph.com/issues/54034
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 76dcf6a881f343bb3d93259701e57ccb572f94e8)

mgr/dashboard: add snmp destination validation

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 81c93a21ff64e6aeeeba2c88db4f61b13352f565)

mgr/dashboard: support snmp-gateway service creation from UI

Fixes: https://tracker.ceph.com/issues/54034
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit ad6fcfc05625b3fd8a088b8a2b5c3d5fbbf2c53a)

ceph-volume/activate: load the config from lv tag

When `ceph-volume lvm trigger` is called with an OSD where the tag
`ceph.cluster_name` is not 'ceph', it fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5ac1ec65cb2a582b2ae550202cc9911f993943f2)

ceph-volume/tests: use centos/stream8 images

Since recent move from CentOS 8 to CentOS Stream 8, let's do the same here.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 2b793952bbac7973b97d245c282165daadeabb51)

ceph-volume/tests: add tests in util/encryption.py

this adds some unit tests in order to cover `luks_format()` and `luks_open()`
in `util/encryption.py`.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit db48850745f218e08cf53ae2d8edf3428f2b4010)

ceph-volume: honour osd_dmcrypt_key_size option

ceph-volume doesn't honour osd_dmcrypt_key_size.
It means the default size is always applied.

It also changes the default value in `get_key_size_from_conf()`

From cryptsetup manpage:

> For XTS mode you can optionally set a key size of 512 bits with the -s option.

Using more than 512bits will end up with the following error message:

```
Key size in XTS mode must be 256 or 512 bits.
```

Fixes: https://tracker.ceph.com/issues/54006
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 47c33179f9a15ae95cc1579a421be89378602656)

mgr/dashboard: change the readFile to readFileSync

Apparently the readFile i added in #44934 is async and that's not what
we want. so changing it to the synchronous call that is readFileSync

Fixes: https://tracker.ceph.com/issues/54190
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit cbfdd551d9c1e67c2757056ac1119c058f4aa704)

Merge pull request #44889 from ceph/pacific-stream8

qa: Default to CentOS 8 Stream

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #44903 from neha-ojha/wip-44868-pacific

pacific: qa/distros/podman: remove centos_8.2 and centos_8.3

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Adam King adking@redhat.com

Merge pull request #44944 from ljflores/wip-54204-pacific

pacific: monitoring: mention PyYAML only once in requirements

mgr/dashboard: set appropriate baseline branch for applitools

All the dashboard PRs are checked against a baseline branch called
'default' in the visual regresstion testing. This will cause issues when
testing PRs in different branches. For eg: currently our master and
pacific has to save two different screenshots since the two of them
differ slightly.

Disabling the applitools logs as well because its too 'noisy'

Fixes: https://tracker.ceph.com/issues/54190
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 40c902ac59b758a314f6a123d71cb59342523dac)

monitoring: mention PyYAML only once in requirements

Following error occurs while running "sudo install-deps.sh" -
ERROR: Double requirement given: PyYAML==6.0 (from -r requirements-lint.txt (line 5)) (already in pyyaml (from -r requirements-alerts.txt (line 1)), name='PyYAML')

PyYAML is mentioned twice as a requirement. It is mentioned once in both
the following files -
monitoring/ceph-mixin/requirements-lint.txt
monitoring/ceph-mixin/requirements-alerts.txt

These requirements were added in commits
44d3e4c264506154373ffaeb13d6c924c580e6b5 and
4750ac0d7766a8a089adf073415af0ac0d3f81d9.

Fixes: https://tracker.ceph.com/issues/54185
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit a6f5efb620c429f81ea13992c2f77b4ca55458bc)

doc: update dashboard kcli test env documentation

Fixes: https://tracker.ceph.com/issues/54105
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 8feb2b8fe03f0c06a0ab09328ca8df0dfe8c0de9)

mgr/dashboard: fix for cephadm e2e failing because of rgw commands getting stuck

Delaying the rgw service creation in the tests until the cluster is
healthy

also changing the node_ip_offset to 110 because in the jenkins I saw

Fixes: https://tracker.ceph.com/issues/54030
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 347fb2e8fe26020a4693d3bbd94ca007c7e3535a)

mgr/dashboard: cephadm e2e job: display info on error & other improvements

- Fix: ensure that on_error trap is called (display more info on error).
- Set static IPs to VMs.
- Remove domain in cluster definition to avoid side effects of potential dns misconfiguration.
- Minor improvements.

Fixes: https://tracker.ceph.com/issues/53991
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 39af61efb24dac6f41ba0752944882d35ad287db)

mgr/dashboard: navigation page e2e fix

Looks like the newly added relative position for the sidebar is
causing cypress to verify that the sidebar is hidden from the user view.

Fixes: https://tracker.ceph.com/issues/53960
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 6b2be10dbba639cea4d40c39fb63900b7bc0ccd0)

qa: remove broken symlinks, add centos_8.stream wherever needed

This commit is not required in master/quincy

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/distros/podman: remove centos_8.2 and centos_8.3

Related to https://tracker.ceph.com/issues/54087

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 8f1d1b7c55a6a5715e12e9586be0951fad15cf49)

Merge pull request #43897 from k0ste/wip-53234-pacific

pacific: mgr/prometheus: Make prometheus standby behaviour configurable

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: k0ste <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: rsommer <NOT@FOUND>

Merge pull request #44775 from p-se/wip-53882-pacific

pacific: mgr/dashboard: fix Grafana OSD/host panels

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: p-se <NOT@FOUND>

qa: Default to CentOS 8 Stream

Signed-off-by: David Galloway <dgallowa@redhat.com>

mgr/dashboard: Improve notifications for osd nearfull, full

This PR adds some visual hints for osds that are near full or full

Fixes: https://tracker.ceph.com/issues/53334
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit f771cd492cd06da13f26e5f7ffe41b2d3c43f950)

Merge pull request #44672 from kamoltat/wip-ksirivad-pacific-backport-44553

pacific: pybind/mgr/progress: enforced try and except on accessing event dictionary
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44840 from mchangir/pacific-avoid-mon-sanity-assertion-on-startup

qa: skip sanity check during upgrade

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #44259 from sseshasa/wip-53551-pacific

pacific: osd/OSDMap: Add health warning if 'require-osd-release' != current release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

PendingReleaseNotes: Release note 'require-osd-release' health warning

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 34f18fa45f3e98de4f35959bdeb1d11730f3f291)

Conflicts:
    PendingReleaseNotes
- Add the release note under the correct release heading.

qa: set pacific require-osd-release to avoid health warning

Fixes: https://tracker.ceph.com/issues/53615
Fixes: bd815bd9d6ecdecaab3d2dd9e0f5a18aa795d749
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit bc2eaba8c6616c3469ac85d36ee21f3e9765cf42)

Conflicts:
- Changed release name in commit message from 'quincy' to 'pacific'.

  ../fs/upgrade/featureful_client/old_client/tasks/2-upgrade.yaml
  ../fs/upgrade/featureful_client/upgraded_client/tasks/2-upgrade.yaml
  ../fs/upgrade/nofs/tasks/1-upgrade.yaml
- Changed release name from 'quincy' to 'pacific' when setting the
  'require-osd-release' flag in the above files.

  ../fs/upgrade/volumes/import-legacy/tasks/2-upgrade.yaml
- Changed release name from 'octopus' to 'pacific' when setting the
  'require-osd-release' flag in the above file.

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: remove msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 6ad7a8a597e6314cf9310f9f7c2f01ef82bc8fa3)

qa: test upgrades with hybrid allocator

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit df67040a4c34593adb4585086671543233d90f5a)

qa: rename octopus install correctly

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 3b15a044550903d4074fc421a2ef1f24fc7e4023)

qa: remove leftovers from nautilus

pglog_hardlimit and msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit ed4bb05bd945c5d30cb70b88b1f8db0eb64a6ab1)

qa/suites/upgrade: Fix/Modify upgrade tests to work with 'pacific' release.

This commit is not a cherry-pick and fixes the following issues unique to
the pacific release:
1. Fixes the nautilus-x and octopus-x upgrade tests to work with the
   pacific release by updating the post upgrade step to set the
   'require-osd-release' flag to 'pacific'. This is done by using
   '.qa/releases/pacific.yaml' for all the tests.
2. Fixed an issue in 'upgrade-mon-osd-mds.yaml' under both the
   nautilus-x/parallel, octopus-x/parallel-no-cephadm tests. The
   'wait-for-healthy' check should not be performed after all the osds
   are upgraded since the 'require-osd-release' warning comes into
   effect. This check is delayed until after the 'require-osd-release'
   flag is set to 'pacific'.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

osd/OSDMap: Add health warning if 'require-osd-release' != current release

After all OSDs are upgraded to a new release, generate a health warning if
the 'require-osd-release' flag doesn't match the the new release version.
This will result in the cluster showing a warning in the health state until
the flag is set properly.

Fixes: https://tracker.ceph.com/issues/51984
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit bd815bd9d6ecdecaab3d2dd9e0f5a18aa795d749)

Conflicts:
    src/osd/OSDMap.cc
- Removed checks for non-existent ceph_release_t 'quincy' flag from
  OSDMap::pending_require_osd_release().

Merge pull request #44387 from trociny/wip-53702-pacific

pacific: qa/tasks: improve backfill_toofull test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44325 from k0ste/wip-53621-pacific

pacific: mgr/devicehealth: fix missing timezone from time delta calculation

Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #44205 from k0ste/wip-53488-pacific

pacific: mgr/prometheus: define module options for standby

Reviewed-by: Adam King adking@redhat.com

Merge pull request #44175 from cfsnyder/wip-51172-pacific

pacific: common/PriorityCache: low perf counters priorities for submodules.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #44181 from myoungwon/pacific-50192

pacific: osd: recover unreadable snapshot before reading ref. count info

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43955 from cfsnyder/wip-53201-pacific

pacific: osd: fix 'ceph osd stop <osd.nnn>' doesn't take effect

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44212 from k0ste/wip-53494-pacific

pacific: mgr: fix locking for MetadataUpdate::finish

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44202 from myoungwon/pacific-53486

pacific: test: increase retry duration when calculating manifest ref. count

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44173 from cfsnyder/wip-51150-pacific

pacific: osd: set r only if succeed in FillInVerifyExtent

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #44096 from cfsnyder/wip-53388-pacific

pacific: osd/OSDMap.cc: clean up pg_temp for nonexistent pgs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #43882 from ifed01/wip-ifed-fix-53011-pac

pacific: os/bluestore: use proper prefix when removing undecodable Share Blob.

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/mgr: Add cli api documentation

Signed-off-by: Waad AlKhoury <walkhour@redhat.com>

mgr/cli Redo  cli api mgr module.

Signed-off-by: Waad AlKhoury <walkhour@redhat.com>
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

qa: skip sanity check during upgrade

Fixes: https://tracker.ceph.com/issues/54064
Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44727 from cfsnyder/wip-51825-pacific

pacific: qa/run-tox-mgr-dashboard: Do not write to /tmp/test_sanitize_password…

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Cory Snyder <csnyder@iland.com>
Reviewed-by: kevinzs2048 <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44540 from kamoltat/wip-ksirivad-backport-pacific-43716

pacific: mgr/autoscaler: Introduce noautoscale flag

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #44660 from sebastian-philipp/pacific-backport-44647

pacific: doc/cephadm: remove duplicate deployment scenario section

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44636 from sebastian-philipp/pacific-backport-44510

pacific: doc/cephadm: improve the development doc a bit

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44584 from vumrao/wip-vumrao-53876

pacific: osd/PeeringState: separate history's pruub from pg's

Reviewed-by: Neha Ojha <nojha@redhat.com>

docs: Added noautoscale to docs + release notes

Updated the docs in
https://docs.ceph.com/en/latest/rados/operations/placement-groups/
and updated the release notes to reflect noautoscale flag.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 9baed0394e03de41f1921693bb33badd1922fa97)

Conflicts:
        PendingReleaseNotes - trivial fix

Merge pull request #44513 from batrick/i53714

pacific: mds: fails to reintegrate strays if destdn's directory is full (ENOSPC)

Reviewed-by: Milind Changire <mchangir@redhat.com>

qa: Added workunit test for noautoscale flag

set and unset the noautoscale flag,
evaluate if the results are what
we expected. As well as, evaluate
if the flag is correct when we
create new pools.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit bb42c71e7e059be2cc4d1d4408e475b15b1c6340)

Conflicts:
        test-noautoscale-flag.yaml
- modified pre-mgr-command to not create
  device health monitor

pybind/mgr/autoscaler: Introduce noautoscale flag

`noautoscale` flag is a feature where the
user can choose to flip the switch between
turning autoscale `on` and `off` for all
pools with a single command.

`osd pool set noautoscale` will turn all
autoscale mode`off` for all pools.

`osd pool unset noautoscale` will turn all
autoscale mode `on` for all pools.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit be17f041bab90d8f93c3e52df74cdf6c28b44ef2)

Conflicts:
src/pybind/mgr/pg_autoscaler/module.py - trivial fix

Merge pull request #44642 from vshankar/wip-53458

pacific: qa: wait for purge queue operations to finish

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44639 from vshankar/wip-53912

pacific: qa: adjust for MDSs to get deployed before verifying their availability

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44623 from lxbsz/wip-53908

pacific: mds: remove the duplicated or incorrect respond

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44622 from lxbsz/wip-53860

pacific: mds: dump tree '/' when the path is empty

Reviewed-by: Kotresh HR khiremat@redhat.com
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44621 from lxbsz/wip-53861

pacific: qa: do not use any time related suffix for *_op_timeouts

Reviewed-by: Kotresh HR khiremat@redhat.com
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44620 from lxbsz/wip-53864

pacific: mds: directly return just after responding the link request

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR khiremat@redhat.com

Merge pull request #44516 from nmshelke/wip-53777-pacific

pacific: mgr/stats: exception handling for ceph fs perf stats command

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44514 from batrick/i53736

pacific: mds: recursive scrub does not trigger stray reintegration

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #44512 from MrFreezeex/wip-52631-pacific

pacific: mds: add mds_dir_max_entries config option

Reviewed-by: Milind Changire <mchangir@redhat.com>

monitoring: Add unit tests for OSD panels in ceph-cluster dashboard

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 7d7488018ea30dc61174bafcad01bb3eac8aa9bb)

monitoring: fix display ceph_osd_in in Grafana panel

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 4a6b2c1dfbbe7182beaf510c4a7297a79c6e2524)

mgr/prometheus: Fix regression with OSD/host details/overview dashboards

Fix issues with PromQL expressions and vector matching with the
`ceph_disk_occupation` metric.

As it turns out, `ceph_disk_occupation` cannot simply be used as
expected, as there seem to be some edge cases for users that have
several OSDs on a single disk.  This leads to issues which cannot be
approached by PromQL alone (many-to-many PromQL erros).  The data we
have expected is simply different in some rare cases.

I have not found a sole PromQL solution to this issue. What we basically
need is the following.

1. Match on labels `host` and `instance` to get one or more OSD names
   from a metadata metric (`ceph_disk_occupation`) to let a user know
   about which OSDs belong to which disk.

2. Match on labels `ceph_daemon` of the `ceph_disk_occupation` metric,
   in which case the value of `ceph_daemon` must not refer to more than
   a single OSD. The exact opposite to requirement 1.

As both operations are currently performed on a single metric, and there
is no way to satisfy both requirements on a single metric, the intention
of this commit is to extend the metric by providing a similar metric
that satisfies one of the requirements. This enables the queries to
differentiate between a vector matching operation to show a string to
the user (where `ceph_daemon` could possibly be `osd.1` or
`osd.1+osd.2`) and to match a vector by having a single `ceph_daemon` in
the condition for the matching.

Although the `ceph_daemon` label is used on a variety of daemons, only
OSDs seem to be affected by this issue (only if more than one OSD is run
on a single disk).  This means that only the `ceph_disk_occupation`
metadata metric seems to need to be extended and provided as two
metrics.

`ceph_disk_occupation` is supposed to be used for matching the
`ceph_daemon` label value.

    foo * on(ceph_daemon) group_left ceph_disk_occupation

`ceph_disk_occupation_human` is supposed to be used for anything where
the resulting data is displayed to be consumed by humans (graphs, alert
messages, etc).

    foo * on(device,instance)
    group_left(ceph_daemon) ceph_disk_occupation_human

Fixes: https://tracker.ceph.com/issues/52974
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 18d3a71618a5e3bc3cbd0bce017fb7b9c18c2ca0)

mgr/prometheus: Refactoring: Introduce type aliases

Fixes: https://tracker.ceph.com/issues/52974
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 154d3525b19135a929851c0b027da19abda20ebe)