mon/MgrStatMonitor: do not spam subscribers (mgr) with service_map

We are comparing the monmap epoch to the service_map epoch!

Fixes: https://tracker.ceph.com/issues/53479
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44161 from neha-ojha/wip-fix-upgrades

qa/suites/upgrade/octopus-x: bunch of fixes and cleanup

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #43149 from sebastian-philipp/cephadm-force-last-admin

mgr/cephadm: Add client.admin keyring when upgrading from older version

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #44109 from sebastian-philipp/doc-crush-types

doc/cephadm: host location: add link to types

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44134 from liewegas/cephadm-device-enhanced-scan

mgr/cephadm: avoid repeated calls to get_module_option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44071 from tchaikov/wip-atomic-mips64

cmake: test for 16-byte atomic support on mips also

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43540 from fengchunsong/dpdk-test

test/msgr: remove DPDK Non-runtime configure items

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43981 from lxbsz/wip-53216

qa/cephfs: correct the parameters' order

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44087 from guits/guits-fix-cv-rootfs

ceph-volume: remove --root param from nsenter cmd

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #44143 from devlikai/master

doc/mgr/diskprediction: fix a typo.

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44116 from lxbsz/caps_doc1

doc: fix the style of the cephfs capability doc

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44129 from spdfnet/patch-1

doc: fix typo in cephadm host management

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa: correct the parameters' order

The parameters' order is incorrect and missing the client_config.

Introduced-by: 242585656c6bc282f5adbd073e83bafa86b5c0d2
Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: move the optional client_config parameter to the end

Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: rename and save the client_config for kernel mount

Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #44038 from lxbsz/wip-53082

client: fix crash when iterating and deleting sessions

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43878 from jtlayton/wip-53214

qa: account for split of the kclient "metrics" debugfs file

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43850 from batrick/i53194

mds: defer messages to bootstrapping ranks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43297 from yongseokoh/test-dir-max-entries

qa: add mds_dir_max_entries workunit test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41334 from vshankar/wip-kcephfs-new-mount-syntax

mount: introduce new device mount syntax

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/*/octopus-x/stress-split-erasure-code-no-cephadm: set quincy flags

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: remove msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: test upgrades with hybrid allocator

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: rename octopus install correctly

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: remove leftovers from nautilus

pglog_hardlimit and msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: set quincy flags

not pacific

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42710 from yehudasa/wip-rgw-mgr-module

mgr/rgw: new rgw manager module

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #44140 into master

* refs/pull/44140/head:
python-common/ceph/deployment/drive_group: fix 'orch ls --format yaml'

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

python-common/rgw: fix style issues

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc/rgw: fix docs build

Workaround rgw modules conflict, as there are two separate modules named
rgw: src/pybind/rgw, src/pybind/mgr/rgw

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #44072 from ronen-fr/wip-rf-latescrub-count

qa/standalone: osd-scrub-repair.sh: fix expected "not scrubbed since"…

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #44118 from sebastian-philipp/cephadm-inventory-changed-while-iterated

mgr/cephadm: Inventory: Fix `dictionary changed size during iteration `

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44115 from rhcs-dashboard/fix-tooltip-fast-diff

mgr/dashboard: avoid tooltip if disk_usage=null and fast-diff enabled

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44083 from wangbo-yw/wangbo-yw

mgr/dashboard: add some test for controllers/pool.py

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43855 from zhangmengqianyw/zmq-unittest

mgr/dashboard:add unittest in test_osd.py

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44082 from pcuzner/fix-prometheus-timings

mgr/prometheus: Fix the per method stats exported

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mgr/diskprediction: fix a typo.

doc: remove extra comma.

This commit remove extra comma of "To disable prediction,:".

Fixes: https://tracker.ceph.com/issues/53433
Signed-off-by: devlikai <likai_lc@inspur.com>

Merge pull request #43529 from curtbruns/rgw-lua-storageclass

rgw/lua: allow read/write of StorageClass field

Merge pull request #42504 from arjune123/rgw-bug-fixes

rgw/notification: assigning the value of zonegroup to awsRegion

doc, man: document new device syntax for kclient mount

Fxies: http://tracker.ceph.com/issues/47277
Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa: use v1 mount-syntax for upgrade tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: mount kclient using new-style (v2) syntax

But, do not throw away the old style mount syntax since we would
want to continue testing it since users (scripts) might still be
using it.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: use mds_mode=prefer-crc with new-style mount syntax if possible

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: optionally do not fall-back to old-style mount syntax

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: introduce new mount syntax

Old mount device (source) have the following problems:

- mounts to the same cluster but with different fsnames
  and/or creds have identical device string which can
  confuse xfstests.

- device shown in /proc/mounts is different that what
  is used to mount.

New proposed syntax is as follows:

  # mount -t ceph cephuser@<cluster-fsid>.mycephfs2=/path

The cluster-id is optional but is always passed to the kernel
via mount() syscall (by fetching it from cluster configuration
file if available).

The mount helper tries the mounting the file system with
the new device syntax, falling back to using the old
syntax if required.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: record cluster fsid when reading ceph config file

This will be required when switching to new mount device
syntax when cluster fsid will be passed to kernel as mount
option.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: accept monitor host (`mon_host`) mount option

With new mount device syntax monitor host(s) can be passed
during mount as option parameter. This option is not mandatory
as it can be fetched from the cluster configuration file if
available.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

python-common/ceph/deployment/drive_group: fix 'orch ls --format yaml'

In f76c02a658ed1de54a380381932bb58b6ab91d41 we may leave out the placement
key when it is empty.  If we take that path, we need to tolerate specs
that are missing that key.

Fixes: f76c02a658ed1de54a380381932bb58b6ab91d41
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43760 from yaarith/release-checklists

doc/dev/release-checklist: add manual review of telemetry reports

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: avoid repeated calls to get_module_option

We already stash these as MgrModule members.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #44018 into master

* refs/pull/44018/head:
mon: fix quorum_age() regression

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #44030 into master

* refs/pull/44030/head:
mgr/cephadm: add some debug output for serve loop
ceph-volume: adjust arguments for 'ceph-volume raw activate'
ceph-volume: add raw support for db/wal for list and activate

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #44107 into master

* refs/pull/44107/head:
qa/tasks/cephadm_cases/test_cli: fix test_daemon_restart

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: add some debug output for serve loop

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: adjust arguments for 'ceph-volume raw activate'

Take a list of devices, so that we can selectively activate a raw osd
with db/wal.

Remove the argument type kludge introduced in 2c228a9a409176c0f1679f176443fd3ead219c7a
since it is no longer needed.

Note that we're making this change because (1) it allows db/wal and (2)
because there are no known users of 'raw activate'.  The only known user
is via 'ceph-volume activate' and we've fixed that caller in this commit.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: add raw support for db/wal for list and activate

Currently 'prepare' doesn't support db/wal, but we want it in list and
activate because 'ceph-volume activate ...' tries raw before lvm.

Note that I'm not sure we really want to accept --block.db and --block.wal
here at all.

Fixes: 3d7ceec684b0ac5b83fae4c397b134236fac485e
Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm_cases/test_cli: fix test_daemon_restart

We cannot schedule a daemon start if there is another daemon action
with a higher priority (including stop) scheduled.  However,
that state isn't cleared until *after* the osd goes down, the
systemctl command returns, and mgr/cephadm gets around to updating
the inventory scheduled_daemon_action state.

Semi-fix: (1) wait for the orch status to change, and then (2)
wait a few more seconds after that.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44100 from adk3798/infer-config-fix

cephadm: only infer conf from mon if fsid matches

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44101 from adk3798/agent-down-multiplier

mgr/cephadm: agent: allow agent down multiplier to be configured

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc: fix typo in cephadm host management

Merge pull request #42378 from sebastian-philipp/no-grafana-admin

mgr/cephadm: Add GrafanaSpec.initial_admin_password

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: serve.py: put _write_client_files into it's own method

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: serve.py: put _calc_client_files into it's own method

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Raise errors to properly set a cli status code

otherwise `ceph orch host rm` will return 0

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Add client.admin keyring when upgrading from older version

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm/inventory: remove unused `filter_by_label`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44011 from adk3798/repr-device

python-common: add string representation for Device and DeviceSelection classes

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44114 from orozery/librbd-memory-leaks

librbd: fix various memory leaks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43530 from myoungwon/wip-seastore-nvme-device

seastore: add nvme commands to nvme device class

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44068 from rzarzynski/wip-crimson-weakref-in-sharedlru

crimson/common: don't assume pointer-from-SharedLRU can't outlive it.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44110 from rzarzynski/wip-crimson-alienstore-syncumountread

crimson/os: fix a shutdown-related race condition in AlienStore.

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: fix the style of the cephfs capability doc

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #43481 from myoungwon/wip-dedup-tool-repair

tool: add repair command to ceph-dedup-tool

Reviewed-by: Samuel Just <sjust@redhat.com>

librbd/crypto: remove unused member from ShutDownCryptoRequest

m_crypto is not used - remove it.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockShutDownCryptoRequest

fix memory leak in TestMockShutDownCryptoRequest.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockCryptoLoadRequest

fix memory leak in TestMockCryptoLoadRequest.CryptoAlreadyLoaded

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockCryptoCryptoObjectDispatch

fix memory leak in TestMockCryptoCryptoObjectDispatch.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd/crypto: fix memory leak in openssl/DataCryptor

Re-initializing the same datacryptor, causes a memory leak of the old encryption key.
This commit fixes this issue.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd/crypto: fix memory leak in ShutDownCryptoRequest

If crypto object dispatch does not exist, a context pointer is leaked.
This commit fixes this issue.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockParentCacheObjectDispatch

fix memory leak in TestMockParentCacheObjectDispatch.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockCryptoLuksFormatRequest

fix memory leak in TestMockCryptoLuksFormatRequest.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockCryptoLuksLoadRequest

fix memory leak in TestMockCryptoLuksLoadRequest.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix bad TearDown in TestCryptoOpensslDataCryptor

Fix the TearDown function in TestCryptoOpensslDataCryptor
to call the right class parent function.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestCryptoOpensslDataCryptor

One of the tests leaks an encryption context.
This commit fixes this issue.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd/crypto: fix memory leak in when DataCryptor fails

If DataCryptor fails, either in init_context or update_context,
the encryption context is not returned, which causes a memory leak.
This commit fixes this issue.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockCryptoBlockCrypto

fix memory leak in TestMockCryptoBlockCrypto.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge PR #43997 into master

* refs/pull/43997/head:
mgr/cephadm: make logging about agent less verbose

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #44079 into master

* refs/pull/44079/head:
mgr/cephadm: skip osd_stats check if osd removal queue is empty

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #44075 into master

* refs/pull/44075/head:
mgr/cephadm: drop osdspec_affinity tracking

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #44073 into master

* refs/pull/44073/head:
pybind/mgr/mgr_module: cache mgr_ip

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: simplify HostCache.get_daemon_types

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Inventory: Fix `dictionary changed size during iteration`

Use `.copy()` for that.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #43936 into master

* refs/pull/43936/head:
qa/tasks/cephadm: pull image to all hosts in parallel
qa/tasks/cephadm: add hosts via mon remote
qa/tasks/cephadm: use shortname for remote directory
qa/tasks/cephadm: deploy no more than 5 mons in roleless mode
qa/tasks/radosbench: default clients to all clients (not client.0)
qa/tasks/ceph_manager: parallelize flush_pg_stats()
qa/suites/big: remove thrasher
qa/suites/big: update for cephadm

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #44080 into master

* refs/pull/44080/head:
mgr/cephadm: record when finished with scheduled daemon action

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: grafana.ini: Set `cookie_secure = true`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Add GrafanaSpec.initial_admin_password

By default, we're not creating any admin accout for Grafana now,
but we're adding an option to set the grafana password manually using:

```yaml
service_type: grafana
spec:
  initial_admin_password: mypassword
```

Users can then easily log into Grafana with the given password.

Fixes: https://tracker.ceph.com/issues/48291
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

python-common: Reparent AlertManagerSpec to MonitoringSpec

And remove duplicated members

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

python-common: Move AlertManagerSpec below MonitoringSpec

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

python-common: test_yaml(): add a few tests

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

python-common: prettify `yaml.dump(MonitoringSpec())`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>