osd/OSDMap: Add health warning if 'require-osd-release' != current release

After all OSDs are upgraded to a new release, generate a health warning if
the 'require-osd-release' flag doesn't match the the new release version.
This will result in the cluster showing a warning in the health state until
the flag is set properly.

Fixes: https://tracker.ceph.com/issues/51984
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #44103 from majianpeng/librbd-pwl-flush-by-blockgurad

librbd/cache/pwl: Using BlockGuard control overlap ops order when flu…

Reviewed-by: Mykola Golub mgolub@suse.com
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #44144 from majianpeng/librbd-fix-discard-granularity

librbd: fix discard granularity for pwl cache

Reviewed-by: mgolub@suse.com
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43848 from dvanders/doc_bench

doc: add disk benchmarking and cache tuning recommendations

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #43691 from curtbruns/use_optimal_for_min_alloc

os/bluestore: Set min_alloc_size to optimal io size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43578 from ronen-fr/wip-rf-log-options

common: hide internal logger configuration strings from clients

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #43493 from myoungwon/wip-52872

test: increase retry duration when calculating manifest ref. count

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43944 from aclamk/wip-aclamk-fix-crush-location-hook-50659

crush: Fix segfault in update_from_hook

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #44194 from ceph/wip-yuriw-crontab-master

qa/tests: switch all gibba machines to smithi

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #44178 from Huber-ming/admin_lc_fix

radosgw-admin: fix error message of OPT::LC_RESHARD_FIX

qa/tests: switch all gibba machines to smithi

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #43336 from ifed01/wip-fix-bluefs-volumes-ops

qa/osd-bluefs-volume-ops: fix bluefs volumes ops test case

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge PR #44035 into master

* refs/pull/44035/head:
mgr/cephadm: less log noise when config checks fail

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44089 from benhanokh/ncb_fsck_fix

os/bluestore: bug-fix for NCB-FSCK

doc: add disk benchmarking and cache recommendations

Advise operators on how to benchmark devices for BlueStore, and how to
tune the volatile write cache for optimal OSD performance.

Fixes: https://tracker.ceph.com/issues/53161
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #44063 from vshankar/tr-52487

qa: wait for purge queue operations to finish

Reviewed-by: Venky Shankar <vshankar@redhat.com>

radosgw-admin: fix error message of OPT::LC_RESHARD_FIX

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge PR #44125 into master

* refs/pull/44125/head:
qa/suites/rados/thrash-old-clients: use better-support cephadm distro/podman

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: less log noise when config checks fail

We are already raising health alerts--there is no need to spam the log
every few seconds when these checks are evaluated.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44161 from neha-ojha/wip-fix-upgrades

qa/suites/upgrade/octopus-x: bunch of fixes and cleanup

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #43149 from sebastian-philipp/cephadm-force-last-admin

mgr/cephadm: Add client.admin keyring when upgrading from older version

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #44109 from sebastian-philipp/doc-crush-types

doc/cephadm: host location: add link to types

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44134 from liewegas/cephadm-device-enhanced-scan

mgr/cephadm: avoid repeated calls to get_module_option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44071 from tchaikov/wip-atomic-mips64

cmake: test for 16-byte atomic support on mips also

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43540 from fengchunsong/dpdk-test

test/msgr: remove DPDK Non-runtime configure items

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43981 from lxbsz/wip-53216

qa/cephfs: correct the parameters' order

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44087 from guits/guits-fix-cv-rootfs

ceph-volume: remove --root param from nsenter cmd

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #44143 from devlikai/master

doc/mgr/diskprediction: fix a typo.

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44116 from lxbsz/caps_doc1

doc: fix the style of the cephfs capability doc

Reviewed-by: Venky Shankar <vshankar@redhat.com>

librbd/cache/pwl: it should in apply_metadata set discard_granularity for pwl cache.

Function apply_meta can overwrite discard_granularity_bytes
based on option.

Fixes:https://tracker.ceph.com/issues/53434

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #44129 from spdfnet/patch-1

doc: fix typo in cephadm host management

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa: correct the parameters' order

The parameters' order is incorrect and missing the client_config.

Introduced-by: 242585656c6bc282f5adbd073e83bafa86b5c0d2
Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: move the optional client_config parameter to the end

Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: rename and save the client_config for kernel mount

Fixes: https://tracker.ceph.com/issues/53216
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: wait for purge queue operations to finish

TestFragmentation.test_deep_split relies on `num_strays`
to reach zero expecting that the purge threads would
have deleted the directory entries. However, checking
`num_strays` cannot be relied on since PurqeQueue merely
journals the purge item (see PurgeQueue::push) followed
by the StrayManager marking the stray as removed thereby
accounting `num_strays`.

So, add an additional condition to check if the purge
threads have finished processing items.

Fixes: http://tracker.ceph.com/issues/52487
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44038 from lxbsz/wip-53082

client: fix crash when iterating and deleting sessions

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43878 from jtlayton/wip-53214

qa: account for split of the kclient "metrics" debugfs file

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43850 from batrick/i53194

mds: defer messages to bootstrapping ranks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43297 from yongseokoh/test-dir-max-entries

qa: add mds_dir_max_entries workunit test case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41334 from vshankar/wip-kcephfs-new-mount-syntax

mount: introduce new device mount syntax

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/*/octopus-x/stress-split-erasure-code-no-cephadm: set quincy flags

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: remove msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: test upgrades with hybrid allocator

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: rename octopus install correctly

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: remove leftovers from nautilus

pglog_hardlimit and msgr2

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/octopus-x/stress-split-no-cephadm: set quincy flags

not pacific

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42710 from yehudasa/wip-rgw-mgr-module

mgr/rgw: new rgw manager module

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #44140 into master

* refs/pull/44140/head:
python-common/ceph/deployment/drive_group: fix 'orch ls --format yaml'

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

python-common/rgw: fix style issues

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc/rgw: fix docs build

Workaround rgw modules conflict, as there are two separate modules named
rgw: src/pybind/rgw, src/pybind/mgr/rgw

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #44072 from ronen-fr/wip-rf-latescrub-count

qa/standalone: osd-scrub-repair.sh: fix expected "not scrubbed since"…

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #44118 from sebastian-philipp/cephadm-inventory-changed-while-iterated

mgr/cephadm: Inventory: Fix `dictionary changed size during iteration `

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/thrash-old-clients: use better-support cephadm distro/podman

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44115 from rhcs-dashboard/fix-tooltip-fast-diff

mgr/dashboard: avoid tooltip if disk_usage=null and fast-diff enabled

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44083 from wangbo-yw/wangbo-yw

mgr/dashboard: add some test for controllers/pool.py

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43855 from zhangmengqianyw/zmq-unittest

mgr/dashboard:add unittest in test_osd.py

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44082 from pcuzner/fix-prometheus-timings

mgr/prometheus: Fix the per method stats exported

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mgr/diskprediction: fix a typo.

doc: remove extra comma.

This commit remove extra comma of "To disable prediction,:".

Fixes: https://tracker.ceph.com/issues/53433
Signed-off-by: devlikai <likai_lc@inspur.com>

Merge pull request #43529 from curtbruns/rgw-lua-storageclass

rgw/lua: allow read/write of StorageClass field

Merge pull request #42504 from arjune123/rgw-bug-fixes

rgw/notification: assigning the value of zonegroup to awsRegion

doc, man: document new device syntax for kclient mount

Fxies: http://tracker.ceph.com/issues/47277
Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa: use v1 mount-syntax for upgrade tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: mount kclient using new-style (v2) syntax

But, do not throw away the old style mount syntax since we would
want to continue testing it since users (scripts) might still be
using it.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: use mds_mode=prefer-crc with new-style mount syntax if possible

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: optionally do not fall-back to old-style mount syntax

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: introduce new mount syntax

Old mount device (source) have the following problems:

- mounts to the same cluster but with different fsnames
  and/or creds have identical device string which can
  confuse xfstests.

- device shown in /proc/mounts is different that what
  is used to mount.

New proposed syntax is as follows:

  # mount -t ceph cephuser@<cluster-fsid>.mycephfs2=/path

The cluster-id is optional but is always passed to the kernel
via mount() syscall (by fetching it from cluster configuration
file if available).

The mount helper tries the mounting the file system with
the new device syntax, falling back to using the old
syntax if required.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: record cluster fsid when reading ceph config file

This will be required when switching to new mount device
syntax when cluster fsid will be passed to kernel as mount
option.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mount: accept monitor host (`mon_host`) mount option

With new mount device syntax monitor host(s) can be passed
during mount as option parameter. This option is not mandatory
as it can be fetched from the cluster configuration file if
available.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

python-common/ceph/deployment/drive_group: fix 'orch ls --format yaml'

In f76c02a658ed1de54a380381932bb58b6ab91d41 we may leave out the placement
key when it is empty.  If we take that path, we need to tolerate specs
that are missing that key.

Fixes: f76c02a658ed1de54a380381932bb58b6ab91d41
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43760 from yaarith/release-checklists

doc/dev/release-checklist: add manual review of telemetry reports

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: avoid repeated calls to get_module_option

We already stash these as MgrModule members.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #44018 into master

* refs/pull/44018/head:
mon: fix quorum_age() regression

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #44030 into master

* refs/pull/44030/head:
mgr/cephadm: add some debug output for serve loop
ceph-volume: adjust arguments for 'ceph-volume raw activate'
ceph-volume: add raw support for db/wal for list and activate

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #44107 into master

* refs/pull/44107/head:
qa/tasks/cephadm_cases/test_cli: fix test_daemon_restart

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: add some debug output for serve loop

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: adjust arguments for 'ceph-volume raw activate'

Take a list of devices, so that we can selectively activate a raw osd
with db/wal.

Remove the argument type kludge introduced in 2c228a9a409176c0f1679f176443fd3ead219c7a
since it is no longer needed.

Note that we're making this change because (1) it allows db/wal and (2)
because there are no known users of 'raw activate'.  The only known user
is via 'ceph-volume activate' and we've fixed that caller in this commit.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: add raw support for db/wal for list and activate

Currently 'prepare' doesn't support db/wal, but we want it in list and
activate because 'ceph-volume activate ...' tries raw before lvm.

Note that I'm not sure we really want to accept --block.db and --block.wal
here at all.

Fixes: 3d7ceec684b0ac5b83fae4c397b134236fac485e
Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm_cases/test_cli: fix test_daemon_restart

We cannot schedule a daemon start if there is another daemon action
with a higher priority (including stop) scheduled.  However,
that state isn't cleared until *after* the osd goes down, the
systemctl command returns, and mgr/cephadm gets around to updating
the inventory scheduled_daemon_action state.

Semi-fix: (1) wait for the orch status to change, and then (2)
wait a few more seconds after that.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #44100 from adk3798/infer-config-fix

cephadm: only infer conf from mon if fsid matches

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44101 from adk3798/agent-down-multiplier

mgr/cephadm: agent: allow agent down multiplier to be configured

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc: fix typo in cephadm host management

Merge pull request #42378 from sebastian-philipp/no-grafana-admin

mgr/cephadm: Add GrafanaSpec.initial_admin_password

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: serve.py: put _write_client_files into it's own method

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: serve.py: put _calc_client_files into it's own method

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Raise errors to properly set a cli status code

otherwise `ceph orch host rm` will return 0

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Add client.admin keyring when upgrading from older version

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm/inventory: remove unused `filter_by_label`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44011 from adk3798/repr-device

python-common: add string representation for Device and DeviceSelection classes

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44114 from orozery/librbd-memory-leaks

librbd: fix various memory leaks

Reviewed-by: Mykola Golub <mgolub@suse.com>

librbd/cache/pwl: revert patch 76f4d29d92be3f9f45767cb1ac6cc50.

In fact, After BlockGuard to fix flush reorder issue, patch
76f4d29d9 don't need. So revert.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd/cache/pwl: Using BlockGuard control overlap ops  order when flush to osd.

In process of tests, we met some inconsistent-data problem. Test case
mainly use write,then discard to detect data consistent.
W/o pwl, write/discard are synchronous ops. After write, data already
located into osd. But w/ pwl, we use asynchronous api to send ops to
osd.
Although we mare sure send order. But send-order don't makre sure
complete order. This mean pwl keep order of write/discard. But it
don't keep the same semantics which use synchronous api. W/ pwl, it make
synchronous to asynchronous. For normal ops, it's not problem. But if
connected-commands w/ overlap, it make data inconsistent.
So we use BlockGuard to solve this issue.

Fixes: https://tracker.ceph.com/issues/49876
Fixes: https://tracker.ceph.com/issues/53108
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #43530 from myoungwon/wip-seastore-nvme-device

seastore: add nvme commands to nvme device class

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44068 from rzarzynski/wip-crimson-weakref-in-sharedlru

crimson/common: don't assume pointer-from-SharedLRU can't outlive it.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44110 from rzarzynski/wip-crimson-alienstore-syncumountread

crimson/os: fix a shutdown-related race condition in AlienStore.

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: fix the style of the cephfs capability doc

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #43481 from myoungwon/wip-dedup-tool-repair

tool: add repair command to ceph-dedup-tool

Reviewed-by: Samuel Just <sjust@redhat.com>

librbd/crypto: remove unused member from ShutDownCryptoRequest

m_crypto is not used - remove it.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockShutDownCryptoRequest

fix memory leak in TestMockShutDownCryptoRequest.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockCryptoLoadRequest

fix memory leak in TestMockCryptoLoadRequest.CryptoAlreadyLoaded

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: fix memory leak in TestMockCryptoCryptoObjectDispatch

fix memory leak in TestMockCryptoCryptoObjectDispatch.

Signed-off-by: Or Ozeri <oro@il.ibm.com>