librbd: update hidden global config when setting pool config override

The new "dev"-level global config setting will be updated when any
pool-level config override is updated. librbd clients will detect
the new global-level config update and trigger a refresh. This avoids
the need for potentially tens of thousands of librbd clients
registering a watch on the pool metadata object or periodically polling
the pool metadata object for updates.

Fixes: https://tracker.ceph.com/issues/46694
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f45df9fe786e8057c491c082e840483759d67e9e)

Conflicts:
src/common/options.cc
- "rbd_quiesce_notification_attempts", "rbd_default_snapshot_quiesce_mode", and
  "rbd_plugins" options have not been backported to Octopus, yet

(cherry picked from commit ea2dec49276c5ea79980d184d97b6a410ee1dec5)

Conflicts:
src/common/options.cc: trivial resolution

librbd: initialize/shut down config watcher during image open/close

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ea6c9bde0437cc163a0f6a52b3ef1f45eaa6f88b)

Conflicts:
src/librbd/ImageCtx.h
- AsioEngine has not been backported yet

(cherry picked from commit ada201d460d37d68275ed4b35f6e3ec293f247fc)

librbd: initial config watcher implementation

The config watcher will initially observe all "rbd_" configuration
updates received from the MON that have not been locally overridden
at the pool and/or image level.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 54819a1aa95fe1be8b8ef44f8417e3fe82d9ec89)

Conflicts:
src/test/librbd/mock/MockImageState.h
- once again, I don't see why git flagged this as a conflict

(cherry picked from commit c830365a8a243bd4976a682af1bf8d768924eaff)

Conflicts:
src/librbd/ConfigWatcher.cc: image_lock -> md_lock
src/test/librbd/mock/MockImageState.h: trivial resolution

librbd: hold ImageCtx::md_lock while computing config overrides

The 'ImageCtx::apply_metadata' is guaranteed to be thread-safe but
a future commit will attempt to utilize the cached config override
set.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c7104d18e190f3718ec1625376f8c63676096efa)
(cherry picked from commit 652952f2d74df907637f03eff542a56e02a1ec8c)

Conflicts:
src/librbd/ImageCtx.cc: image_lock -> md_lock

Merge pull request #36415 from shyukri/wip-46638-nautilus

nautilus: doc/rbd: add rbd-target-gw enable and start

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #36390 from smithfarm/wip-46799-nautilus

nautilus: rgw: append obj: prevent tail from being GC'ed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #36494 from rhcs-dashboard/wip-make_check-nautilus

nautilus: run-make-check.sh: extract run-make.sh + run sudo with absolute path

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36379 from dzafman/wip-46096-nautilus-smithfarm

nautilus: mon: Warn when too many reads are repaired on an OSD

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36493 from rishabh-d-dave/cv-naut-test-fix

nautilus: ceph-volume: fix test_lvm.TestVolume.test_is_not_ceph_device

Merge pull request #35878 from ceph/wip-nautilus-32493

nautilus: ceph-volume: don't use container classes in api/lvm.py

Merge pull request #35881 from ceph/wip-nautilus-34744

nautilus: ceph-volume: handle idempotency with batch and explicit scenarios

Merge pull request #36435 from yuriw/wip-yuriw-cleanup-nautilus

qa/tests: moved client-upgrade* under new dir

Reviewed-by: Neha Ojha <nojha@redhat.com>

14.2.11

osd, test: Special Nautilus handling because there is no health mute

osd: Add clear_shards_repaired osd tell command
test: Replace mute handling in test case

Signed-off-by: David Zafman <dzafman@redhat.com>

test: flush_pg_stats() ignore OSDs that don't respond to getting sequence

This eliminates bogus errors in the logs and returned from flush_pg_stats()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 41322eaa62be0de20fdedb44d5817a3f1916ab5e)

mgr: Warn when too many reads are repaired on an OSD

Include test case
Configurable by setting mon_osd_warn_num_repaired (default 10)
Ignore new health warning with random eio injection test

Fixes: https://tracker.ceph.com/issues/41564
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 661996d4342c427209b1eae4b0247f8210a00fc3)

Conflicts:
PendingReleaseNotes
- add pending release note under version heading that makes sense for
  nautilus
        src/mon/PGMap.cc
- d0eb22f3ba557b9e98836995c813ea77c5e7c2a5 is not backported to
  nautilus, so the "health_check_t& add()" function takes only three
  arguments: omit the fourth

Merge pull request #36526 from batrick/i46860-followup

qa: skip cache_size check

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa: skip cache_size check

Now that we're using the memory limit, this is check is not necessary.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36513 from batrick/i46860

nautilus: mds: do not raise "client failing to respond to cap release" when client working set is reasonable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #36512 from batrick/i46858

nautilus: qa: add debugging for volumes plugin use of libcephfs

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36511 from batrick/i46856

nautilus: client: static dirent for readdir is not thread-safe

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #36462 from ukernel/nautilus-46831

nautilus: mds: initialize MDSlaveUpdate::waiter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35272 from jan--f/wip-doc-allow_standby_replay-nautilus

nautilus: doc: obsolete entries for allow_standby_replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use real entity for config_get

So this changed in Octopus/Pacific to allow

    ceph config get mds foo

but it doesn't work in Nautilus which expects a real entity:

https://pulpito.ceph.com/yuriw-2020-08-07_15:03:29-kcephfs-wip-yuri4-testing-2020-08-07-1350-nautilus-distro-basic-smithi/5308647/

    2020-08-07T18:56:49.830 INFO:teuthology.orchestra.run.smithi106.stderr:2020-08-07 18:56:49.833 7fbe0d7fa700  1 -- 172.21.15.106:0/369883181 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mds", "key": "mds_min_caps_per_client"}]=-22 unrecognized entity 'mds' v1) v1 ==== 131+0+0 (secure 0 0 0) 0x7fbdf8000940 con 0x7fbdf40064c0
    2020-08-07T18:56:49.830 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: unrecognized entity 'mds'

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add tests for mds_min_caps_working_set

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit f4fc138849cc821d1d327ac1c82206c14dac8028)

Conflicts:
qa/tasks/cephfs/test_client_limits.py

mds: add working set minimum for caps

A client may hold many inodes pinned in its cache for open files. That
client may be unable to release those caps to respond to cache pressure
from the MDS (or quiescent client cap recall). We should not complain if
that number of capabilities is reasonable (< 10k by default).

Fixes: https://tracker.ceph.com/issues/46830
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 63392e1b65fbead6ef8c7acd6a70e6ef5b322390)

qa: use config_set/config_get

It's simpler and does not require MDS restarts.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit edc5c14d1cb194760f91dcdb0385007df32aef45)

Conflicts:
qa/tasks/cephfs/test_client_limits.py

qa: do not append file names to dirname

Otherwise the files generated are not actually under the sub-directory!
This is correcting a confusing aspect of the test infrastructure but
doesn't actually require any changes to the tests.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit ac6c150eb07ddd61c33fc4fad034fee6f5fbad35)

qa: add exception for test timeouts

To make this easier to catch. It is still a RuntimeError so it should
not affect current tests by default.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit dbc44e5e342605c16e1213120bec0bfec1c0091d)

qa: log entrance/exit into the volumes plugin

Fixes: https://tracker.ceph.com/issues/46851
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit a89b1633e1940d8bd35baaa46ce894fb46670cd5)

Conflicts:
src/pybind/mgr/volumes/module.py

qa: add debug client for volumes plugin qa

Fixes: https://tracker.ceph.com/issues/46851
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 9bcda7ca42d5ad0cc60a41812e046e1ce7057474)

client: use non-static dirent for thread-safety

Fixes: https://tracker.ceph.com/issues/46832
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 9b00083489d1d4777519da38f3003cf164dae480)

ceph-volume: fix Volume mock in lvm test case

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit cb432fe41d4ea8cb71aa592e0727d2da1978121f)
(cherry picked from commit f713bfa695833cb7acc5c39ff1b3754f794b6f90)

run-make-check.sh: extract run-make.sh

so we can reuse run-make.sh for building the artifact used by other
tests than "make check", for instance, dashboard's E2E test and
crimson's performance test.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5e9a1d95c9f735955b92204df1f711d5098e2b2d)

Conflicts:
run-make-check.sh
        - Remove flags: -DWITH_CEPHFS_SHELL=ON -DWITH_SPDK=ON
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

run-make-check.sh: run sudo with absolute path

there is chance that `/sbin` is not listed in `secure_path`, see
sudoers(5) for more details

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 11f9aae4ca0da80bcb6ddfa1f9f912a8f03350de)
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mds: initialize MDSlaveUpdate::waiter

introduced by backport commit 4273de4cfaf
"mds: preserve ESlaveUpdate::OP_PREPARE logevent before doing commit"

Fixes: https://tracker.ceph.com/issues/46831
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

ceph-volume: filter by osd-id or osd-fsid when zapping

2f5c10c12c37e6865ce54bb4940d3779353cba4f introduced a bug:

`ceph-volume lvm zap` command fails under certain conditions.

when passing `--osd-id` or `--osd-fsid` to `ceph-volume lvm zap` command
it tries to zap additionnal devices that have nothing to do with the osd
being zapped.

When calling `api.get_lvs()` in `ensure_associated_lvs()` we have to
pass the osd-id/osd-fsid information so only related devices are
returned by `get_lvs()` method

Closes: https://tracker.ceph.com/issues/46627
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e94aef6f96189648d0f204d8118501577dde5f52)

ceph-volume: delete devices.lvm.prepare.Prepare.get_lv

Since it calls get_lv_from_argument() internally and use get_first_lv()
instead and update tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit ca1d3e694c6390632c86e6dfcf01a2b3c6a67f0d)

ceph-volume: don't use api.lvm.VolumeGroups

Use api.lvm.get_vgs() and api.lvm.get_first_vg() instead and update
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit dfe6b1b90384482e8ec38e3be7024971fb252968)

ceph-volume: don't use api.lvm.PVolumes

Update api.lvm.get_first_pv() and api.lvm.get_pvs() instead and update
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit a6e3c69c6cd561ac6680fceb3017b9a2afe3d9de)

ceph-volume: don't use api.lvm.get_lv

Use api.lvm.get_lvs() and api.lvm.get_first_lv() instead and update the
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit b2254050c01d872ebc906708a3b4c8d4fa2105d7)

ceph-volume: don't use api.lvm.get_lv_from_argument

Use api.lvm.get_lvs() and api.lvm.get_first_lv() instead and update
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 5301c4959e5169b10cc92716de6222f7516770e6)

ceph-volume: don't use class api.lvm.Volumes

Replace code using class Volumes by methods get_lvs() and
get_first_lv(). Also, update the tests accordingly.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 2f5c10c12c37e6865ce54bb4940d3779353cba4f)

qa/tests: moved client-upgrade* under new dir

The motivation for this chnage is to enable more developers to use and run upgrades tests in more simplified way as "-s upgrades/"

"client-upgrade*" don't fit this logic as they need to use suites from older branches, so that's why they are separated. 

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #36439 from batrick/i46533

nautilus: mds: fix nullptr dereference in MDCache::finish_rollback

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix nullptr dereference in MDCache::finish_rollback

introduced by commit 4940ab62e0d
"mds: preserve ESlaveUpdate::OP_PREPARE logevent before doing commit"

Fixes: https://tracker.ceph.com/issues/46533
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 6715514aa577842942a1221a0f25924c53e03834)

doc/rbd: add rbd-target-gw enable and start

This commit adds the following commands to  the "Configuring the iSCSI Target Using the Command Line" page: "systemctl enable rbd-target-gw" and "systemctl start rbd-target-gw"

Fixes: https://tracker.ceph.com/issues/45987
Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 4b126d30e5935c5efddf2dfd23023d8b3aa64057)

Merge pull request #35733 from idryomov/wip-msgr21-nautilus

nautilus: New msgr2 crc and secure modes (msgr2.1)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: append obj: prevent tail from being GC'ed

append object tail gets GC'ed otherwise as the state has a manifest similar to
atomic obj processor, but if the manifest exists and the position is correct, it
is not an overwrite and shouldn't be GC'ed

Fixes: https://tracker.ceph.com/issues/42670
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 2dad52a18f889cca4309e78a88c6c5c2c04d4947)

Merge pull request #36377 from batrick/i46763

nautilus: qa/tasks: call super class's setUp()

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36339 from smithfarm/wip-46741-nautilus

nautilus: osd: fix crash in _committed_osd_maps if incremental osdmap crc fails

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

ceph-volume: handle idempotency with batch and explicit scenarios

If you used --wal-devices or --db-devices with batch and too
many devices are filtered out then a RuntimeError was raised.

However, if --report and --format=json is used then we
should return valid json indicating that no OSDS will be created
so that ceph-ansible and other systems can use that for idempotency checks.

Resolves: rhbz#1827349

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9b8254edc8ee6c91809119120970b722edb100b7)

qa/tasks: call super class's setUp()

to address the regression introduced by
87292811215f6ded9a784d3216a910faaef648e2

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7d262db1145a14a73da481acba66e51b5f30e4af)

Merge pull request #36353 from neha-ojha/wip-46731-nautilus

nautilus: qa/*/nautilus-p2p-stress-split: rework objectstore backends

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #36314 from kotreshhr/wip-46641-nautilus

nautilus: qa: Fix name collisions

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/suites/rados/thrash/crc-failures: randomly inject bad incremental osdmap crc

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit f9399b508c6584e539e323da858372fa7115be3b)

osd: don't write transaction when inc crc failed

80da5f9a987c6a48b93f25228fdac85890013520 exposed a flaw in how
handle_osd_map falls back to a full osdmap if the crc of an incremental
failed.

If the first message in a map message had a crc error, then the
loop would exit with last < start, which would then cause a null
dereference in _committed_osd_maps.

Fixes: https://tracker.ceph.com/issues/46443
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
(cherry picked from commit 99463cb523949289384a6210ea250114fbe852f4)

qa/standalone/osd: add bad-inc-map.sh

Test that the osd doesn't crash when it gets a bad incremental osdmap.

Related-to: https://tracker.ceph.com/issues/46443
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
(cherry picked from commit b550112dba23528655ec150cf9d6d093318abb8c)

qa/*/nautilus-p2p-stress-split: rework objectstore backends

Earlier we were symlinking to qa/objectstore, which causes older versions of
nautilus to fail when trying to use hybrid allocator.

Fixes: https://tracker.ceph.com/issues/46731
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36250 from dillaman/wip-46673

nautilus: librbd: new 'write_zeroes' API methods to suppliment the `discard` APIs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #36292 from dzafman/wip-46706-nautilus

nautilus: osd: Cancel in-progress scrubs (not user requested)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36190 from smithfarm/wip-46458-nautilus

nautilus: build/ops: selinux: allow ceph_t amqp_port_t:tcp_socket

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36161 from smithfarm/wip-46090-nautilus

nautilus: osd/PG: fix history.same_interval_since of merge target again

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: Fix name collisions

Addresses the name collisions of volumes, subvolumes,
clones, subvolume groups and snapshots within the tests.

Fixes: https://tracker.ceph.com/issues/43517
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 3b6f6f1abf1beba54c7b4eb610ae05985cdf42b1)

Conflicts:
    qa/tasks/cephfs/test_volumes.py: Few of the test cases were not
present in nautilus which were present in master, removed them.

Merge pull request #36167 from ajarr/nautilus-mgr-fs-volumes-misc

nautilus: mgr/fs/volumes misc fixes

Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #36294 from ajarr/wip-46470

nautilus: client: release the client_lock before copying data in read

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36215 from vshankar/wip-46388

nautilus: client: introduce timeout for client shutdown

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36180 from ajarr/wip-46464-nautilus

nautilus: pybind/cephfs: fix custom exception raised by cephfs.pyx

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36179 from smithfarm/wip-46523-nautilus

nautilus: mds: fix hang issue when accessing a file under a lost parent directory

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36178 from smithfarm/wip-46521-nautilus

nautilus: mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36177 from smithfarm/wip-46517-nautilus

nautilus: cephfs: client: fix directory inode can not call release callback

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #36175 from smithfarm/wip-46474-nautilus

nautilus: mds: make threshold for MDS_TRIM configurable

Merge pull request #36173 from smithfarm/wip-46409-nautilus

nautilus: cephfs: client: fix setxattr for 0 size value (NULL value)

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #36172 from smithfarm/wip-46310-nautilus

nautilus: tests: qa/tasks/cephfs/cephfs_test_case.py: skip cleaning the core dumps when in program case

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #36171 from smithfarm/wip-46200-nautilus

nautilus: tests: Revert "Revert "qa/suites/rados/mgr/tasks/module_selftest: whitelist …

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #36170 from smithfarm/wip-46189-nautilus

nautilus: mds: reset heartbeat in EMetaBlob replay

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36169 from smithfarm/wip-46187-nautilus

nautilus: cephfs: client: fix snap directory atime

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #36270 from yuriw/wip-yuriw-nautilus-p2p-nautilus

nautilus: qa/tests: changed rbd-python.yaml workunit to v14.2.10

Reviewed-by: Neha Ojha <nojha@redhat.com>

nautilus: qa/test_volumes: Modify tests not to use json for 'ceph fs status'

Signed-off-by: Kotresh HR <khiremat@redhat.com>

nautilus: mgr/volumes: use MgrModule's mon_command()

... instead of 'check_mon_command' which is not in nautilus, and not
compatible with python2 and early versions of py3.

Signed-off-by: Ramana Raja <rraja@redhat.com>

nautilus: mgr/volumes: fix import typing error

... for python versions earlier than 3.5.

Signed-off-by: Ramana Raja <rraja@redhat.com>

pybind/mgr/volumes: fix volume search for pools

Loop logic would bail out if it first sees any file system that does not
match the volume it's looking for.

Fixes: https://tracker.ceph.com/issues/46277
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit be74a81e19ef3b5e8204d52e6a3f7f0c57d49854)

mgr/volumes: Validate mon_allow_pool_delete before volume deletion

Volume deletion wasn't validating mon_allow_pool_delete config
before destroying volume metadata. Hence when mon_allow_pool_delete
is set to false, it was deleting metadata but failed to delete pool
resulting in inconsistent state. This patch validates the config
before going ahead with deletion.

Fixes: https://tracker.ceph.com/issues/45662
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit e770bb9075d931913847a572d121e02a2e349ca8)

qa: manage config changes through mons

This provides a generic framework for modifying Ceph configuration
changes in tests through the monitors rather than the asok interface or
local ceph.conf changes. Any changes are reverted during test teardown.

A future patch will convert existing tests manipulating the local
ceph.conf or admin socket.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 87292811215f6ded9a784d3216a910faaef648e2)

mgr/volumes: Fix pool removal on volume deletion

While volume deletion, the associated pools are not always
removed. The pools are removed only if the volume is created
using mgr plugin and not if created with custom osd pools.
This is because mgr plugin generates pool names with specific
pattern. Both create and delete volume relies on it. This
patch fixes the issue by identifying the pools of the volume
without relying on the pattern.

Fixes: https://tracker.ceph.com/issues/45910
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit d07ea8db471b1a1d9082756e1cb775d5543b1307)

Conflicts:
src/pybind/mgr/volumes/fs/operations/volume.py:
- In nautilus, fs volume create doesn't have placement arg
src/pybind/mgr/volumes/fs/volume.py:
- In nautilus, VolumeClient code not moved to mgr_util.py

client: release the client_lock before copying data in read

Fixes: https://tracker.ceph.com/issues/46025
Signed-off-by: Chencan <chen.can2@zte.com.cn>
(cherry picked from commit db292da7d5571ed3addedfb4c5ab8c61fe9bb5e8)

Conflicts:
       src/client/Client.cc
- in nautilus, not switched bufferlist::copy() to iterators

Merge pull request #35500 from ifed01/wip-ifed-hybrid-alloc-nau

nautilus: os/bluestore: implement Hybrid allocator

Reviewed-by: akupczyk@redhat.com

qa/objectstore: test hybrid allocator instead of avl one.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 0f639740a3b99d0600f6508d90d426ca12501a03)

os/bluestore: set hybrid allocator as a default for bluefs/bluestore

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit cacc7391dca779c7e6e3b57e6d1753a53b8c49b9)

 Conflicts:
src/test/objectstore/store_test.cc
 trivial

os/bluestore: improve adjustent extents merge in hybrid allocatoc

Bitmap counterpart is inspected for adjustent free extents when
releasing extent and trying to add it to AVL tree.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit c27490e3d641f2b8ea65759fd79640e5f6dd398c)

os/bluestore: remove HybridAllocator::init_add_free method.

It's a 100% replica of AvlAllocator one which is inherited.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit cccbc62528e03f4e49882137342adfea20b9a0f2)

os/bluestore: configure hybrid allocator threshold to fallback.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit b682d9a8e8d3f41d55ad10d2fb3deca8edf8540a)

 Conflicts:
src/common/options.cc
 trivial

os/bluestore: add hybrid (avl+bitmap) allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit c90eee34955734a5d4d2d4f0d1ff04b275374fb1)

 Conflicts:
src/os/CMakeLists.txt
src/test/objectstore/Allocator_bench.cc
src/test/objectstore/Allocator_test.cc
src/test/objectstore/CMakeLists.txt
  GTEST_SKIP not available, +trivial

os/bluestore: do not require min_length to be power of two for bmap
allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 21532045efeb8d357518686df84a529ee3d32a60)

 Conflicts:
src/os/bluestore/fastbmap_allocator_impl.h
 variable rename

os/bluestore: extend AvlAllocator interface.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit cccf94da463aa15370fe674bda24d168d331ceca)

tests/test_bluestore_types: show AvlAllocator's range_seg_t size.

Extend corresponding test case to be aware of the struct's sizeof.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 719cd7d9423d1d16e3005cc0b807699ed24533de)

os/bluestore: make AvlAllocator overridable.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 59c0aa5eb40957364ddb0e9d62dffa3449991249)

os/bluestore: intdoduce spillover mechanics to AvlAllocator.

This to be utilized in AvlAllocator's descendant class(es) to be able
to limit amount of entries allocator tracks. Hence bringing a cap for
RAM used by the allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit efedcc077c7ed4e1e5ea26ae989fa3d6cbd287d4)

os/bluestore: avoid constant name replica in bitmap allocator.
This was introduced by Adam's https://github.com/ceph/ceph/pull/29425

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 683f7bab5a571cc3a6c7106fdb1d4017cc843c5d)

os/bluestore: implement original allocator fragmentation inspection.

Using admin socket interface: bluestore allocator fragmentation
[block|bluefs-db|bluefs-wal|bluefs-slow]

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 11b5feb01d273365c32ec6419faead629fb8c20b)