doc/ceph-volume: add lvm migrate/new-db/new-wal

The inital PR was implementing those new commands only adds the man page
and not the online documentation.

Fixes: https://tracker.ceph.com/issues/51814
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 22de626220e6145fce8e1fd963802170f099f733)

Merge pull request #43076 from guits/wip-52525-pacific

pacific: ceph-volume: pvs --noheadings replace pvs --no-heading

Merge pull request #42937 from batrick/i51977

pacific: client: make sure only to update dir dist from auth mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43007 from ifed01/wip-ifed-fix-52311-pac

pacific: os/bluestore: fix using incomplete bluefs log when dumping it.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #43001 from ktdreyer/pacific-52449-rm-lsb-release

pacific: Remove dependency on lsb_release

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43000 from ktdreyer/pacific-52471-rm-gdbm

pacific: ceph.spec.in: drop gdbm from build deps

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42987 from MrFreezeex/wip-52461-pacific

pacific: rbd-mirror: add perf counters to snapshot replayer

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #42980 from idryomov/wip-50522-pacific

pacific: rbd: fix default pool handling for nbd map/unmap

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #42979 from idryomov/wip-52063-pacific

pacific: rbd-mirror: fix potential async op tracker leak in start_image_replayers

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #42972 from idryomov/wip-51867-pacific

pacific: pybind/rbd: fix mirror_image_get_status

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43029 from sebastian-philipp/pacific-backport-42535-42682-42252-42293-42816-42589-42590-42592-42690-42405-

pacific: cephadm: September batch 1

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

ceph-volume: pvs --noheadings replace pvs --no-heading

Fixes: https://tracker.ceph.com/issues/52482
Signed-off-by: FengJiankui <fengjiankui@inspur.com>
(cherry picked from commit 7083c4ffc63274bdd3103e9c5296bc70acbe745f)

cephadm: Add test_command_list_networks

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 8b768153b8829692fe5ad833a8f86c70f8101360)

cephadm: test_networks: Move tests from test_cephadm

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 419dfd6afae96739efe52a979603b257dd000568)

cephadm: list-networks: Avoid duplicated IPs

Fixes: https://tracker.ceph.com/issues/52083
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit b65822fc8cd3cbf541d50e9ee69d77ef9d1e9cfd)

cephadm: Fix TestCephVolume.test_noop

Fixes `PermissionError: [Errno 13] Permission denied: '/var/lib/ceph/760b...5a5fbb'`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 1978cd33238351045a8b7d856a1e35e36221f09a)

cephadm: Set tcmalloc env var for Ceph daemons

We don't need to set the TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES environment
variable for non Ceph daemons.

Fixes: https://tracker.ceph.com/issues/52334
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit c08c56324fd53a60518cd75859fe8ec8771e4104)

cephadm:Add listening ports to gather-facts output

This patch adds tcp and udp listening ports to the data
returned by gather-facts. This can be used to check port
availability prior to trying to deploying daemons, to
catch port conflicts earlier. IPv4 and IPv6 are supported

Fixes: #52038
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 0c4e88993bc04d248b1616ba4d46ebe24bf17381)

cephadm: check for openntpd.service as time sync service

openntpd is an alternative implementation of time synchronization
by the openbsd project and is packaged for debian and ubuntu
since at least jessie / 18.04 with the service named openntpd.service

Signed-off-by: Oleander Reis <oleander.reis@hostserver.de>
(cherry picked from commit dadf786e77f0adc2f62fd91a70f4ed31a82c1e32)

mgr/cephadm: make scheduler able to accomodate offline/maintenance hosts

Fixes: https://tracker.ceph.com/issues/51027
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 01a580b3082f83347181bf835b9ef698cecfc015)

cephadm: better port stripping

in case the specified port number contains leading zeros

Signed-off-by: Yuxiang Zhu <vfreex@gmail.com>
(cherry picked from commit 496a2c40c87858c188467b7dbacf5e2385ad521c)

cephadm: propagate environment variables to subprocesses

so that I can use an http(s) proxy for external network access when running cephadm.

e.g.
```sh
http_proxy=http://proxy:8080 https_proxy=http://proxy:8080 cephadm pull
```

Signed-off-by: Yuxiang Zhu <vfreex@gmail.com>
(cherry picked from commit 8f72f774892be6aec458452fc2678578f2436e90)

cephadm: Fix bootstrap error with IPv6 mon-ip

Fix the following error by removing the enclosing square brackets in `ctx.mon_ip`.

This issue also occurs on pacific.

```sh
$ sudo cephadm --verbose bootstrap --mon-ip fd49:277:42:2020:c4ba:e6ff:fedf:604

/usr/bin/ceph: stderr > ssh -F ssh_config -i ~/cephadm_private_key root@[fd49:277:42:2020:c4ba:e6ff:fedf:604]
Traceback (most recent call last):
  File "/usr/sbin/cephadm", line 3776, in prepare_ssh
    cli(args)
  File "/usr/sbin/cephadm", line 4076, in cli
    return CephContainer(
  File "/usr/sbin/cephadm", line 3263, in run
    out, _, _ = call_throws(self.ctx, self.run_cmd(),
  File "/usr/sbin/cephadm", line 1453, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16 -e NODE_NAME=ceph-node2 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/fee89f24-f3af-11eb-87b1-c6bae6df0604:/var/log/ceph:z -v /tmp/ceph-tmpnjeaui27:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjfm82r9d:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16 orch host add ceph-node2 [fd49:277:42:2020:c4ba:e6ff:fedf:604]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/cephadm", line 8230, in <module>
    main()
  File "/usr/sbin/cephadm", line 8218, in main
    r = ctx.func(ctx)
  File "/usr/sbin/cephadm", line 1759, in _default_image
    return func(ctx)
  File "/usr/sbin/cephadm", line 4142, in command_bootstrap
    prepare_ssh(ctx, cli, wait_for_mgr_restart)
  File "/usr/sbin/cephadm", line 3778, in prepare_ssh
    raise Error('Failed to add host <%s>: %s' % (host, e))
__main__.Error: Failed to add host <ceph-node2>: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16 -e NODE_NAME=ceph-node2 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/fee89f24-f3af-11eb-87b1-c6bae6df0604:/var/log/ceph:z -v /tmp/ceph-tmpnjeaui27:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjfm82r9d:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16 orch host add ceph-node2 [fd49:277:42:2020:c4ba:e6ff:fedf:604]
Releasing lock 140115256165328 on /run/cephadm/fee89f24-f3af-11eb-87b1-c6bae6df0604.lock
Lock 140115256165328 released on /run/cephadm/fee89f24-f3af-11eb-87b1-c6bae6df0604.lock
```

Signed-off-by: Yuxiang Zhu <vfreex@gmail.com>
(cherry picked from commit d3f5aaf40267a168f9709be0292b160638297999)

mgr/cephadm/upgrade: Upgrade status: We also upgrade non-ceph daemons

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit bcd127b46a097d18f43ee829ff8b97033902b05e)

doc/cephadm: rewrite client-setup.rst

This improves the text in client-setup.rst.

We should make certain that the technical details
in this file remain current in July 2021. This
file was origingally written in November 2019.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 18a15891df23222a60affdf9cfcd5760cfb0fa05)

mgr/dashboard: set rgw credentials: fix api tests

Fixes: https://tracker.ceph.com/issues/44605
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit a682b9d7a49fbd50c69494f947a6c0565ad0c78e)

mgr/dashboard: run-frontend-e2e-tests.sh: remove unneeded rgw setting

Fixes: https://tracker.ceph.com/issues/44605
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 8e482f36f2b4eca228f8990690414c0ec513bf88)

mgr/dashboard: rgw service creation form: add realm and zone to service spec.

Align rgw service id pattern with cephadm: https://github.com/ceph/ceph/pull/39877
  - Update rgw pattern to allow service id for non-multisite config.
  - Extract realm and zone from service id (when detected) and add them to the service spec.

Fixes: https://tracker.ceph.com/issues/44605
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 0575844192502ded32962b75a91cf51de22e97e6)

mgr/dashboard: connect-rgw: rename to set-rgw-credentials; refactoring

- Rename the dashboard command to better reflect its behavior.
- Rename '_radosgw_admin' method to 'send_rgwadmin_command' for consistency with
  'send_mon_command' and move it to the mgr_module.py .
- Cleanup: remove unneeded rgw settings.
- Better error handling and test coverage.

Fixes: https://tracker.ceph.com/issues/44605
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 6e20ef1dd35f3681d14cd4e08ca63eb20edc2c88)

mgr/dashboard: connect-rgw: adaptation and test coverage

- Align Dashboard with cephadm: configure credentials using the same logic.
- Fix: create a 'dashboard' user per realm (before: only on 1st realm).
- Lint fixes, test coverage, method renaming to better reflect behavior and method visibility.

Fixes: https://tracker.ceph.com/issues/44605
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 0fcf0a7827cf4e8748a382613f9c8d1715c4a1e8)

mgr/cephadm: re-check dashboard <-> rgw creds when rgw daemons created/destroyed

We don't always know when a realm is created/destroyed, but we can use
service config and purge to cover most such cases.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 91e185d956495885575f052848204169937efde7)

mgr/dashboard: add 'dashboard connect-rgw' command

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 478c53ba68feb0448d9b4bb8be2f5012f7d85b9b)

doc/mgr/dashboard: simplify dashboard+rgw config docs

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 599116a0680296776445d614c55b63f145a700b6)

cephadm: no need to explicitly enable prometheus module

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 4dc860feef4802d0432c54f3a40584ad7e9138bf)

mgr/cephadm: enable prometheus module before deploying prometheus

The mon will restart the mgr when the module is enabled, so we don't
really have to do anything here.  The raise is there just in case the
mgr doesn't immediately get the new mgrmap and respawn, although there is
likely no harm done if we continue to deploy prometheus in the meantime,
even if we're interrupted partway through.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 50dc1d0decb2fbf7b9129bddba940969410be5cd)

mgr/cephadm: drop daemon_id arg to CephadmService.config()

Unused (and nonsensical since this is *service* config).

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit e9dabdab6540d949bd3401fef66ec583368b33de)

doc/cephadm: no need to manually enable the prometheus module

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit afa509ec572b1099c2e8d8c69787a764a1fcc44a)

doc/install/containers: quay.io!

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit d1c744ee534a6196964ef546f4e9bbd302907dfe)

mgr/cephadm: add ability to remove offline host

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>
(cherry picked from commit 219887301dfdeac55c75a7105bf3a2851fe56387)

Merge pull request #42939 from batrick/i51832

pacific: mds: META_POP_READDIR, META_POP_FETCH, META_POP_STORE, and cache_hit_rate are not updated

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #42938 from batrick/i51932

pacific: mds: MDCache.cc:5319 FAILED ceph_assert(rejoin_ack_gather.count(mds->get_nodeid()))

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #42936 from batrick/i51198

pacific: msg: active_connections regression

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #42935 from batrick/i51935

pacific: mds: add debugging when rejecting mksnap with EPERM

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #42956 from ifed01/wip-ifed-fix-huge-omap-rename-pac

pacific: os/bluestore: cap omap naming scheme upgrade transaction.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42534 from liewegas/use-quay-pacific

pacific: cephadm: use quay, not docker

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42889 from rhcs-dashboard/wip-52378-pacific

pacific: mgr/dashboard: stats=false not working when listing buckets

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

os/bluestore: fix using incomplete bluefs log when dumping it.

BlueFS superblock might contain incomplete list of physical extents for
bluefs log. Hence we should alway replay ops for ino 1 to get them properly.

Fixes: https://tracker.ceph.com/issues/52311
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 47632fe9baf7f20921c3e6888e29c4478fa5ff2d)

Merge pull request #42974 from s0nea/wip-52308-pacific

pacific: mgr/dashboard: don't notify for suppressed alerts

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

rpm, debian: remove lsb-release

Nothing requires this any more, so we can remove it from the RPM
and Debian packaging.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit cbc0127f3435a1c62344df377e19577c4931b376)

 Conflicts:
debian/control - trivial ordering

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

make-debs.sh: remove lsb_release

The lsb_release utility brings in a lot of other dependencies. Remove
it from the make-debs script.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 92f73a2bb94d7ff645072b98c2b9c7eea8208782)

test/docker-test: remove lsb_release

The lsb_release utility brings in a lot of other dependencies. Remove
it from the docker-test script.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 24af043ee57419bd28d72ffc85cd1078df9af123)

workunits/rbd: remove lsb_release

The lsb_release utility brings in a lot of other dependencies. Remove
it from the RBD workunit script.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 65f4d6eb3664a4cc6120031679b8368cbc02a4a5)

 Conflicts:
qa/workunits/rbd/qemu-iotests.sh - trivial fix

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

workunits/rgw: remove lsb_release

The lsb_release utility brings in a lot of other dependencies. Remove
it from the RGW workunit Perl scripts.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit db66f6f35735d2aae2170af3520d222cf748ef9b)

ceph.spec.in: drop gdbm from build deps

neither gdbm nor libgdbm is used when building ceph, so let's drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 9d05d7337460322b78d676bf32a52c4519c7ae1f)

Merge pull request #42898 from smithfarm/wip-52380-pacific

pacific: cmake: exclude "grafonnet-lib" target from "all"

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #42897 from ifed01/wip-ifed-fix-spurious-read-errors-warn-pac

pacific: os/bluestore: respect bluestore_warn_on_spurious_read_errors setting

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42888 from callithea/wip-51951-pacific

pacific: Don't persist report data

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: cap omap naming scheme upgrade transactoin.

We shouldn't use single per-onode transaction for such an upgrade when onode's omap list is huge. This results in similarly sized WAL/SST files which are inefficient, might cause high memory usage and sometimes error-prone.

Fixes: https://tracker.ceph.com/issues/49170
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit e897fa243c1dd38329733b452872616023f14ac8)

Merge pull request #42969 from idryomov/wip-krbd-escape-match-sysattr-pacific

pacific: krbd: escape udev_enumerate_add_match_sysattr values

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

rbd-mirror: add perf counters to snapshot replayer

Add bytes replayed and statistics about number of snapshots synchronized
and how long it took to do so.

Fixes: https://tracker.ceph.com/issues/50973
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit cf9317b58f966c64554f46092fc4162b5e8fd5fb)

qa/workunits/rbd-nbd: add new test for map/unmap

This patch includes twe new test cases:
 a. map/unmap test with only image name and
 b. map/unmap test after changing default pool which expects the image
 to come from new default pool.

Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
(cherry picked from commit b19905272a7ef510e23b0ddf6156a90633a2831a)

qa/workunits/rbd: use `rbd device` wrapper where appropriate

Use `rbd device` wrapper call instead of making a direct call to rbd-nbd.

Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
(cherry picked from commit 4f6370b8843d3ddf2bcb0a53c7bf62037021014a)

rbd: fix default pool handling for nbd/ggate map/unmap

Default pool is not being picked up when providing only image name
during map/unmap. However, if the pool name is provided manually map/unmap
works as expected.

Fixes: https://tracker.ceph.com/issues/50522
Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
(cherry picked from commit 987e017f20dc419f45f222a35c9ccca79551af7d)

rbd-mirror: fix potential async op tracker leak in start_image_replayers

Fixes: https://tracker.ceph.com/issues/52063
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 2f83b43afc6f6864655dc5e37e4b8cfb88b4a805)

mgr/dashboard: don't notify for suppressed alerts

Fixes: https://tracker.ceph.com/issues/51987
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
(cherry picked from commit e9f316d678dd24e25e82474da95dbfbb72d763b5)

cephadm: quay.io for non-ceph images too

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm,mgr/cephadm: use quay for ceph image

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/rbd: explain why "primary" isn't exposed in mirror_image_status_list()

"primary" is part of mirror image info (rbd_mirror_image_info_t) and
is exposed in mirror_image_get_status().  mirror_image_status_list(),
even though it is often thought of as an equivalent of repeated calls
to mirror_image_get_status(), doesn't actually fetch the mirror image
info.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 1d2a142108b2aa102103e62ccdf3bd17688b783b)

pybind/rbd: actually append site_status dict to remote_statuses

Using += operator is wrong -- only site_status keys get appended
(and repeatedly at that in case there is more than one remote site
as the keys are added one by one).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 89888beb1266be5a661fef8643bb6bef0c720f5f)

rbd: Fix mirror_image_get_status in rbd python bindings

When retrieving the status of a mirrored image from the Python rbd
library, a TypeError is raised.

*To Reproduce:*

Set up two Ceph clusters for block storage, and configure image
mirroring between their pools.  Create a least one image with mirroring
enabled, then run the following script on either cluster (once the image
exists everywhere):

```python
import rados, rbd

CONF_PATH = "YOUR-CONF-PATH"
POOL_NAME = "YOUR-POOL-NAME"
IMAGE_LABEL = "YOUR-IMAGE-LABEL"

with rados.Rados(conffile=CONF_PATH) as cluster:
  with cluster.open_ioctx(POOL_NAME) as ioctx:
    with rbd.Image(ioctx, IMAGE_LABEL) as image:
      image.mirror_image_get_status()
```

This will result in the following stack trace:

```
Traceback (most recent call last):
  File "repo-bug.py", line 10, in <module>
    image.mirror_image_get_status()
  File "rbd.pyx", line 3363, in rbd.requires_not_closed.wrapper
  File "rbd.pyx", line 5209, in rbd.Image.mirror_image_get_status
TypeError: list indices must be integers or slices, not str
```

Fixes: https://tracker.ceph.com/issues/51867
Signed-off-by: Will Smith <wsmith@linode.com>
(cherry picked from commit 5dfda932b2012bb11a1860d8a81de3208b17f622)

krbd: escape udev_enumerate_add_match_sysattr values

libudev uses fnmatch(3) for matching attributes, meaning that shell
glob pattern matching is employed instead of literal string matching.
Escape glob metacharacters to suppress pattern matching.

Fixes: https://tracker.ceph.com/issues/52425
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 8841029b0a4705825ba394541240b3cb4eb2cf5c)

Merge pull request #42428 from kamoltat/wip-ksirivad-backport-pacific-42036

pacific: mgr/pg_autoscaler: Introduce autoscaler scale-down feature
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: add autoscaler profile

Added autoscaler profile release notes for
>=16.2.6

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit ea094f769f554ff7d5e4a17e76130d00044ef4e4)

doc/rados/operations/placement-groups: added bias + profile

Added documentations on the autoscale profile and bias

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 65fec053e003967aa3ffa5008730661baa46c868)

pybind/mgr/pg_autoscaler: Added PROFILE to autoscale-status

autoscaler now shows the PROFILE status of each pool
in `ceph osd pool autoscale-status`

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 4a2912bbad72b00ae9fe0d892f07ca760e5cefb4)

pybind/mgr/autoscaler: don't scale pools with overlapping roots

In the previous version of get_subtree_resource_status() in
src/pybind/mgr/pg_autoscaler/module.py we ignore overlapping
pools which in some cases if combined with the new `scale-down`
algorithm in https://github.com/ceph/ceph/pull/38805 can cause
some pools to scale up/down to inapproriate amount of pgs.

Therefore, the PR identifies the overlapping roots and prevent the pools
with such roots from scaling. This only happens with `scale-down` profile
as we see no problem with the default `scale-up` profile.

Removed the variable `pool_root` since it is not used anywhere in
the code, it only gets assigned and reassigned

Also included a unit test test_overlapping_roots.py that tests the function
identify_subtrees_and_overlaps() as well as edited test_cal_final_pg_target.py
to account for pools that contain overlapping roots, therefore, those pools
are expected not to scale.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 16eb42a1d8cef5cf008b04b27d51e13dbd6ec495)

pybind/mgr/pg_autoscaler: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 1b0b32e44cddacbd7acf7024181247f4251c92f9)

Conflicts:
src/mypy.ini - trivial fix

pybind/mgr/pg_autoscaler: do not index pool_root with pool_name

pool_root is indexed by pool_id, and we never index it with pool_name.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 83f841295d86d10aadf3b4dff4e3eaa80fe139c3)

pybind/mgr/pg_autoscaler: do not create a tuple long_desc

long_desc is supposed to be a str, not a tuple of str.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f725e09ae63702097f0eef1395391933083a9d96)

pybind/mgr/pg_autoscaler: extract CrushSubtreeResourceStatus out

as it also serves as part of interface of get_subtree_resource_status(),
not only its internals. to ease adding the type annotations, this class
is promoted out of the class.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 9bb55e84809e17e0aa1cb1aa4ae712a3ea4fffed)

pybind/mgr/pg_autoscaler: define commands using CLICommand

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4efd7400804ff42b5965ad2691223471b4ae38a7)

pybind/mgr/pg_autoscaler: define options using Option

more consistent and less error-prune this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b40de19c23b6b49f8d63bb3911bdebe9a2e19255)

pybind/mgr/pg_autoscaler: Added autoscale-profile feature

The autoscaler by default will start out each pool with minimal
pgs and `scale-up` the pgs when there is more usage in each pool.

Users can now use the commands:

`osd pool set autoscale-profile scale-down` to make the pools
start out with a full complement of pgs and only `scale-down`
when usage ratio across the pools are not even.

`osd pool set autoscale-profile scale-up` (by default) to make the pools
start out with minimal pgs and `scale-up` the pgs when there
is more usage in each pool.

Edited KVMonitor.cc file to make the `autoscale_profile` variable
persistent.

Edited tests/test_cal_final_pg_target.py so that it takes into account
the new `profile` argument when calling cal_final_pg_target(). Also,
added some new test cases for when profile is `scale-up`

Renamed tests/test_autoscaler.py to a more appropriate name:
tests/test_cal_ratio.py

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 0e49683b538026e4c51a040950da3dbb9c7e92d5)

mgr/pg_autoscaler: avoid scale-down until there is pressure

The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.

Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools

Edited workunit script to reflect the change
of how pgs are calculated and distrubted.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit daeb6f6ac0c8f77ae07147f9d1e2ed18d6d8e4cc)

Merge pull request #42914 from vshankar/wip-51544

pacific: mgr/volumes: use dedicated libcephfs handles for subvolume calls and …

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #42900 from ajarr/wip-51834-pacific

pacific: mds: create file system with specific ID

Reviewed-by: Ramana Raja <rraja@redhat.com>

mds: update META_POP_READDIR/FETCH/STORE and cache_hit_rate for dump loads

Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>
(cherry picked from commit 24e131a58ed255cb8f6845c020ab956450a5fc10)

mds: check rejoin_ack_gather before enter rejoin_gather_finish

Fixes: https://tracker.ceph.com/issues/48422
Signed-off-by: chencan <chen.can2@zte.com.cn>
(cherry picked from commit 88548775330f66226b4e58a34256f1cc5a72221f)

client:make sure only to update dir dist from auth mds
Fixes: https://tracker.ceph.com/issues/51857
Signed-off-by: Xue Yantao <xueyantao2114@163.com>
(cherry picked from commit 1cafa0868797d67583031de63c10150a87fd05a4)

qa/tasks/cephfs/test_sessionmap: reap connections immediately

We have to reap connections promptly for this test to work.

This test was broken indirectly by d51d80b3234e17690061f65dc7e1515f4244a5a3,
which moved the counter decrement to reap time instead of mark_down/stop
time.

The reaping is asynchronous, so allow for a delay in the count change.

Fixes: https://tracker.ceph.com/issues/50622
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit c8c5071dcd4b0b788f5e924a678095ce5dc1d7f8)

msg/async: configurable threshold for reaping dead connections

It is helpful to set this to 1 for tests.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 8129d6bb953015cc05db458afa6aa9b8f5f62614)

Conflicts:
src/common/options/global.yaml.in

mds: add debugging when rejecting mksnap with EPERM

Fixes: https://tracker.ceph.com/issues/51543
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 92ec79fd57b841c54bd9a5490031f62166ed6ab1)

Merge pull request #42083 from kotreshhr/wip-51411-pacific

pacific: mgr/mgr_util: switch using unshared cephfs connections whenever possible

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #42908 from cfsnyder/wip-52330-pacific

pacific: cls/cmpomap: empty values are 0 in U64 comparisons

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/volumes: use dedicated libcephfs handles for subvolume calls and async jobs

Fixes: http://tracker.ceph.com/issues/51271
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit cb2883feac1a5c141a3d72120c2320f7a8ffdea8)

Merge pull request #42860 from ideepika/wip-52336-pacific

pacific: mon/PGMap: remove DIRTY field in ceph df detail when cache tiering is not in use

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42844 from aclamk/resharding-column-options-pacific

pacific: kv/RocksDBStore: Add handling of block_cache option for resharding

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42615 from neha-ojha/wip-51830-pacific

pacific: common/options: Set osd_client_message_cap to 256.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42300 from callithea/wip-51241-pacific

pacific: debian/control: ceph-mgr-modules-core does not Recommend ceph-mgr-roo…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42578 from batrick/i51983

pacific: mon/MDSMonitor: do not pointlessly kill standbys that are incompatible with current CompatSet

Reviewed-by: Ramana Raja <rraja@redhat.com>