Merge pull request #47435 from guits/quincy-bp-pr47434

quincy: backport: ceph-volume/tests: fix test_exception_returns_default

Merge pull request #47329 from rhcs-dashboard/wip-56947-quincy

quincy: mgr/dashboard: added pattern validaton for form input

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47430 from rhcs-dashboard/wip-57009-quincy

quincy: mgr/dashboard: remove token logging

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47442 from ceph/quincy-rtd

quincy: .readthedocs.yml: Always build latest doc/releases pages

Merge pull request #47268 from adk3798/quincy-osd-config

quincy: mgr/cephadm: add parsing for config on osd specs

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

.readthedocs.yml: Always build latest doc/releases pages

We don't backport PRs merged into doc/releases.  Therefore, when one browses to an older Ceph release version on docs.ceph.com (e.g., https://docs.ceph.com/en/pacific/), the information is out of date at best.

The doc/releases page is only accurate if browsing https://docs.ceph.com/en/latest/, for example.

So this post_checkout command will make sure we've checked out doc/releases from main before building and publishing.

Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit 055fe1f825b0629b7685d6d3d4d629ffc37a2d7c)

mgr/cephadm: add parsing for config on osd specs

Cephadm, while parsing spec files, can parse ceph configuration
for almost all the services, except for OSDs, where it fails
with a nasty "unexpected keyword argument config".

This commit fixes this issue.

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit 74273e7e6e79c474e33fc41480399dc27ad2ae64)

Merge pull request #47311 from adk3798/wip-56744-quincy

quincy: mgr/cephadm: Add disk rescan feature to the orchestrator

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47312 from adk3798/wip-56742-quincy

quincy: mgr/cephadm: support for miscellaneous config files for daemons

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47316 from adk3798/wip-56611-quincy

quincy: mgr/cephadm: support for os tuning profiles

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47313 from adk3798/wip-56741-quincy

quincy: cephadm: reduce spam to cephadm.log

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47314 from adk3798/wip-56740-quincy

quincy: cephadm: add "su root root" to cephadm.log logrotate config

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47315 from adk3798/wip-56738-quincy

quincy: qa/workunits/cephadm: update test_repos master -> main

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47317 from adk3798/wip-56584-quincy

quincy: mgr/cephadm: check for events key before accessing it

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47373 from adk3798/wip-56986-quincy

quincy: mgr/cephadm: clear error message when resuming upgrade

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47374 from adk3798/wip-56983-quincy

quincy: cephadm: Fix repo_gpgkey should return 2 vars

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #46467 from mdw-at-linuxbox/mdw-quincy-mankey

quincy -- sse s3 changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume/tests: fix test_exception_returns_default

There are cases, like running tests as root user, where this test fails
because root user can always read the file content.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 47aec3b86bcc850292f66d24670c5d8dc0a33c85)

Merge pull request #47356 from rhcs-dashboard/wip-56965-quincy

quincy: mgr/dashboard: add required validation for frontend and monitor port

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: remove token logging

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit fdaf909fba0c399b8d9bbc3a29a5af871546d9d9)

Merge pull request #47403 from aaSharma14/wip-57003-quincy

quincy: mgr/dashboard: Show error on creating service with duplicate service id

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47195 from adamemerson/wip-54208-quincy

quincy: rgw: Fix data race in ChangeStatus

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47191 from adamemerson/wip-56585-quincy

quincy: rgw: Guard against malformed bucket URLs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47415 from zdover23/wip-doc-2022-08-03-backport-47197-to-quincy

quincy: doc/dev: Elaborate on boost .deb creation

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: Elaborate on boost .deb creation

Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit 3222485d6582ba6ab3ee908c13b6459c2e2b6302)

Merge pull request #47149 from guits/wip-56496-quincy

quincy: ceph-volume: fix `simple scan`

Merge pull request #47406 from guits/quincy-cv-backports

ceph-volume: Quincy backports

ceph-volume: fix is_ceph_disk_member()

`dev['NAME']` can't match `part` given that it's the name of the
parent device being compared to the partition name.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 24899a1f8d77bc3c10ef158bb463349d615f7c57)

ceph-volume: filter out non-existing devices

[1] has broken teuthology tests.

```
[root@smithi097 /]# ls -l /sys/block/
total 0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 Jul 25 14:12 nvme0n1 -> ../devices/pci0000:00/0000:00:02.0/0000:02:00.0/nvme/nvme0/nvme0n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1c1n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme1/nvme1c1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2c2n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme2/nvme2c2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys2/nvme2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3c3n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme3/nvme3c3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys3/nvme3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4c4n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme4/nvme4c4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys4/nvme4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:12 sda -> ../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda
[root@smithi097 /]#
```

devices like `nvme1c1n1` exist in `/sys/block` but aren't present in `/dev`

```
[root@smithi097 /]# ls -l /dev/nvme*
crw------- 1 root root  10, 122 Jul 25 14:16 /dev/nvme-fabrics
crw------- 1 root root 240,   0 Jul 25 14:12 /dev/nvme0
brw-rw---- 1 root disk 259,   0 Jul 25 14:15 /dev/nvme0n1
crw------- 1 root root 240,   1 Jul 25 14:16 /dev/nvme1
brw-rw---- 1 root disk 259,   2 Jul 25 15:24 /dev/nvme1n1
crw------- 1 root root 240,   2 Jul 25 14:16 /dev/nvme2
brw-rw---- 1 root disk 259,   4 Jul 25 15:24 /dev/nvme2n1
crw------- 1 root root 240,   3 Jul 25 14:16 /dev/nvme3
brw-rw---- 1 root disk 259,   6 Jul 25 15:24 /dev/nvme3n1
crw------- 1 root root 240,   4 Jul 25 14:16 /dev/nvme4
brw-rw---- 1 root disk 259,   8 Jul 25 15:24 /dev/nvme4n1
[root@smithi097 /]#
```

If a device isn't actually present in `/dev`, `get_block_devs_sysfs()` shouldn't return it.

[1] https://github.com/ceph/ceph/commit/c7f017b21ade3762ba5b7b9688bed72c6b6

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 0ee8351e4d564724aa68a2b8d03bc53e281e7fc9)

ceph-volume: drop `ceph-bluestore-tool` call in Device._parse()

disk.has_bluestore_label() does the same thing without a subprocess call.

Fixes: https://tracker.ceph.com/issues/56622
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f5f3117510c287d817534c9f7c25af5f208c2f56)

ceph-volume: fix simple scan

When the class `Device` is instantiated with a path instead of a
block device, it fails like following.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 130, in __init__
    self._parse()
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 233, in _parse
    self.ceph_device = disk.has_bluestore_label(self.path)
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 906, in has_bluestore_label
    with open(device_path, "rb") as fd:
IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/osd/ceph-0/'
```

passing a path instead of a block device is valid, `simple scan` needs it.

Fixes: https://tracker.ceph.com/issues/56969
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5a10e3e9e865aafd9d3f90beb186777060d17f08)

ceph-volume/tests: migrate to pyfakefs

ceph-volume unit tests shouldn't actually create contents on the
filesystem from where it runs (even though they are written in a tmp
dir), let's use pyfakefs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1274ba34b4d21c5abf2cd2da035d6d25ae8d03da)

ceph-volume/tests: fix test_path_is_valid()

When ceph-volume tests are run from a host where there's no `/dev/sda`
device, this test fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7ab9d99420db2995542dfab7c416021dcc1c4a47)

ceph-volume: report slave devices in inventory

`ceph-volume inventory` currently reports the following:

```

Device Path               Size        rotates available Model name
/dev/mapper/mpatha        50.00 GB    True    True
/dev/mapper/mpathaa       6.00 GB     True    True
/dev/mapper/mpathab       6.00 GB     True    True
/dev/mapper/mpathac       6.00 GB     True    True
/dev/mapper/mpathad       6.00 GB     True    True
/dev/mapper/mpathae       6.00 GB     True    True
```

whereas something like following would be useful:

```

Device Path               Size         Device nodes    rotates available Model name
/dev/mapper/mpatha        50.00 GB     sdf,sde         True    True
/dev/mapper/mpathaa       6.00 GB      sdbe,sdat       True    True
/dev/mapper/mpathab       6.00 GB      sdav,sdbf       True    True
/dev/mapper/mpathac       6.00 GB      sdbb,sdbl       True    True
/dev/mapper/mpathad       6.00 GB      sdas,sdbc       True    True
/dev/mapper/mpathae       6.00 GB      sdax,sdbh       True    True
```

Fixes: https://tracker.ceph.com/issues/56624
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1ac2a8270cf4aece58ae3f952cc4d1ef978384ed)

ceph-volume: support symlinks as devices

This makes ceph-volume support passing symlinks as devices.

Fixes: https://tracker.ceph.com/issues/49103
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 9ddc5b77372cd2e0d5badd433c93d147cb25c6ee)

ceph-volume: drop self.abspath in Device()

seems useless to have both self.path and self.abspath attributes.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 01112ba6894834287092d3b672ec548cb3a4d676)

ceph-volume: improve mpath devices reporting

An environment with mpath devices looks like following:

`lsblk` output:

```
sdy        65:128  0    6G  0 disk
`-mpathm  252:8    0    6G  0 mpath
sdz        65:144  0    6G  0 disk
`-mpathm  252:8    0    6G  0 mpath
```

`multipath -ll` output:

```
mpathm (3600140575bbe2d3c0c6493fb6e6ed84c) dm-8 LIO-ORG,TCMU device
size=6.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 2:0:0:30 sdz  65:144 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 3:0:0:30 sdy  65:128 active ready running
```

`ceph-volume inventory` output:
```
Device Path               Size         rotates available Model name
/dev/mapper/mpathm        6.00 GB      True    True
/dev/sdy                  6.00 GB      True    False     TCMU device
/dev/sdz                  6.00 GB      True    False     TCMU device
```

ceph-volume shouldn't report devices `/dev/sdy` and `/dev/sdz`, they will never be
available in such a scenario. Considering this, in a host with a bunch of mpath devices
it will pollute the inventory output.

Fixes: https://tracker.ceph.com/issues/56621
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e892f02a1a9126cbb520c90aeef0afb3590ddbbb)

ceph-volume: Remove refs to get_block_devs_lsblk

It's been replaced by get_block_devs_sysfs

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 2023e71f695c1962534c6012d5b331bebd10ec31)

ceph-volume: filter RBD devices from the device inventory

Avoid running `blkid` or deploying OSDs on RBD devices by ensuring they
do not appear in the `ceph-volume inventory`

Fixes: https://tracker.ceph.com/issues/53846
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 47325ec3ec5ce1d53c5eae2952f631e95b7135fe)

ceph-volume: Rename env var; add warning

So that we can have a nice big warning that fires once per invocation, I
think using a callable class with a class attribute seems like a decent
approach. A closure could work too.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 69f58f51a2d7967d597a8d61c0ab20b52e8dc374)

ceph-volume: Optionally consume loop devices

A similar proposal was rejected in #24765; I understand the logic
behind the rejection, but this will allow us to run Ceph clusters on
machines that lack disk resources for testing purposes. We just need to
make it impossible to accidentally enable, and make it clear it is
unsupported.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit c7f017b21ade3762ba5b7b9688bed72c6b60dc0e)

Merge pull request #47293 from MrFreezeex/wip-56630-quincy

quincy: ceph-volume: fix fast device alloc size on mulitple device

Merge pull request #47348 from guits/wip-56711-quincy

quincy: ceph-volume: do not call get_device_vgs() per devices

mgr/dashboard: Show error on creating service with duplicate service id

Fixes: https://tracker.ceph.com/issues/56689
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 07cfd44193f6ebd552d13325858e8b5b5c131bfb)

Merge pull request #47248 from idryomov/wip-56676-quincy

quincy: librbd: tweak misleading "image is still primary" error message

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #47162 from idryomov/wip-56549-quincy

quincy: librbd: bail from schedule_request_lock() if already lock owner

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47159 from idryomov/wip-48038-quincy

quincy: qa/suites/rbd: disable workunit timeout for dynamic_features_no_cache

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #47144 from idryomov/wip-56561-quincy

quincy: rbd: don't default empty pool name unless namespace is specified

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47129 from idryomov/wip-50734-quincy

quincy: qa/suites/rbd/pwl-cache: ensure recovery is actually tested

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47126 from idryomov/wip-56516-quincy

quincy: rbd-mirror: remove bogus completed_non_primary_snapshots_exist check

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #47297 from aclamk/wip-56668-quincy

quincy: os/bluestore: Fix collision between BlueFS and BlueStore deferred writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: add required validation for frontend and monitor port

Fixes: https://tracker.ceph.com/issues/56688
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit ea770fd858cae9a8b52c60b293f48cc4dbc925f8)

Merge pull request #47344 from ceph/quincy-release

v17.2.3

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47391 from adk3798/quincy-fix-flake8

quincy: pybind/mgr: fix flake8

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: remove bogus completed_non_primary_snapshots_exist check

This check was added in commit ecd3778a6f9a ("rbd-mirror: ensure that
the last non-primary snapshot cannot be pruned") as an additional
safeguard against pruning an incomplete non-primary snapshot in case
there is no predecessor mirror snapshot.  However it still fires if the
predecessor is there but happens to be a primary demotion snapshot.
A bogus "incomplete local non-primary snapshot" error is reported and
the replayer gets stuck.

Remove completed_non_primary_snapshots_exist tracking as the presence
of the predecessor in the incomplete non-primary snapshot pruning arm
is already ensured by "m_local_snap_id_start > 0" condition.

Fixes: https://tracker.ceph.com/issues/56516
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit a581509381ba84b49c906a1fe440ca3ddcab418c)

test/rbd-mirror: add last_copied_object_number == 0 coverage

Incomplete non-primary snapshot handling is bifurcated depending
on whether any data objects have been copied.  If no data objects
have been copied, an incomplete non-primary snapshot is assumed to
be malformed and gets pruned; the sync is restarted from scratch.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 13aa47ae2ee89babca7571165ac88eaa665d2fc0)

qa/tasks: rename persistent write log cache trash task

It doesn't really thrash anything, just repeatedly restarts the
workload on top of a dirty cache file.  rbd_pwl_cache_recovery is
more on point and gets covered by existing CODEOWNERS.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 2de0574382e2c1c63f20745d6870ac7f82b27b9f)

qa/tasks: add thrash test for persistent write log cache

add thrash test for persistent write log cache. run rbd bench
on persistent write log cache, thrashes rbd bench, test the
recovery function of persistent write log cache.

Signed-off-by: Yin Congmin <congmin.yin@intel.com>
(cherry picked from commit 0eab8de3c017d8318bd6c846991bb3f7c51fa97d)

rbd: drop unused default_empty_pool_name argument

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 365300f253fd8066ae6f8cbd36c94ff4b145ab8d)

rbd: don't default empty pool name unless namespace is specified

Commit 96f05a7956b3 ("rbd: delay determination of default pool name")
broke "rbd perf image iostat" and "rbd perf image iotop" GLOBAL_POOL_KEY
support (the ability to blend all rbd pools together into a single
view).

Fixes: https://tracker.ceph.com/issues/56561
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit b2137e205862e6cfc316c11036266da65a78d26d)

qa/suites/rbd: disable workunit timeout for dynamic_features_no_cache

The I/O workload in this test is xfstests (qa/run_xfstests_qemu.sh)
which isn't subjected to any timeout other than global max_job_time
limit in any other subsuite (e.g. qemu/workloads/qemu_xfstests.yaml).
But here, there is a parallel "op" workload defined as a workunit.
The workunit task has a default timeout of 3 hours which is effectively
imposed on the entire job.  In the "rbd cache = false" configuration,
it's sometimes exceeded.

Fixes: https://tracker.ceph.com/issues/48038
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 0a6a70760a6f771b848852e0779a221e22c8775d)

librbd: bail from schedule_request_lock() if already lock owner

Race condition may be hit if there are multiple pending locks for the
same image and pending callbacks. Abort exclusive lock process if
already exclusive lock owner.

Fixes: https://tracker.ceph.com/issues/56549
Signed-off-by: Christopher Hoffman <choffman@redhat.com>
(cherry picked from commit 3527d2c764626c09c5ede80ae844551fd8845756)

librbd: tweak misleading "image is still primary" error message

m_promotion_state == PROMOTION_STATE_NON_PRIMARY doesn't say anything
about the remote image.  It could still be primary but it could also be
demoted.

Fixes: https://tracker.ceph.com/issues/56676
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 95dd89b4d771a8e01ce52beecc701548bf8321e8)

mgr/cephadm: fix flake8

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit a124f6c47b119a8741f347ea5a809f3fb48d6679)

cephadm: ceph.manual.gpg cleanup

Cleanup file /etc/apt/trusted.gpg.d/ceph.manual.gpg when repo is removed

Signed-off-by: Laurent Barbe <laurent@ksperis.com>
(cherry picked from commit bf0951353a80dc17edcc85a7914cfaf2b956d544)

cephadm: Fix repo_gpgkey should return 2 vars

when option --gpg-url is specified, the name used for the gpg filename is missing and throws an exception
this adds the string "manual" to the gpg key : /etc/apt/trusted.gpg.d/ceph.manual.gpg

Fixes: https://tracker.ceph.com/issues/56950
Signed-off-by: Laurent Barbe <laurent@ksperis.com>
(cherry picked from commit 79c805546c9bf4747a9019f4ad59f55e03c9fcef)

mgr/cephadm: clear error message when resuming upgrade

the message field in the output of "ceph orch upgrade status"
will first take the value of the error field of the UpgradeState,
and if only if it blank/None, display an info string we periodically
update throughout the upgrade with useful info such as that
we're upgrading a daemon of a particular type or pulling an image
on a certain host. When an upgrade fails, we set the error field
of the UpgradeState, pause the upgrade and raise a health warning.
Sometimes, the user is able to resolve the issue and simply resume
the upgrade. The issue here is, in that case, the error field of
the UpgradeState is still set, so instead of seeing the useful info
messages, it will continue to display an error message that may
no longer be relevant. By emptying the error field of the UpgradeState
when upgrades are resumed, we return to normal behavior of
displaying the info string, and will only show another error message
if another error actually occurs.

Fixes: https://tracker.ceph.com/issues/56714
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 429561ccb7b524f071214ff3aad99ba8830a924c)

cephadm: give disk rescan its own parser object

It was using the same name as the agent. Missed this
in https://github.com/ceph/ceph/pull/46420/commits/6584c49bdf254e8750d23ed529478e6a9289e12f

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit f8eae87accf1f79646d761e50656838d1933662d)

cephadm: Add tests for rescan function

Adds unittets for the rescan code

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 972fb0ab3c519141b2f07ac4f2d6bbb6b855441a)

Conflicts:
src/cephadm/tests/test_cephadm.py

docs: Add docs for new host rescan feature

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 4525113fa123a5b1e8a23d6aefc306e6772326bc)

mgr/cephadm: Add new host rescan command

Adds a host rescan command to invoke cephadm's
rescan-disks subcommand

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit cc9eb1812e9aa138d7af7426b42657ace9020bea)

cephadm: Add rescan_disks subcommand

Add a new subcommand to probe each HBA to
force discovery of decies not automatically detected
by the kernel.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 6584c49bdf254e8750d23ed529478e6a9289e12f)

Merge pull request #44641 from rzarzynski/wip-bs-lazy4freebsd-quincy

quincy: bdev: fix FTBFS on FreeBSD, keep the huge paged read buffers.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

bdev: fix FTBFS on FreeBSD, keep the huge paged read buffers.

Special thanks to Willem Jan Withagen!

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit d04ea43cfd0a834f470677d17a15d9c9d5c1c2e7)

ceph-volume: fix shebang of install command

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit 5255d4be74f15032331ba24b708ad04193ff87f3)

ceph-volume: fix fast device alloc size on mulitple device

The size computed by get_physical_fast_allocs() was wrong when the
function had multiple devices to treat.

For instance if there is 4 OSDs and 2 fast devices of each 10G while
allocating 2 slots per fast devvices. The behavior before was that each
slot would be 2.5G meaning that both fast devices would half full. The
behavior now is that each slot will take 5G so that the fast devices
would be full.

Fixes: https://tracker.ceph.com/issues/56031
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit d0f9e93914e2b7feac41a634311d74c146c8868b)

Merge pull request #47359 from zdover23/wip-doc-2022-07-29-backport-47347-to-quincy

quincy: doc/radosgw: Uppercase s3

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: make s3 uppercase

s/s3/S3/

(cherry picked from commit 73f0d5707d275529416d5110160b9ff5ead23d22)

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

ceph-volume: do not call get_device_vgs() per devices

let's call `ceph_volume.api.lvm.get_all_devices_vgs` only one time instead
so we avoid a bunch of subprocess calls that slow down the process when running
`ceph-volume inventory` command.

Fixes: https://tracker.ceph.com/issues/56623
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f5bd0f6da9a3c2f60df2ed6dfaa4c26831a3be37)

mgr/dashboard: added pattern validaton for form input

Fixes: https://tracker.ceph.com/issues/56133
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
(cherry picked from commit ec688a96d32eac8185351ff8bf3239131b083423)

ceph-volume: fix `simple scan`

`lsblk_all()` should return an empty dict `{}` if nothing was found.
If we raise `RuntimeError()` then the loop in `scan.Scan.main` will stop
and make ceph-volume fails because we don't try to catch this exception.
`scan.Scan.main()` has its own logic in order to detect the given path
is a ceph-disk created OSD anyway.

Fixes: https://tracker.ceph.com/issues/56482
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 2630a50acb4654332c0b6801e9214564bd09d3d0)

17.2.3

Signed-off-by: Ceph Release Team <ceph-maintainers@ceph.io>

Merge pull request #47270 from vumrao/wip-vumrao-56702

quincy: libcephsqlite: ceph-mgr crashes when compiled with gcc12

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

libcephsqlite: ceph-mgr crashes when compiled with gcc12

regex in libcephsqlite, when compiled with GCC12 treats '-' as a range
operator resulting in the following error.
"Invalid start of '[x-x]' range in regular expression"

Fixes: https://tracker.ceph.com/issues/55304
Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
(cherry picked from commit ac043a09c5ffb4b434b8644920004b3d5b7f9d8c)
(cherry picked from commit 5a10875d4570356afa3af024f63ac229caaa765b)

mgr/cephadm: check for events key before accessing it
Fixes: https://tracker.ceph.com/issues/55808
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 992d63849b005bdc97b89991060f11666965d536)

doc/cephadm: os tuning profile documentation

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 39cc904ca6a032490868cb6d87d2ecd8e5895430)

mgr/cephadm: unit tests for tuned os profiles

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 4f6ee4e20b74bc545b0c44db0208b94a54c648e1)

mgr/cephadm: support for os tuning profiles

Fixes: https://tracker.ceph.com/issues/55819
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 91e6e80ce38031d5eba148efd7c4aaede09021ac)

Conflicts:
src/python-common/ceph/deployment/service_spec.py

qa/workunits/cephadm: update test_repos master -> main

Missed this one, it seems the test was still passing until
recently but I guess the "master" builds are now gone

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 9b08edf939752d16b5933e1d7fd5bc33d08c2d4a)

cephadm: add "su root root" to cephadm.log logrotate config

Fixes: https://tracker.ceph.com/issues/56639
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit c0929e7e3ea14b0f8bfc71dc8ff74efddafb3ffc)

cephadm: use new QUIET log level to reduce spam to cephadm.log

The QUIET log level only gets sent to the console or log file
if --verbose is passed to cephadm. This means our regulat metadata
gathering operations won't spam the log file as we don't pass --verbose
when running them, but users can still pass --verbose when running
the commands manually for debugging purposes

Fixes: https://tracker.ceph.com/issues/56552
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit e98d51cbc1db671c935f2d3c20470f72703b0922)

cephadm: add QUIET log level

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit d1ff637b16414fd5388dc36fcf6f0fec019cac5f)

doc/cephadm: custom config files documentation

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit fa08b55acea0d2fe11e6bfdcf1b414bbf92f5566)

mgr/cephadm: unit testing for custom config files

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 78598ea58410891cbd38a450500753d7124d476e)

mgr/cephadm: allow mounting custom conf files

Fixes: https://tracker.ceph.com/issues/56394
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit de43dab44012fc239e4c8c09be264bf5a5a7981c)

src/python_common: Add CustomConfig class

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 61a6ec0399ac33843d39c9b4bd9ca23ad72e4154)

Conflicts:
src/python-common/ceph/deployment/drive_group.py

os/bluestore: Fix deferred writes corrupting RocksDB

Deferred writes can sometimes update regions that are no longer mapped to any object.
This cannot happen when BlueStore is running, as blobs are being held,
and allocations are not released until deferred op is executed.
However in case of restart allocations that deferred is targetting are already freed.
Deferred replay is done on BlueStore bootup, before any new object can be allocated,
so no collision with object is possible.
But BlueFS can allocate space from block with deferred ops still pending.

Fixes: https://tracker.ceph.com/issues/54547
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 2afb951a490aac9ca4d01614867396cbd06b793d)

Conflicts: src/os/bluestore/BlueStore.cc
Modified non-critical BlueFS call foreach_block_extents to get_block_extents.

common/interval_set: Add operator-- to iterator

Only ++ was allowed. Extend iterator manipulation to -- too.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 58860ce3f60489d258aaa10fd783e68083261937)

test/objectstore: Add test for deferred writes

Add test that recreates situation when BlueStore deferred writes
can cause RocksDB files corruption.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 42de47ae8468d000d5bca62bd537d7a1028bae42)

libcephsqlite: ceph-mgr crashes when compiled with gcc12

regex in libcephsqlite, when compiled with GCC12 treats '-' as a range
operator resulting in the following error.
"Invalid start of '[x-x]' range in regular expression"

Fixes: https://tracker.ceph.com/issues/55304
Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
(cherry picked from commit ac043a09c5ffb4b434b8644920004b3d5b7f9d8c)