fuse_ll: fix compile error when libfuse version is larger than 3.0

Fixes: https://tracker.ceph.com/issues/56517
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 2f3f9d36afc891d55b1e4f13ca212c5aa040ee31)

ceph-fuse: add dedicated snap stag map for each directory

This will fix the fino colliding bug, which is caused when the
snapid is later than 0xffff.

From mds 'mds_max_snaps_per_dir' option, we can see that the max
snapshots for each directory is 4_K, and in ceph-fuse we have
around 64_K, which is from 0xffff - 2, stags could be used to make
the fake fuse inode numbers for each directory.

Fixes: https://tracker.ceph.com/issues/54653
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 7e4424a821f1f0162cc1f2bfddb1eb501e321702)

 Conflicts:
src/client/fuse_ll.cc

ceph-fuse: return EINVAL if get invalid fino instead of assert

All the snap ids of the finos returned to libfuse from libcephfs
will be recorded in the map of 'stag_snap_map', and will never be
erased before unmounting. So if libfuse passes invalid fino the
ceph-fuse should return EINVAL errno instead of crash itself.

Fixes: https://tracker.ceph.com/issues/54653
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit a6e83d8dece53d11d0753f4843a6235eb1bb9834)

ceph-fuse: reserve stag number 1 for snapdirs

There have two stags will be reserved, 0 for CEPH_NOSNAP and 1 for
CPEH_SNAPDIR.

This will always make sure that for the nonsnap and snapdir inode
numbers to be consistent for all the ceph-fuse mounts.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 2349083a9d3f503dbe48fb1bdc3a88c881ed11a9)

mds-client: make the fake inos option unchangeable in runtime

If the flags is empty then in option.h in can_update_at_runtime()
it will return true. That means this opetion could be changed in
runtime, which is buggy. Because if this is false, ceph-fuse will
use its own fake inos instead of libcephfs'. If this is changed
during runtime, we will hit inos dosn't exist assert bugs.

Fixes: https://tracker.ceph.com/issues/54653
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 28d17ff81a93d5442d2107abbd51296ac822073a)

 Conflicts:
src/common/options/mds-client.yaml.in

Merge pull request #47838 from zdover23/wip-doc-2022-08-28-backport-47736-to-pacific

pacific: doc: Update release process doc to accurately reflect current process

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: Update release process doc to accurately reflect current process

Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit 4829af891ce9fbe41f4b48b10d342e774366dd52)

Merge pull request #47525 from NitzanMordhai/wip-57076-pacific

pacific: osd: remove invalid put on message

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47807 from adk3798/pacific-rgw-bad-example

pacific: doc/cephadm: fix example for specifying networks for rgw

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc/cephadm: fix example for specifying networks for rgw

count_per_host must be used with underscores rather
than dashes to work, you need to pass service_id not
service_name and the option for the port is called
rgw_frontend_port not just "port"

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 6b6f56a5b623f68d59af8902eae5e1157f47e382)

Merge pull request #47781 from zdover23/wip-doc-2022-08-24-backport-47751-to-pacific

pacific: doc/mgr: edit orchestrator.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #47791 from zdover23/wip-doc-2022-08-25-backport-47785-pacific-2

pacific: docs: fix doc link pointing to master in dashboard.rst

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

docs: fix doc link pointing to master in dashboard.rst

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 79bbaa555307639cd10819c35c0d0c2f4cd7f0fd)

doc/mgr: edit orchestrator.rst

This PR improves the English language in the "Orchestrator CLI"
section of the MGR documentation. It adds a couple of section
headers in order to signpost the information in the document
a bit more than had already been done, but it makes no major
structural changes to the presentation of the information here.

This PR was motivated by feedback from the 2022 Ceph User Survey
in which one of the respondents wrote "better ceph orch documen-
tation".

The final section on this page, "Current Implementation Status",
must be verified by someone who is familiar with the current state
of "ceph orch" and a date stamp should be applied to the top of
the section so that the word "current" has a meaningful referent.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 2172b7ec98bacc183b0e3a22d95dc8ad80af5a8a)

Merge pull request #47446 from kamoltat/wip-ksirivad-backport-pacific-47138

pacific: qa/tasks/ceph_manager.py: increase test_pool_min_size timeout
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #47134 from Matan-B/wip-56579-pacific

pacific: osd/SnapMapper: fix legacy key conversion in snapmapper class

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #46642 from ifed01/wip-ifed-avl-update-cursor-pac

pacific: os/bluestore: Always update the cursor position in AVL near-fit search.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #47180 from lxbsz/wip-55929

pacific: mds: fix crash when exporting unlinked dir

Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #47059 from lxbsz/wip-56465

pacific: mds: switch to use projected inode instead

Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #47492 from petrutlucian94/wip-57053-pacific

pacific: common: use boost::shared_mutex on Windows

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #47451 from NitzanMordhai/wip-57029-pacific

pacific: librados: rados_ioctx_destroy check for initialized ioctx

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46901 from dparmar18/wip-56112-pacific

pacific: qa/cephfs: fallback to older way of get_op_read_count

Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #47321 from adk3798/wip-56504-pacific

pacific: doc/cephadm/services: the config section of service specs

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #47369 from kotreshhr/wip-56978-pacific

pacific: mgr/volumes: Fix subvolume creation in FIPS enabled system.

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #47587 from zdover23/wip-doc-2022-08-15-backport-47580-to-pacific

pacific: doc/rados/operations: add prompts to operating.rst

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados/operations: add prompts to operating.rst

This commit adds ".. prompt:: bash $"-style prompts to operating.rst.
This brings this file up to the standard established in 2020 when
Kefu added support for the ".. prompt::" directive.

This commit is a part of an initiative to modernize the presentation
of all BASH commands in the RADOS documentation.

The progress of this project can be tracked here:
https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 69d62c05ef0ac8b903944d9e8c8fe4fbb3f73275)

Merge pull request #47578 from idryomov/wip-alloc-size-doc-fixup-pacific

pacific: doc/man/rbd: Mention changed `bluestore_min_alloc_size`

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

doc/man/rbd: Mention changed `bluestore_min_alloc_size`

This change was done in: https://github.com/ceph/ceph/pull/34588

Signed-off-by: Niklas Hambüchen <mail@nh2.me>
(cherry picked from commit 254025343b1c49e1d0aa72fdea0cbe0483704245)

Merge pull request #47307 from yaarith/wip-56486-pacific

pacific: mgr/telemetry: reset health warning after re-opting-in

Merge pull request #47337 from ifed01/wip-ifed-fix-mempool-cache-other-pac

pacific: os/bluestore: fix AU accounting in bluestore_cache_other mempool

Reviewed-by: akuczyk@redhat.com

Merge pull request #47357 from rhcs-dashboard/wip-56966-pacific

pacific: mgr/dashboard: add required validation for frontend and monitor port

Merge pull request #47454 from rhcs-dashboard/wip-57038-pacific

pacific: mgr/dashboard: host list tables doesn't show all services deployed

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

osd: remove invalid put on message
message converted to use smart pointer, the put is extra step that
causing the double put and valgrind error
the regression was introduced by d4a71c3

Fixes: https://tracker.ceph.com/issues/52124
Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>
(cherry picked from commit c449638812daeafa89c592b9c396eaa66327f85f)

Merge pull request #47497 from tchaikov/pacific-pr-47486

pacific: mgr/dashboard: bump up teuthology

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #47211 from kamoltat/wip-ksirivad-backport-pacific-fix-56151

pacific:src/mgr/DaemonServer.cc: fix typo in output gap >= max_pg_num_change

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #46748 from myoungwon/pacific-53855

pacific: test: fix TierFlushDuringFlush to wait until dedup_tier is set on bas…

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #47111 from kotreshhr/wip-56527-pacific

pacific: mds: Don't blocklist clients in any replay state

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #46797 from mchangir/pacific-mgr-snap_schedule-persist-all-changes-to-rados

pacific: mgr/snap_schedule: persist all updates to RADOS

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #47372 from adk3798/wip-56743-pacific

pacific: mgr/cephadm: Add disk rescan feature to the orchestrator

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47484 from tchaikov/pacific-pr-47478

pacific: cmake: remove spaces in macro used for compiling cython code

Reviewed-by: Tim Serong <tserong@suse.com>

mgr/dashboard: bump up teuthology

to include the fix of e7c5d67e10fe29da22180f9e09b8973ae166c8fc,
see https://github.com/ceph/teuthology/pull/1746.
to address the test failure on ubuntu jammy. where we have python3.10

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 0aa1fa68d354badd3363a4ea6cacbce373c3316e)

Merge pull request #47413 from guits/pacific-cv-backports

ceph-volume: Pacific backports

common: use boost::shared_mutex on Windows

The winpthreads shared mutex implementation causes deadlocks on
Windows [1][2]. Specifically, async RBD IO calls are hanging. This
also prevents the images from being unmounted.

For this reason, we're switching to boost::shared_mutex when using
MinGW.

[1] https://github.com/cloudbase/wnbd/issues/63#issuecomment-1161547102
[2] https://github.com/msys2/MINGW-packages/issues/3319
Trace: https://pastebin.com/raw/i3jpTyS3

Fixes: https://tracker.ceph.com/issues/56480
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 35b67b2558fd4b7c39ca10e5b13b4dc6c983ce3c)

cmake: remove spaces in macro used for compiling cython code

we are facing following FTBFS on jammy + GCC-11.2 + Cython 0.29 +
CMake 3.22:

creating /home/jenkins-build/build/workspace/ceph-api/build/lib/cython_modules/temp.linux-x86_64-3.10/home/jenkins-build/build/workspace/ceph-api/build/src/pybind/cephfs
compile options: '-I/usr/include/python3.10 -I/usr/include/python3.10 -c'
extra options: '-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -iquote/home/jenkins-build/build/workspace/ceph-api/src/include -w -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2'
cc: /home/jenkins-build/build/workspace/ceph-api/build/src/pybind/cephfs/cephfs.c
cc: warning: ##: linker input file unused because linking not done
cc: error: ##: linker input file not found: No such file or directory
cc: warning: 0: linker input file unused because linking not done
cc: error: 0: linker input file not found: No such file or directory

it seems cython is not able to escape the space in the "extra options"
anymore, so the "##" and "0" are considered as object files passed to
compiler in addition to cephfs.c.

in this change the spaces are removed to help cython to make the right
decision.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 5824fed5b427f1d055fb7104fea2e68cd36e6844)

Conflicts:
cmake/modules/Distutils.cmake: trivial resolution

Merge pull request #47266 from cfsnyder/wip-56646-pacific

pacific: rgw: maintain object instance within RGWRadosObject::get_obj_state method

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

ceph-volume: filter out non-existing devices

[1] has broken teuthology tests.

```
[root@smithi097 /]# ls -l /sys/block/
total 0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 Jul 25 14:12 nvme0n1 -> ../devices/pci0000:00/0000:00:02.0/0000:02:00.0/nvme/nvme0/nvme0n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1c1n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme1/nvme1c1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2c2n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme2/nvme2c2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys2/nvme2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3c3n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme3/nvme3c3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys3/nvme3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4c4n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme4/nvme4c4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys4/nvme4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:12 sda -> ../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda
[root@smithi097 /]#
```

devices like `nvme1c1n1` exist in `/sys/block` but aren't present in `/dev`

```
[root@smithi097 /]# ls -l /dev/nvme*
crw------- 1 root root  10, 122 Jul 25 14:16 /dev/nvme-fabrics
crw------- 1 root root 240,   0 Jul 25 14:12 /dev/nvme0
brw-rw---- 1 root disk 259,   0 Jul 25 14:15 /dev/nvme0n1
crw------- 1 root root 240,   1 Jul 25 14:16 /dev/nvme1
brw-rw---- 1 root disk 259,   2 Jul 25 15:24 /dev/nvme1n1
crw------- 1 root root 240,   2 Jul 25 14:16 /dev/nvme2
brw-rw---- 1 root disk 259,   4 Jul 25 15:24 /dev/nvme2n1
crw------- 1 root root 240,   3 Jul 25 14:16 /dev/nvme3
brw-rw---- 1 root disk 259,   6 Jul 25 15:24 /dev/nvme3n1
crw------- 1 root root 240,   4 Jul 25 14:16 /dev/nvme4
brw-rw---- 1 root disk 259,   8 Jul 25 15:24 /dev/nvme4n1
[root@smithi097 /]#
```

If a device isn't actually present in `/dev`, `get_block_devs_sysfs()` shouldn't return it.

[1] https://github.com/ceph/ceph/commit/c7f017b21ade3762ba5b7b9688bed72c6b6

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 0ee8351e4d564724aa68a2b8d03bc53e281e7fc9)

ceph-volume/tests: fix test_exception_returns_default

There are cases, like running tests as root user, where this test fails
because root user can always read the file content.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 47aec3b86bcc850292f66d24670c5d8dc0a33c85)

ceph-volume: drop `ceph-bluestore-tool` call in Device._parse()

disk.has_bluestore_label() does the same thing without a subprocess call.

Fixes: https://tracker.ceph.com/issues/56622
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f5f3117510c287d817534c9f7c25af5f208c2f56)

ceph-volume: fix is_ceph_disk_member()

`dev['NAME']` can't match `part` given that it's the name of the
parent device being compared to the partition name.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 24899a1f8d77bc3c10ef158bb463349d615f7c57)
(cherry picked from commit 66d0a07b34faa4514b1b7cb2d46ed90d83b91963)

ceph-volume/tests: migrate to pyfakefs

ceph-volume unit tests shouldn't actually create contents on the
filesystem from where it runs (even though they are written in a tmp
dir), let's use pyfakefs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1274ba34b4d21c5abf2cd2da035d6d25ae8d03da)

ceph-volume/tests: fix test_path_is_valid()

When ceph-volume tests are run from a host where there's no `/dev/sda`
device, this test fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7ab9d99420db2995542dfab7c416021dcc1c4a47)

ceph-volume: report slave devices in inventory

`ceph-volume inventory` currently reports the following:

```

Device Path               Size        rotates available Model name
/dev/mapper/mpatha        50.00 GB    True    True
/dev/mapper/mpathaa       6.00 GB     True    True
/dev/mapper/mpathab       6.00 GB     True    True
/dev/mapper/mpathac       6.00 GB     True    True
/dev/mapper/mpathad       6.00 GB     True    True
/dev/mapper/mpathae       6.00 GB     True    True
```

whereas something like following would be useful:

```

Device Path               Size         Device nodes    rotates available Model name
/dev/mapper/mpatha        50.00 GB     sdf,sde         True    True
/dev/mapper/mpathaa       6.00 GB      sdbe,sdat       True    True
/dev/mapper/mpathab       6.00 GB      sdav,sdbf       True    True
/dev/mapper/mpathac       6.00 GB      sdbb,sdbl       True    True
/dev/mapper/mpathad       6.00 GB      sdas,sdbc       True    True
/dev/mapper/mpathae       6.00 GB      sdax,sdbh       True    True
```

Fixes: https://tracker.ceph.com/issues/56624
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1ac2a8270cf4aece58ae3f952cc4d1ef978384ed)

ceph-volume: support symlinks as devices

This makes ceph-volume support passing symlinks as devices.

Fixes: https://tracker.ceph.com/issues/49103
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 9ddc5b77372cd2e0d5badd433c93d147cb25c6ee)

ceph-volume: improve mpath devices reporting

An environment with mpath devices looks like following:

`lsblk` output:

```
sdy        65:128  0    6G  0 disk
`-mpathm  252:8    0    6G  0 mpath
sdz        65:144  0    6G  0 disk
`-mpathm  252:8    0    6G  0 mpath
```

`multipath -ll` output:

```
mpathm (3600140575bbe2d3c0c6493fb6e6ed84c) dm-8 LIO-ORG,TCMU device
size=6.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 2:0:0:30 sdz  65:144 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 3:0:0:30 sdy  65:128 active ready running
```

`ceph-volume inventory` output:
```
Device Path               Size         rotates available Model name
/dev/mapper/mpathm        6.00 GB      True    True
/dev/sdy                  6.00 GB      True    False     TCMU device
/dev/sdz                  6.00 GB      True    False     TCMU device
```

ceph-volume shouldn't report devices `/dev/sdy` and `/dev/sdz`, they will never be
available in such a scenario. Considering this, in a host with a bunch of mpath devices
it will pollute the inventory output.

Fixes: https://tracker.ceph.com/issues/56621
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e892f02a1a9126cbb520c90aeef0afb3590ddbbb)

ceph-volume: fix simple scan

When the class `Device` is instantiated with a path instead of a
block device, it fails like following.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 130, in __init__
    self._parse()
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 233, in _parse
    self.ceph_device = disk.has_bluestore_label(self.path)
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 906, in has_bluestore_label
    with open(device_path, "rb") as fd:
IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/osd/ceph-0/'
```

passing a path instead of a block device is valid, `simple scan` needs it.

Fixes: https://tracker.ceph.com/issues/56969
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5a10e3e9e865aafd9d3f90beb186777060d17f08)

ceph-volume: Remove refs to get_block_devs_lsblk

It's been replaced by get_block_devs_sysfs

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 2023e71f695c1962534c6012d5b331bebd10ec31)

ceph-volume: Rename env var; add warning

So that we can have a nice big warning that fires once per invocation, I
think using a callable class with a class attribute seems like a decent
approach. A closure could work too.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 69f58f51a2d7967d597a8d61c0ab20b52e8dc374)

ceph-volume: Optionally consume loop devices

A similar proposal was rejected in #24765; I understand the logic
behind the rejection, but this will allow us to run Ceph clusters on
machines that lack disk resources for testing purposes. We just need to
make it impossible to accidentally enable, and make it clear it is
unsupported.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit c7f017b21ade3762ba5b7b9688bed72c6b60dc0e)

ceph-volume: drop self.abspath in Device()

seems useless to have both self.path and self.abspath attributes.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 01112ba6894834287092d3b672ec548cb3a4d676)

ceph-volume: do not call get_device_vgs() per devices

let's call `ceph_volume.api.lvm.get_all_devices_vgs` only one time instead
so we avoid a bunch of subprocess calls that slow down the process when running
`ceph-volume inventory` command.

Fixes: https://tracker.ceph.com/issues/56623
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f5bd0f6da9a3c2f60df2ed6dfaa4c26831a3be37)

ceph-volume: fix fast device alloc size on mulitple device

The size computed by get_physical_fast_allocs() was wrong when the
function had multiple devices to treat.

For instance if there is 4 OSDs and 2 fast devices of each 10G while
allocating 2 slots per fast devvices. The behavior before was that each
slot would be 2.5G meaning that both fast devices would half full. The
behavior now is that each slot will take 5G so that the fast devices
would be full.

Fixes: https://tracker.ceph.com/issues/56031
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit d0f9e93914e2b7feac41a634311d74c146c8868b)

ceph-volume: fix shebang of install command

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit 5255d4be74f15032331ba24b708ad04193ff87f3)

ceph-volume: fix `simple scan`

`lsblk_all()` should return an empty dict `{}` if nothing was found.
If we raise `RuntimeError()` then the loop in `scan.Scan.main` will stop
and make ceph-volume fails because we don't try to catch this exception.
`scan.Scan.main()` has its own logic in order to detect the given path
is a ceph-disk created OSD anyway.

Fixes: https://tracker.ceph.com/issues/56482
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 2630a50acb4654332c0b6801e9214564bd09d3d0)

Merge pull request #47341 from cbodley/wip-56954

pacific: rgw: better tenant id from the uri on anonymous access

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw: PutObj initializes source bucket for copy-source

cherry-picks one more thing from 72d1a363263cf707d022ee756122236ba175cda2
to resolve a crash in CopyObj

Fixes: https://tracker.ceph.com/issues/56646
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: maintain object instance within RGWRadosObject::get_obj_state method

It seems that the object instance was being stripped from the rgw_obj
inadvertently wihtin the RGWRadosObject::get_obj_state method. This
change fixes some S3 behavior related to concurrency and ECANCELED errors.

This is a partial cherry-pick from:
https://github.com/ceph/ceph/commit/72d1a363263cf707d022ee756122236ba175cda2

Fixes: https://tracker.ceph.com/issues/56646
Signed-off-by: Cory Snyder <csnyder@iland.com>

Merge pull request #47041 from ifed01/wip-ifed-fix-s3-lock-pac

pacific: rgw: do not permit locked object version removal

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #47194 from adamemerson/wip-56586-pacific

pacific: rgw: Guard against malformed bucket URLs

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #47247 from idryomov/wip-56676-pacific

pacific: librbd: tweak misleading "image is still primary" error message

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #47161 from idryomov/wip-56549-pacific

pacific: librbd: bail from schedule_request_lock() if already lock owner

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47158 from idryomov/wip-48038-pacific

pacific: qa/suites/rbd: disable workunit timeout for dynamic_features_no_cache

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #47143 from idryomov/wip-56561-pacific

pacific: rbd: don't default empty pool name unless namespace is specified

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47128 from idryomov/wip-50734-pacific

pacific: qa/suites/rbd/pwl-cache: ensure recovery is actually tested

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47118 from idryomov/wip-56516-pacific

pacific: rbd-mirror: remove bogus completed_non_primary_snapshots_exist check

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #47295 from SUSE/wip-pacific-include-memory

pacific: include/buffer: include <memory>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #47196 from adamemerson/wip-54491-pacific

pacific: rgw: Fix data race in ChangeStatus

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47175 from yuvalif/wip-51943-pacific

pacific: rgw/notifications: Change in multipart upload notification behavior

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46579 from cbodley/wip-55969

pacific: rgw: check object storage_class when check_disk_state

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/dashboard: add cephadm e2e tests for checking count for services instances

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit 25b03d85153e368098ebe5bf066f0872449af729)

mgr/dashboard: cluster > hosts: host list tables doesn't show all services deployed

Fixes: https://tracker.ceph.com/issues/53210
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Service instances was displaying only the ceph services, but with these changes it'll display instances
of cephadm services as well.

(cherry picked from commit 4d50da7629145d40da3a2820c3b5c8cdb2bca33f)

Merge pull request #47404 from aaSharma14/wip-57004-pacific

pacific: mgr/dashboard: Show error on creating service with duplicate service id

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47105 from rhcs-dashboard/wip-56562-pacific

pacific: mgr/dashboard: rbd image pagination

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47431 from rhcs-dashboard/wip-57010-pacific

pacific: mgr/dashboard: remove token logging

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

librados: rados_ioctx_destroy check for initialized ioctx

If rados_ioctx_destroy called with un-initilized ioctx we will hit segmented fault
checking ioctx before calling put will prevent that.

Fixes: https://tracker.ceph.com/issues/55001
Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>
(cherry picked from commit e465cfc62ae0f982edb60291c67f4d76a71efd15)

Merge pull request #47443 from ceph/pacific-rtd

pacific: .readthedocs.yml: Always build latest doc/releases pages

qa/tasks/ceph_manager.py: Increase timeout

In test_pool_min_size():

1. Provided buffer time before we check
for recovery in ceph_manager.wait_for_recovery()

2. Increased timeout in ceph_manager.wait_for_clean()

3. Increased sleep time for
ceph_manager.all_active_or_peered()

Fixes:
https://tracker.ceph.com/issues/49777
https://tracker.ceph.com/issues/54511
https://tracker.ceph.com/issues/51904

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit ed73288102f4aa67da3485c70b2409b9eca1873b)

qa/tasks/ceph_manager.py: improve loggings

1. When `test_pool_min_size` hit the case where
`not all PGs are active or peered` we dump
each PG state that doesn't have active or
peered state

2. Improve logs message in `inject_pause()`.

3. Add logs for the `test_map_discontinuity()`.

4. In the function, `choose_action()`,
added more logs regarding `chance_down`.

5. Added more loggings to
`primary_affinity()`,
`thrash_pg_upmap_items()`,
`thrash_pg_upmap()`.

6. Make self.is_clean() dump the pgs that
are not active+clean.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 9ed53c82cf5d56bd4555084d3bf1c3c9948c4fe3)

.readthedocs.yml: Always build latest doc/releases pages

We don't backport PRs merged into doc/releases.  Therefore, when one browses to an older Ceph release version on docs.ceph.com (e.g., https://docs.ceph.com/en/pacific/), the information is out of date at best.

The doc/releases page is only accurate if browsing https://docs.ceph.com/en/latest/, for example.

So this post_checkout command will make sure we've checked out doc/releases from main before building and publishing.

Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit 055fe1f825b0629b7685d6d3d4d629ffc37a2d7c)

cephadm: give disk rescan its own parser object

It was using the same name as the agent. Missed this
in https://github.com/ceph/ceph/pull/46420/commits/6584c49bdf254e8750d23ed529478e6a9289e12f

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit f8eae87accf1f79646d761e50656838d1933662d)

Conflicts:
src/cephadm/cephadm

cephadm: Add tests for rescan function

Adds unittets for the rescan code

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 972fb0ab3c519141b2f07ac4f2d6bbb6b855441a)

Conflicts:
src/cephadm/tests/test_cephadm.py

docs: Add docs for new host rescan feature

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 4525113fa123a5b1e8a23d6aefc306e6772326bc)

mgr/cephadm: Add new host rescan command

Adds a host rescan command to invoke cephadm's
rescan-disks subcommand

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit cc9eb1812e9aa138d7af7426b42657ace9020bea)

cephadm: Add rescan_disks subcommand

Add a new subcommand to probe each HBA to
force discovery of decies not automatically detected
by the kernel.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
(cherry picked from commit 6584c49bdf254e8750d23ed529478e6a9289e12f)

Conflicts:
src/cephadm/cephadm

Merge pull request #47375 from adk3798/wip-56985-pacific

pacific: mgr/cephadm: clear error message when resuming upgrade

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47376 from adk3798/wip-56984-pacific

pacific: cephadm: Fix repo_gpgkey should return 2 vars

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

mgr/dashboard: add required validation for frontend and monitor port

Fixes: https://tracker.ceph.com/issues/56688
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit ea770fd858cae9a8b52c60b293f48cc4dbc925f8)

mgr/dashboard: remove token logging

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit fdaf909fba0c399b8d9bbc3a29a5af871546d9d9)

qa: Fix use of 'sudo' args

https://tracker.ceph.com/issues/56727
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 5a33f7e6f6c1fbaa5078983ca2f0901296b65588)

mgr/volumes: Fix subvolume creation in FIPS enabled system.

The md5 checksum is used in the construction of legacy
subvolume config filename. It's not used for security reason.
Hence marking the 'usedforsecurity' flag to false to
make it FIPs compliant.

The usage of md5 was always in there. The commit 373a04cf734
made it to get exercised in 'open_subvol' which is pre-requisite
for all the subvolume operations and hence subvolume
creation has failed.

Fixes: https://tracker.ceph.com/issues/56727
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit ced3fac48d3da2320827c6c86ece3b87953badc7)

Merge pull request #47271 from vumrao/wip-vumrao-56701

pacific: libcephsqlite: ceph-mgr crashes when compiled with gcc12

Reviewed-by: Laura Flores <lflores@redhat.com>