mgr/cephadm: loop over all vips when trying to find ingress' interface

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit 1b9a6a0f58a9a7550e8b93573b3191816da5f900)

Split single and multiple vips test into 2 functions

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit f6d4ab9f25e5c9ee1872dbfd18bebbaf9a72a2d0)

mgr/cephadm: update haproxy/keepalive service test for newly generated files

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit a69a6fb4f5275af8a2757003f7fb5ca1f1ab9d2f)

mgr/cephadm: set explicit * bind for haproxy when using multiple vips

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit a11e181b98ffccff40939068d86254e7f8a98c06)

mgr/cephadm: update doc for multiple vips for ingress

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit 7b064e8b0eab0b577470122534e1b2647f5191cc)

mgr/cephadm: set test for multiple vips options for ingress service

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit 5915d2ecd08c1289da38d4fbeb646898f9c5dccf)

mgr/cephadm: allow for multiple vip configuration on ingress service

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit 0193a6f73659f7aa4ac1d000cf11c6544ad6ab6d)

Merge pull request #47579 from idryomov/wip-alloc-size-doc-fixup-quincy

quincy: doc/man/rbd: Mention changed `bluestore_min_alloc_size`

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

doc/man/rbd: Mention changed `bluestore_min_alloc_size`

This change was done in: https://github.com/ceph/ceph/pull/34588

Signed-off-by: Niklas Hambüchen <mail@nh2.me>
(cherry picked from commit 254025343b1c49e1d0aa72fdea0cbe0483704245)

Merge pull request #47549 from ceph/quincy-spec2

quincy: ceph.spec.in backports

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

ceph.spec.in: disable system_pmdk on aarch64

Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit fcf48cda0a8933c998d1ce51adecca4cd2b0efe2)

ceph.spec.in: always use stock compiler on el9

as RHEL/CentOS 9, we have the access to GCC-11, which is good enough for
compiling main HEAD even with WITH_SEASTAR=ON.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 215c07ba8b21481e1951dfbfb2d52fbc0e58eecc)

ceph.spec.in: define %gts_prefix

less repeatings this way, also  allow maintainer to use other version of GTS.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 53302f5f7f793aea73f521e8403eedda42357a92)

ceph.spec.in: %enable_devtoolset11 only if the macro is defined

there is chance that we are using `yum-builddep` to prepare the
build dependencies. in that case, gcc-toolset-11-build is not
installed. it's like a chicken-egg dilemma, but the point is
`yum-builddep` is able to pull in the gcc-toolset-11-build. once
gcc-toolset-11-build is installed, we will have the %enable_devtoolset11
rpm macro.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 75dd92353381c6ec5ac7251585a276059b8a2253)

ceph.spec.in: add libatomic to BuildRequires on fedora

otherwise we'd have failures like

/opt/compiler-explorer/gcc-trunk-20220808/bin/../lib/gcc/x86_64-linux-gnu/13.0.0/../../../../x86_64-linux-gnu/bin/ld:
/tmp/ccVlMbVh.o: in function `std::atomic<tagged_ptr>::store(tagged_ptr,
std::memory_order)':
/opt/compiler-explorer/gcc-trunk-20220808/include/c++/13.0.0/atomic:273:
undefined reference to `__atomic_store_16'

when generating the building system using CMake on fedora 36.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 7e122341d901c7bf02d388d69058c167626d55b0)

ceph.spec.in: BuildRequires gcc-toolset-11-libatomic-devel for aarch64

to address following failure when generating the building system
using CMake:

```
-- Performing Test HAVE_LIBATOMIC
-- Performing Test HAVE_LIBATOMIC - Failed
CMake Error at cmake/modules/CheckCxxAtomic.cmake:66 (message):
  Host compiler /opt/rh/gcc-toolset-11/root/usr/bin/c++ requires libatomic,
but it is not found
```

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 3811e8835d5dadd006f67f53ea50386a6f98f407)

ceph.spec.in: call %enable_devtoolset11 on centos8

before this change %enable_devtoolset11 is called only when building
with crimson on centos8.

after this change %enable_devtoolset11 is called when building on
centos8. because we've started using gcc-toolset-11 for building
rpm packages on centos8 after the C++20 migration. so, to build
with gcc-11, we need to enable it.

also, because gcc-toolset-11 is used, we have to disable
annotated_build.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 336fe86af86c6e7becd8c3f9e8c88db4f22fbe02)

ceph.spec.in: set BuildRequires: gcc-c++ >= 11 for SUSE distros

This is much simpler than explicitly requiring gcc 11, and
anyway, openSUSE Tumbleweed is up to gcc 12 now.

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 80949babab4aac86647ce9973f6b44adb92fae95)

ceph.spec.in: add gcc-toolset-11-annobin-plugin-gcc

gcc-toolset-11-annobin is already installed, but ceph.spec.in adds
"-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1" which needs the gcc
plugin too

resolves this failure during the cmake configure step:

-- Check for working CXX compiler: /opt/rh/gcc-toolset-11/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/gcc-toolset-11/root/usr/bin/c++ - broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCXXCompiler.cmake:59 (message):
  The C++ compiler

    "/opt/rh/gcc-toolset-11/root/usr/bin/c++"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11497-gfae772b8/rpm/el8/BUILD/ceph-17.0.0-11497-gfae772b8/x86_64-redhat-linux-gnu/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_09421/fast && /usr/bin/gmake  -f CMakeFiles/cmTC_09421.dir/build.make CMakeFiles/cmTC_09421.dir/build
    gmake[1]: Entering directory '/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11497-gfae772b8/rpm/el8/BUILD/ceph-17.0.0-11497-gfae772b8/x86_64-redhat-linux-gnu/CMakeFiles/CMakeTmp'
    Building CXX object CMakeFiles/cmTC_09421.dir/testCXXCompiler.cxx.o
    /opt/rh/gcc-toolset-11/root/usr/bin/c++   -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -o CMakeFiles/cmTC_09421.dir/testCXXCompiler.cxx.o -c /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11497-gfae772b8/rpm/el8/BUILD/ceph-17.0.0-11497-gfae772b8/x86_64-redhat-linux-gnu/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
    cc1plus: fatal error: inaccessible plugin file plugin/annobin.so expanded from short plugin name annobin: No such file or directory
    compilation terminated.

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 46a1aa48748e0ee7d41a74cad1bc1e611524fe02)

rpm: BuildRequires gcc-toolset-10-gcc-c++ >= 10.2

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 610c0d51110e139237a412f9c07d11378d503bee)

ceph.spec.in: disable WITH_JAEGER if "with seastar"

this is an intermediate solution for building crimson rpm packages.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 40039e7b0050ecbb1679866ccca644a455e6ad2a)

ceph.spec.in: enable WITH_SEASTAR if "with seastar"

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit be0b9534db80c96412f3c65f7bd91464c727b82c)

ceph.spec.in: use %enable_devtoolset11 to enable GTS-11

%enable_devtoolset11 redefines %___build_pre by appending
`source scl_source enable gcc-toolset-11` to it. `___build_pre` should
be able to populate this setting to both %build and %install. and hence
address the FTBFS where we need to use the tool chain from GTS-11.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 49459d92fec553246b35719e9defc0ad7582f701)
(cherry picked from commit 89a6a2d80236db0cb2ef54325ebc483c5491da25)

ceph.spec.in: move and undef _annotated_build only for rhels.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit ddf3721a9649b9b29593f17e663a1f0bf20afc19)

ceph.spec.in: enable toolset 11 also in install

This can be surprising but we actually compile things during
the `install` stage of `rpm-build`. The example is the pybind's
`setup.py` which builds `rados_dummy.c`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit f05b671798a54dce0e4aa340f76b31f401061a27)

ceph.spec.in: use gcc-toolset-11 for building crimson

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 13f3653f59f61c1f7e90c05e37e43e0ef912d1ea)

ceph.spec.in: use gcc-toolset-10 for building crimson

This commit bumps up the toolset version but only to build crimson.
That is, the classical OSD stays unaffected.

The reason behind the upgrade is the following FTBFS:

```
[ 32%] Building CXX object src/seastar/CMakeFiles/seastar.dir/src/core/reactor.cc.o
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/src/core/reactor.cc: In constructor ‘seastar::reactor::reactor(std::shared_ptr<seastar::smp>, seastar::alien::instance&, unsigned int, seastar::reactor_backend_selector, seastar::reactor_config)’:
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/src/core/reactor.cc:926:90: error: use of deleted function ‘seastar::condition_variable::condition_variable()’
  926 |     , _thread_pool(std::make_unique<thread_pool>(this, seastar::format("syscall-{}", id))) {
      |                                                                                          ^
In file included from /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/include/seastar/core/reactor.hh:74,
                 from /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/src/core/reactor.cc:32:
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11345-ga3bb1485/rpm/el8/BUILD/ceph-17.0.0-11345-ga3bb1485/src/seastar/include/seastar/core/condition-variable.hh:157:5: note: ‘seastar::condition_variable::condition_variable() noexcept’ is implicitly deleted because its exception-specification does not match the implicit exception-specification ‘’
  157 |     condition_variable() noexcept = default;
```

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 30f75143171d5ed1ca94a1a56ce0556db17735b6)

Merge pull request #46580 from cbodley/wip-55968

quincy: rgw: check object storage_class when check_disk_state

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #47368 from kotreshhr/wip-56979-quincy

quincy: mgr/volumes: Fix subvolume creation in FIPS enabled system.

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #47251 from idryomov/wip-56491-quincy

quincy: ceph.spec.in: disable system_pmdk on s390x

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47210 from kamoltat/wip-ksirivad-backport-quincy-fix-56151

quincy: src/mgr/DaemonServer.cc: fix typo in output gap >= max_pg_num_change

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47086 from kamoltat/wip-ksirivad-quincy-backport-46029

quincy: mon/Elector: notify_rank_removed erase rank from both live_pinging and dead_pinging sets for highest ranked MON

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47020 from sseshasa/wip-56498-quincy

quincy: osd: Set initial mClock QoS params at CONF_DEFAULT level

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46623 from idryomov/wip-pmdk-wno-error-quincy

quincy: cmake: pass -Wno-error when building PMDK

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #46155 from tserong/wip-55552-quincy

quincy: ceph.spec.in: openSUSE: require gcc11-c++, disable parquet

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #47445 from kamoltat/wip-ksirivad-backport-quincy-47138

quincy: qa/tasks/ceph_manager.py: increase test_pool_min_size timeout

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47490 from sseshasa/wip-57052-quincy

quincy: osd: Handle oncommits and wait for future work items from mClock queue

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #47483 from tchaikov/quincy-pr-47478

quincy: cmake: remove spaces in macro used for compiling cython code

Reviewed-by: Tim Serong <tserong@suse.com>

osd: Handle oncommits and wait for future work items from mClock queue

When a worker thread with the smallest thread index waits for future work
items from the mClock queue, oncommit callbacks are called. But after the
callback, the thread has to continue waiting instead of returning back to
the ShardedThreadPool::shardedthreadpool_worker() loop. Returning results
in the threads with the smallest index across all shards to busy loop
causing very high CPU utilization.

The fix involves reacquiring the shard_lock and waiting on sdata_cond
until notified or until time period lapses. After this, the smallest
thread index repopulates the oncommit queue from the context_queue
if there were any additions.

Fixes: https://tracker.ceph.com/issues/56530
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 180a5a7bffd4d96c472cc39447717958dd51bbd9)

cmake: remove spaces in macro used for compiling cython code

we are facing following FTBFS on jammy + GCC-11.2 + Cython 0.29 +
CMake 3.22:

creating /home/jenkins-build/build/workspace/ceph-api/build/lib/cython_modules/temp.linux-x86_64-3.10/home/jenkins-build/build/workspace/ceph-api/build/src/pybind/cephfs
compile options: '-I/usr/include/python3.10 -I/usr/include/python3.10 -c'
extra options: '-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -iquote/home/jenkins-build/build/workspace/ceph-api/src/include -w -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2'
cc: /home/jenkins-build/build/workspace/ceph-api/build/src/pybind/cephfs/cephfs.c
cc: warning: ##: linker input file unused because linking not done
cc: error: ##: linker input file not found: No such file or directory
cc: warning: 0: linker input file unused because linking not done
cc: error: 0: linker input file not found: No such file or directory

it seems cython is not able to escape the space in the "extra options"
anymore, so the "##" and "0" are considered as object files passed to
compiler in addition to cephfs.c.

in this change the spaces are removed to help cython to make the right
decision.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 5824fed5b427f1d055fb7104fea2e68cd36e6844)

Merge pull request #47342 from cbodley/wip-56955

quincy: rgw: better tenant id from the uri on anonymous access

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47289 from yaarith/wip-56486-quincy

quincy: mgr/telemetry: reset health warning after re-opting-in

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #47201 from aaSharma14/wip-56647-quincy

quincy: mgr/dashboard:Get different storage class metrics in Prometheus dashboard

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47435 from guits/quincy-bp-pr47434

quincy: backport: ceph-volume/tests: fix test_exception_returns_default

Merge pull request #47329 from rhcs-dashboard/wip-56947-quincy

quincy: mgr/dashboard: added pattern validaton for form input

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #47430 from rhcs-dashboard/wip-57009-quincy

quincy: mgr/dashboard: remove token logging

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47442 from ceph/quincy-rtd

quincy: .readthedocs.yml: Always build latest doc/releases pages

Merge pull request #47268 from adk3798/quincy-osd-config

quincy: mgr/cephadm: add parsing for config on osd specs

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

qa/tasks/ceph_manager.py: Increase timeout

In test_pool_min_size():

1. Provided buffer time before we check
for recovery in ceph_manager.wait_for_recovery()

2. Increased timeout in ceph_manager.wait_for_clean()

3. Increased sleep time for
ceph_manager.all_active_or_peered()

Fixes:
https://tracker.ceph.com/issues/49777
https://tracker.ceph.com/issues/54511
https://tracker.ceph.com/issues/51904

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit ed73288102f4aa67da3485c70b2409b9eca1873b)

qa/tasks/ceph_manager.py: improve loggings

1. When `test_pool_min_size` hit the case where
`not all PGs are active or peered` we dump
each PG state that doesn't have active or
peered state

2. Improve logs message in `inject_pause()`.

3. Add logs for the `test_map_discontinuity()`.

4. In the function, `choose_action()`,
added more logs regarding `chance_down`.

5. Added more loggings to
`primary_affinity()`,
`thrash_pg_upmap_items()`,
`thrash_pg_upmap()`.

6. Make self.is_clean() dump the pgs that
are not active+clean.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 9ed53c82cf5d56bd4555084d3bf1c3c9948c4fe3)

.readthedocs.yml: Always build latest doc/releases pages

We don't backport PRs merged into doc/releases.  Therefore, when one browses to an older Ceph release version on docs.ceph.com (e.g., https://docs.ceph.com/en/pacific/), the information is out of date at best.

The doc/releases page is only accurate if browsing https://docs.ceph.com/en/latest/, for example.

So this post_checkout command will make sure we've checked out doc/releases from main before building and publishing.

Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit 055fe1f825b0629b7685d6d3d4d629ffc37a2d7c)

mgr/cephadm: add parsing for config on osd specs

Cephadm, while parsing spec files, can parse ceph configuration
for almost all the services, except for OSDs, where it fails
with a nasty "unexpected keyword argument config".

This commit fixes this issue.

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>
(cherry picked from commit 74273e7e6e79c474e33fc41480399dc27ad2ae64)

Merge pull request #47311 from adk3798/wip-56744-quincy

quincy: mgr/cephadm: Add disk rescan feature to the orchestrator

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47312 from adk3798/wip-56742-quincy

quincy: mgr/cephadm: support for miscellaneous config files for daemons

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47316 from adk3798/wip-56611-quincy

quincy: mgr/cephadm: support for os tuning profiles

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47313 from adk3798/wip-56741-quincy

quincy: cephadm: reduce spam to cephadm.log

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47314 from adk3798/wip-56740-quincy

quincy: cephadm: add "su root root" to cephadm.log logrotate config

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47315 from adk3798/wip-56738-quincy

quincy: qa/workunits/cephadm: update test_repos master -> main

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47317 from adk3798/wip-56584-quincy

quincy: mgr/cephadm: check for events key before accessing it

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47373 from adk3798/wip-56986-quincy

quincy: mgr/cephadm: clear error message when resuming upgrade

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #47374 from adk3798/wip-56983-quincy

quincy: cephadm: Fix repo_gpgkey should return 2 vars

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #46467 from mdw-at-linuxbox/mdw-quincy-mankey

quincy -- sse s3 changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume/tests: fix test_exception_returns_default

There are cases, like running tests as root user, where this test fails
because root user can always read the file content.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 47aec3b86bcc850292f66d24670c5d8dc0a33c85)

qa: Fix use of the 'sudo' args

https://tracker.ceph.com/issues/56727
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 5a33f7e6f6c1fbaa5078983ca2f0901296b65588)

mgr/volumes: Fix subvolume creation in FIPS enabled system.

The md5 checksum is used in the construction of legacy
subvolume config filename. It's not used for security reason.
Hence marking the 'usedforsecurity' flag to false to
make it FIPs compliant.

The usage of md5 was always in there. The commit 373a04cf734
made it to get exercised in 'open_subvol' which is pre-requisite
for all the subvolume operations and hence subvolume
creation has failed.

Fixes: https://tracker.ceph.com/issues/56727
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit ced3fac48d3da2320827c6c86ece3b87953badc7)

Merge pull request #47356 from rhcs-dashboard/wip-56965-quincy

quincy: mgr/dashboard: add required validation for frontend and monitor port

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: remove token logging

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit fdaf909fba0c399b8d9bbc3a29a5af871546d9d9)

Merge pull request #47403 from aaSharma14/wip-57003-quincy

quincy: mgr/dashboard: Show error on creating service with duplicate service id

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #47195 from adamemerson/wip-54208-quincy

quincy: rgw: Fix data race in ChangeStatus

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47191 from adamemerson/wip-56585-quincy

quincy: rgw: Guard against malformed bucket URLs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #47415 from zdover23/wip-doc-2022-08-03-backport-47197-to-quincy

quincy: doc/dev: Elaborate on boost .deb creation

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: Elaborate on boost .deb creation

Signed-off-by: David Galloway <dgallowa@redhat.com>
(cherry picked from commit 3222485d6582ba6ab3ee908c13b6459c2e2b6302)

Merge pull request #47149 from guits/wip-56496-quincy

quincy: ceph-volume: fix `simple scan`

Merge pull request #47406 from guits/quincy-cv-backports

ceph-volume: Quincy backports

ceph-volume: fix is_ceph_disk_member()

`dev['NAME']` can't match `part` given that it's the name of the
parent device being compared to the partition name.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 24899a1f8d77bc3c10ef158bb463349d615f7c57)

ceph-volume: filter out non-existing devices

[1] has broken teuthology tests.

```
[root@smithi097 /]# ls -l /sys/block/
total 0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 Jul 25 14:12 nvme0n1 -> ../devices/pci0000:00/0000:00:02.0/0000:02:00.0/nvme/nvme0/nvme0n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1c1n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme1/nvme1c1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2c2n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme2/nvme2c2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys2/nvme2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3c3n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme3/nvme3c3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys3/nvme3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4c4n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme4/nvme4c4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys4/nvme4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:12 sda -> ../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda
[root@smithi097 /]#
```

devices like `nvme1c1n1` exist in `/sys/block` but aren't present in `/dev`

```
[root@smithi097 /]# ls -l /dev/nvme*
crw------- 1 root root  10, 122 Jul 25 14:16 /dev/nvme-fabrics
crw------- 1 root root 240,   0 Jul 25 14:12 /dev/nvme0
brw-rw---- 1 root disk 259,   0 Jul 25 14:15 /dev/nvme0n1
crw------- 1 root root 240,   1 Jul 25 14:16 /dev/nvme1
brw-rw---- 1 root disk 259,   2 Jul 25 15:24 /dev/nvme1n1
crw------- 1 root root 240,   2 Jul 25 14:16 /dev/nvme2
brw-rw---- 1 root disk 259,   4 Jul 25 15:24 /dev/nvme2n1
crw------- 1 root root 240,   3 Jul 25 14:16 /dev/nvme3
brw-rw---- 1 root disk 259,   6 Jul 25 15:24 /dev/nvme3n1
crw------- 1 root root 240,   4 Jul 25 14:16 /dev/nvme4
brw-rw---- 1 root disk 259,   8 Jul 25 15:24 /dev/nvme4n1
[root@smithi097 /]#
```

If a device isn't actually present in `/dev`, `get_block_devs_sysfs()` shouldn't return it.

[1] https://github.com/ceph/ceph/commit/c7f017b21ade3762ba5b7b9688bed72c6b6

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 0ee8351e4d564724aa68a2b8d03bc53e281e7fc9)

ceph-volume: drop `ceph-bluestore-tool` call in Device._parse()

disk.has_bluestore_label() does the same thing without a subprocess call.

Fixes: https://tracker.ceph.com/issues/56622
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f5f3117510c287d817534c9f7c25af5f208c2f56)

ceph-volume: fix simple scan

When the class `Device` is instantiated with a path instead of a
block device, it fails like following.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 130, in __init__
    self._parse()
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 233, in _parse
    self.ceph_device = disk.has_bluestore_label(self.path)
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 906, in has_bluestore_label
    with open(device_path, "rb") as fd:
IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/osd/ceph-0/'
```

passing a path instead of a block device is valid, `simple scan` needs it.

Fixes: https://tracker.ceph.com/issues/56969
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5a10e3e9e865aafd9d3f90beb186777060d17f08)

ceph-volume/tests: migrate to pyfakefs

ceph-volume unit tests shouldn't actually create contents on the
filesystem from where it runs (even though they are written in a tmp
dir), let's use pyfakefs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1274ba34b4d21c5abf2cd2da035d6d25ae8d03da)

ceph-volume/tests: fix test_path_is_valid()

When ceph-volume tests are run from a host where there's no `/dev/sda`
device, this test fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7ab9d99420db2995542dfab7c416021dcc1c4a47)

ceph-volume: report slave devices in inventory

`ceph-volume inventory` currently reports the following:

```

Device Path               Size        rotates available Model name
/dev/mapper/mpatha        50.00 GB    True    True
/dev/mapper/mpathaa       6.00 GB     True    True
/dev/mapper/mpathab       6.00 GB     True    True
/dev/mapper/mpathac       6.00 GB     True    True
/dev/mapper/mpathad       6.00 GB     True    True
/dev/mapper/mpathae       6.00 GB     True    True
```

whereas something like following would be useful:

```

Device Path               Size         Device nodes    rotates available Model name
/dev/mapper/mpatha        50.00 GB     sdf,sde         True    True
/dev/mapper/mpathaa       6.00 GB      sdbe,sdat       True    True
/dev/mapper/mpathab       6.00 GB      sdav,sdbf       True    True
/dev/mapper/mpathac       6.00 GB      sdbb,sdbl       True    True
/dev/mapper/mpathad       6.00 GB      sdas,sdbc       True    True
/dev/mapper/mpathae       6.00 GB      sdax,sdbh       True    True
```

Fixes: https://tracker.ceph.com/issues/56624
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1ac2a8270cf4aece58ae3f952cc4d1ef978384ed)

ceph-volume: support symlinks as devices

This makes ceph-volume support passing symlinks as devices.

Fixes: https://tracker.ceph.com/issues/49103
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 9ddc5b77372cd2e0d5badd433c93d147cb25c6ee)

ceph-volume: drop self.abspath in Device()

seems useless to have both self.path and self.abspath attributes.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 01112ba6894834287092d3b672ec548cb3a4d676)

ceph-volume: improve mpath devices reporting

An environment with mpath devices looks like following:

`lsblk` output:

```
sdy        65:128  0    6G  0 disk
`-mpathm  252:8    0    6G  0 mpath
sdz        65:144  0    6G  0 disk
`-mpathm  252:8    0    6G  0 mpath
```

`multipath -ll` output:

```
mpathm (3600140575bbe2d3c0c6493fb6e6ed84c) dm-8 LIO-ORG,TCMU device
size=6.0G features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 2:0:0:30 sdz  65:144 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 3:0:0:30 sdy  65:128 active ready running
```

`ceph-volume inventory` output:
```
Device Path               Size         rotates available Model name
/dev/mapper/mpathm        6.00 GB      True    True
/dev/sdy                  6.00 GB      True    False     TCMU device
/dev/sdz                  6.00 GB      True    False     TCMU device
```

ceph-volume shouldn't report devices `/dev/sdy` and `/dev/sdz`, they will never be
available in such a scenario. Considering this, in a host with a bunch of mpath devices
it will pollute the inventory output.

Fixes: https://tracker.ceph.com/issues/56621
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e892f02a1a9126cbb520c90aeef0afb3590ddbbb)

ceph-volume: Remove refs to get_block_devs_lsblk

It's been replaced by get_block_devs_sysfs

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 2023e71f695c1962534c6012d5b331bebd10ec31)

ceph-volume: filter RBD devices from the device inventory

Avoid running `blkid` or deploying OSDs on RBD devices by ensuring they
do not appear in the `ceph-volume inventory`

Fixes: https://tracker.ceph.com/issues/53846
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 47325ec3ec5ce1d53c5eae2952f631e95b7135fe)

ceph-volume: Rename env var; add warning

So that we can have a nice big warning that fires once per invocation, I
think using a callable class with a class attribute seems like a decent
approach. A closure could work too.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 69f58f51a2d7967d597a8d61c0ab20b52e8dc374)

ceph-volume: Optionally consume loop devices

A similar proposal was rejected in #24765; I understand the logic
behind the rejection, but this will allow us to run Ceph clusters on
machines that lack disk resources for testing purposes. We just need to
make it impossible to accidentally enable, and make it clear it is
unsupported.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit c7f017b21ade3762ba5b7b9688bed72c6b60dc0e)

Merge pull request #47293 from MrFreezeex/wip-56630-quincy

quincy: ceph-volume: fix fast device alloc size on mulitple device

Merge pull request #47348 from guits/wip-56711-quincy

quincy: ceph-volume: do not call get_device_vgs() per devices

mgr/dashboard: Show error on creating service with duplicate service id

Fixes: https://tracker.ceph.com/issues/56689
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 07cfd44193f6ebd552d13325858e8b5b5c131bfb)

Merge pull request #47248 from idryomov/wip-56676-quincy

quincy: librbd: tweak misleading "image is still primary" error message

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #47162 from idryomov/wip-56549-quincy

quincy: librbd: bail from schedule_request_lock() if already lock owner

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47159 from idryomov/wip-48038-quincy

quincy: qa/suites/rbd: disable workunit timeout for dynamic_features_no_cache

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #47144 from idryomov/wip-56561-quincy

quincy: rbd: don't default empty pool name unless namespace is specified

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47129 from idryomov/wip-50734-quincy

quincy: qa/suites/rbd/pwl-cache: ensure recovery is actually tested

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #47126 from idryomov/wip-56516-quincy

quincy: rbd-mirror: remove bogus completed_non_primary_snapshots_exist check

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #47297 from aclamk/wip-56668-quincy

quincy: os/bluestore: Fix collision between BlueFS and BlueStore deferred writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: add required validation for frontend and monitor port

Fixes: https://tracker.ceph.com/issues/56688
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit ea770fd858cae9a8b52c60b293f48cc4dbc925f8)

Merge pull request #47344 from ceph/quincy-release

v17.2.3

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>