Merge pull request #54251 from ajarr/wip-62994

mgr/rbd_support: fix recursive locking on CreateSnapshotRequests lock

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47730 from ifed01/wip-ifed-alloc-logging

os/bluestore: assert on improper release in AvlAllocator

Reviewed-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #54281 from zdover23/wip-doc-2023-11-01-cephadm-troubleshooting-1-of-x

doc/cephadm: edit troubleshooting.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54262 from barakda/nvmeof-bump-version-v0.0.5

Update default nvmeof gw version to v0.0.5

Merge pull request #54210 from rishabh-d-dave/progress-submodule-update

add --progress flag to git submodule update commands

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephadm: edit troubleshooting.rst (1 of x)

Edit doc/cephadm/troubleshooting.rst. This commit and the PR of which it
is a part was raised in response to
https://github.com/ceph/ceph/pull/53976. The limits of reStructuredText
are particularly visible here in every instance of a BASH for-loop and
in every instance of a command stretched over multiple lines.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54226 from adamemerson/wip-build-experiment

build: Fix ARM build and make install-deps.sh more robust

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54216 from adamemerson/wip-fmt-felicity

build: fmt felicity

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: add --progress flag to git submodule update commands

Ceph has lots of submodules that needs to be cloned before building
binaries from the repository. Seeing the progress when these submodules
are being cloned is useful, especially when developers/users have a
network issue or a slow network.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #54272 into main

* refs/pull/54272/head:
cmake: do not issue CMAKE_BUILD_TYPE warning if set

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53720 from ljflores/bz_2241104

osd: fix logic in check_pg_upmaps

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: do not issue CMAKE_BUILD_TYPE warning if set

Fixes: b8c6ba01b262d33084b9c607d829503ac8ae25c5
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54238 from zdover23/wip-doc-2023-10-30-rados-config-osd-config-ref-scrubbing

doc/rados: improve "scrubbing" explanation

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Update default nvmeof gw version to v0.0.5

Signed-off-by: barakda <barak.davidov@gmail.com>

Merge pull request #50821 from cbodley/wip-cmake-uring

cmake: fix WITH_SYSTEM_LIBURING and make uring available outside of src/blk

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: improve "scrubbing" explanation

Improve the English describing the operation of scrubbing PGs in
doc/rados/configuration/osd-config-ref.rst.

Co-authored-by: Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54239 from zdover23/wip-doc-2023-10-30-glossary-bluestore

doc/glossary: improve "BlueStore" entry

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve "BlueStore" entry

Initially s/backend/back end/ but then I added a little more information
about BlueStore's use of RocksDB to map object names to block locations
on disk.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54247 from rhcs-dashboard/fix-no-multisite-error

mgr/dashboard: fix rgw multisite error when no rgw entity is present

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54240 from rhcs-dashboard/fix-63347-main

mgr/dashboard: Block Ui fails in angular with target es2022

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54142 from YiteGu/add-bluestore-fragmentation-to-metrics

os/bluestore: add bluestore fragmentation micros to prometheus

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #53993 from shminjs/fix-out-and-down-osd-daemon-report

mgr: remove out&down osd from mgr daemons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53689 from leonid-s-usov/mds-raise

common/admin_socket: add a command to raise a signal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53563 from ifed01/wip-ifed-pg-osd-stats

osd: introduce dump_osd_pg_stats admin socket command.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54177 from ronen-fr/wip-rf-map-msg

osd: fix use-after-move in build_incremental_map_msg()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #54198 from ceph/wip-yuriw-release-process-main

docs: revised Distro Codemap, removed `octopus`, added `reef`

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #54252 from rhcs-dashboard/quincy-release-notes

doc: Release notes 17.2.7 with updated dashboard info

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

doc: Release notes 17.2.7 with updated dashboard info

Signed-off-by: Nizamudeen A <nia@redhat.com>

build: Fix `WITH_FMT_HEADER_ONLY`

Previously it was still linking in the submodule's static libraries
and ldd showed it linking to the system fmt's shared libraries.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

mgr/rbd_support: remove CreateSnapshotRequests __del__()

There is no need for CreateSnapshotRequests.__del__() that calls
CreateSnapshotRequests.wait_for_pending().
MirrorSnapshotScheduleHandler.shutdown() already calls
CreateSnapshotRequests.wait_for_pending().

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/rbd_support: fix recursive locking on CreateSnapshotRequests lock

The MirrorSnapshotScheduleHandler's run thread issues asynchronous
create snapshot requests using a CreateSnapshotRequests instance. When
the thread invokes a CreateSnapshotRequests instance's get_ioctx(),
the instance's class variable lock is acquired. With the class
variable lock held, the garbage collection of a CreateSnapshotRequests
instance may race in the thread. The thread would then call
CreateSnapshotRequests __del__() that tries to acquire the class
variable lock that the thread already holds. Fix this
recursive deadlock by converting the CreateSnapshotRequests lock from
a class variable to an instance variable. There is no need to share
the lock across CreateSnapshotRequests instances.

Also convert MirrorSnapshotScheduleHandler, PerfHandler and
TrashPurgeScheduleHandler class variables to instance variables
that don't need to be shared across the instances.

Fixes: https://tracker.ceph.com/issues/62994
Signed-off-by: Ramana Raja <rraja@redhat.com>
Co-Authored-By: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54246 from ceph/epuertat-patch-1

create-backport-trackers: pin to Python <3.12

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Update default nvmeof gw version to v0.0.5

Signed-off-by: barakda <barak.davidov@gmail.com>

mgr/dashboard: fix rgw multisite error when no rgw entity is present

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #54241 from rhcs-dashboard/fix-63348-main

mgr/dashboard: update rgw multisite import form helper info

Reviewed-by: Nizamudeen A <nia@redhat.com>

build: Add `WITH_SYSTEM_FMT` option

The `WITH_SYSTEM_FMT` option builds against system fmt and will fail
if a suitable version is not found. It is defaulted to `OFF`.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54153 from ceph/wip-yuriw-release-17.2.7-main

doc: 17.2.7 Release Notes

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

build: Fix fmt version check

Currently, when attempting to build ceph on a system with fmt
installed, we try to build against it whatever the version. This
constantly breaks people's builds, since newer versions of fmt often
change the API.

This change specifies that versions must be below 10 as well as at or
above 8.1.1, so that on systems with a new format, we fall back to
using the submodule.

It also removes the `Findfmt.cmake` module, as that does not check
the installed version. Instead, we use the cmake config file installed by
the system package of fmt and does support version checking.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54151 from rkachach/fix_issue_63291

mgr/rook: fix the namespaces used in rook and removes hard-coded references

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@ibm.com>

create-backport-trackers: Python >= 3.12 deprecated distutils

See https://docs.python.org/3.10/whatsnew/3.10.html#distutils-deprecated.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #50593 from pkalever/fix-feature-disable

rbd-nbd: fix stuck with disable request

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: update rgw multisite import form helper info

Change 'To obtain the token, generate it from your secondary Ceph cluster' to 'To obtain the token, generate it from your primary Ceph cluster' in rgw multisite import form helper

Fixes: https://tracker.ceph.com/issues/63348
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Block Ui fails in angular with target es2022

The BlockUi element is failing in angular with target es2022..change it to es2020

Fixes: https://tracker.ceph.com/issues/63347
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #54232 from zdover23/wip-doc-2023-10-29-ceph-volume-lvm-activate-2

doc/ceph-volume: improve front matter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54231 from zdover23/wip-doc-2023-10-29-ceph-volume-lvm-activate

doc/ceph-volume: explain idempotence

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/ceph-volume: improve front matter

Simplify and improve the readability of the front matter of
doc/ceph-volume/lvm/activate.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53848 from kchheda3/wip-topic-policy

rgw/notification: Add support to set topic policy while topic creation

reviewed-by: yuvalif, anthonyeleven

Merge pull request #54230 from zdover23/wip-doc-2023-10-29-install-cephadm

doc/install: simplify cephadm docs

doc/ceph-volume: explain idempotence

Improve the grammar of a note explaining that the execution of the call
that activates ceph-volume is idempotent.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/install: simplify cephadm docs

Simplify the explanation of cephadm.

Signed-off-by: Zac Dover <zac.dover@proton.me>

build: Don't kill install-deps.sh if we can't get ceph-libboost

We can still use the submodule.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54082 from pdvian/wip-qa-suites-fix

qa/smoke,orch,perf-basic: add POOL_APP_NOT_ENABLED to ignorelist

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge branch 'main' into wip-topic-policy

Merge pull request #53003 from alimaredia/wip-rgw-perf-counters-cache

Perf counters cache + Labeled RGW Op counters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54218 from zdover23/wip-doc-2023-10-27-rados-troubleshooting-troubleshooting-pg

doc/rados: edit troubleshooting-pg.rst

Merge pull request #54219 from zdover23/wip-doc-2023-10-27-rados-ops-pools-cache-tiering-removal

doc/rados: remove cache-tiering-related keys

Merge pull request #53976 from phlogistonjohn/jjm-cephadm-gdb-extd-doc

cephadm: document extended gdb debugging techniques

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/rados: remove cache-tiering-related keys

Remove information related to cache-tiering-related keys from
doc/rados/operations/pools.rst. Cache-tiering is deprecated in Reef.
This PR is suitable for backporting to the Reef release branch, but not
to release branches prior to Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/librbd/fsx: wait for resize to propagate in krbd_resize()

With this changes resize request will not be blocked until the resize is
completed. Because of this the fsx test fails as it assumes that the
request to resize immediately implies changes on the device size.

Hence we have to add a wait in resize handler of fsx for the device to
actually get resized.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

doc/rados: edit troubleshooting-pg.rst

s/placement group/pool/ in a sentence that, prior to this change, was
confusing. Suitable for backports prior to Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54200 from zdover23/wip-doc-2023-10-26-hitset-remove-for-reef-and-later

doc/rados: remove HitSet-related key information

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados: remove HitSet-related key information

Remove HitSet-related key information from
doc/rados/operations/pools.rst. HitSet-related keys are relevant only to
releases of Ceph that support cache tiering. Only Quincy and earlier
(inclusive) releases of Ceph support cache tiering. Backport this commit
from main to Reef, but not to Quincy or to release branches earlier than
Quincy.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: 17.2.7 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53909 from aclamk/aclamk-fix-63121

os/kv_test: Fix estimate functions

docs: revised Distro Codemap, removed `octopus`, added `reef`

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #54155 from jrchyang/fix-divide-zero

os/bluestore: fix crash caused by dividing by 0

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #53807 from rhcs-dashboard/fix-63088-main

mgr/dashboard: Consider null values as zero in grafana panels

Reviewed-by: Nizamudeen A <nia@redhat.com>

osd: fix use-after-move in build_incremental_map_msg()

Fixes: https://tracker.ceph.com/issues/63310
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

doc/cephadm: extend container debugging to cover live processes

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/cephadm: clean up debugging section

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a --dry-run option to cephadm shell

Instead of creating the shell, the --dry-run option prints the container
command that would be used. This can be used as a starting point for
creating custom container commands similar to what cephadm shell would
generate but with tweaks.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/rook: fixing the namespaces used to access some resources/config

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/rook: remove all the hardcoded refs to the 'rook-ceph' namespace
Fixes: https://tracker.ceph.com/issues/63291
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #54148 from ivoalmeida/fix-edit-user-bucket-quota

mgr/dashboard: fixed edit user quota form error

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

os/bluestore: fix crash caused by dividing by 0

In Allocator::create() => ZonedAllocator::ZonedAllocator()
num_zones is calculated by size / zone_size,
causing a crash when zone_size is set to 0

Signed-off-by: Jrchyang Yu <yuzhiqiang_yewu@cmss.chinamobile.com>

rgw/notification: Add support to set topic policy while topic creation

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

mgr/dashboard: fixed edit user quota form error

Fixes: https://tracker.ceph.com/issues/63287
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge PR #54030 into main

* refs/pull/54030/head:
client: always refresh mds feature bits on session open

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54025 from yuvalif/wip-yuval-fix-63205

rgw/notifications/test: fix rabbitmq and kafka issues in centos9

reviewed-by: cbodley

rgw/notifications/test: install java for kafka build

also use kafka binaries insted of building from source

Fixes: https://tracker.ceph.com/issues/63205
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54136 from yuvalif/wip-yuval-remove-pubsub-doc

rgw/doc: pubsub capabilities reference was removed from docs

reviewed-by: @zdover23

doc/rgw: pubsub capabilities reference was removed from docs

most pubsub docs were removed in
0fecbe2476afbb06de7ea731b7c92066ae40cfc4
however, the text removed described functionality that is only
supported bia pubsub

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54107 from anthonyeleven/anthonyeleven/clarify-checklist

.github: Clarify checklist details

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49954 from caffeinism/fix-bug-in-ceph-volume-device

ceph-volume: fix a bug in _check_generic_reject_reasons

rgw: remove move's, rename put/get counters, revert exporter changes

Signed-off-by: Ali Maredia <amaredia@redhat.com>

client: always refresh mds feature bits on session open

Fixes: http://tracker.ceph.com/issues/63188
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54109 from ceph/wip-yuri-crontab-main

qa/tests: enabled smoke tests on main, quincy and reef

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53923 from adk3798/test-mgr-cephadm-image-base

mgr/cephadm: add unit test to for upgrade check with --ceph-version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

rgw/admin: remove unused asio.hpp include

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/librados: include io_context.hpp and rename io_service

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use granular asio includes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

immutable-object-cache: use granular asio headers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

exporter: use granular asio headers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: Graylog uses granular asio headers

and use the updated class name io_context

Signed-off-by: Casey Bodley <cbodley@redhat.com>

neorados: use granular includes instead of boost/asio.hpp

avoid the boost/asio.hpp convenience header, as it pulls in a dependency
on liburing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: include posix/stream_descriptor.hpp header

the use of boost::asio::posix::stream_descriptor requires the
concrete header instead of posix/basic_stream_descriptor.hpp

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: hide asio header from common/error_code.h

this added a dependency on uring headers where it wasn't necessary.
pick_address.cc was relying on that for the IFF_UP define

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: define BOOST_ASIO_HAS_IO_URING for WITH_LIBURING

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: fix Finduring.cmake

liburing-devel package provides `/usr/lib64/liburing.so`. without this
fix, `find_package(uring)` failed with this `CMAKE_FIND_DEBUG_MODE`
output:
```
  find_library considered the following locations:

    /home/cbodley/ceph/build/virtualenv/bin//(lib)liburing.a(\.so|\.a)
    /home/cbodley/ceph/build/virtualenv/bin/(lib)liburing.a(\.so|\.a)
    /usr/local/bin//(lib)liburing.a(\.so|\.a)
    /usr/local/bin/(lib)liburing.a(\.so|\.a)
    /usr/local/sbin//(lib)liburing.a(\.so|\.a)
    /usr/local/sbin/(lib)liburing.a(\.so|\.a)
    /usr/bin//(lib)liburing.a(\.so|\.a)
    /usr/bin/(lib)liburing.a(\.so|\.a)
    /usr/sbin//(lib)liburing.a(\.so|\.a)
    /usr/sbin/(lib)liburing.a(\.so|\.a)
    /usr/local/lib64//(lib)liburing.a(\.so|\.a)
    /usr/local/lib64/(lib)liburing.a(\.so|\.a)
    /usr/local/lib//(lib)liburing.a(\.so|\.a)
    /usr/local/lib/(lib)liburing.a(\.so|\.a)
    /usr/local//(lib)liburing.a(\.so|\.a)
    /usr/local/(lib)liburing.a(\.so|\.a)
    /usr/lib64//(lib)liburing.a(\.so|\.a)
    /usr/lib64/(lib)liburing.a(\.so|\.a)
    /usr/lib//(lib)liburing.a(\.so|\.a)
    /usr/lib/(lib)liburing.a(\.so|\.a)
    /usr//(lib)liburing.a(\.so|\.a)
    /usr/(lib)liburing.a(\.so|\.a)
    /lib64//(lib)liburing.a(\.so|\.a)
    /lib64/(lib)liburing.a(\.so|\.a)
    /lib//(lib)liburing.a(\.so|\.a)
    /lib/(lib)liburing.a(\.so|\.a)
    /opt//(lib)liburing.a(\.so|\.a)
    /opt/(lib)liburing.a(\.so|\.a)
    /usr/lib64/X11//(lib)liburing.a(\.so|\.a)
    /usr/lib64/X11/(lib)liburing.a(\.so|\.a)
```

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #54042 into main

* refs/pull/54042/head:
cmake: populate liburing include and library paths down to rocksdb external project
cmake: promote uring package search to top-level

Reviewed-by: Casey Bodley <cbodley@redhat.com>