rgwlc: fix notification test imports

Found by Yuval Lifshitz review.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: send version_id with notifications

Also fix return value from handle_multipart_expiration bucket-entry
lambda.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: treat persistent notify reservation as failure as fatal

(Because that's consistent with what we have been doing for expiration.)

This is possibly a controversial choice, and it might be better to allow
setting a site policy, in future.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: fix remove expired for non-versioned case

Plus, reformat some adjacent code.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: split versioned and non-versioned transition notifications

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: constify notification event_type

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: remove extra handles to sal bucket and object

I missed the addition of those to lc_op_ctx.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: log error on failure to publish_commit

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: dispatch S3 notifications on transition and mpu abort

Fixes: https://tracker.ceph.com/issues/58641
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #52545 from Matan-B/wip-matanb-trim-61962

osd/OSD: fix trim_maps() - possible leak on `skip_maps`

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50081 from NitzanMordhai/wip-nitzan-subscription-manager-rocksdb-librados

qa/*/test_envlibrados_for_rocksdb.sh: subscribe repo

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #52171 from myoungwon/wip-ceph-dedup-tool-improve-print

tool/ceph-dedup-tool: improve the progress report

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53346 from linuxbox2/wip-build-fio

cmake: BuildFIO.cmake should not introduce -std=gnu++17

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54141 from phlogistonjohn/jjm-cephadm-more-assertions

cephadm: add more deployment test assertions

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54040 from Matan-B/wip-wip-matanb-crimson-do_osd_ops_execute-v3

crimson/osd/pg: do_osd_ops_execute refactor

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53535 from ajarr/wip-62891

qa/suites/rbd: add test to check rbd_support module recovery

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #54251 from ajarr/wip-62994

mgr/rbd_support: fix recursive locking on CreateSnapshotRequests lock

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #47730 from ifed01/wip-ifed-alloc-logging

os/bluestore: assert on improper release in AvlAllocator

Reviewed-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #54281 from zdover23/wip-doc-2023-11-01-cephadm-troubleshooting-1-of-x

doc/cephadm: edit troubleshooting.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54262 from barakda/nvmeof-bump-version-v0.0.5

Update default nvmeof gw version to v0.0.5

Merge pull request #54210 from rishabh-d-dave/progress-submodule-update

add --progress flag to git submodule update commands

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephadm: edit troubleshooting.rst (1 of x)

Edit doc/cephadm/troubleshooting.rst. This commit and the PR of which it
is a part was raised in response to
https://github.com/ceph/ceph/pull/53976. The limits of reStructuredText
are particularly visible here in every instance of a BASH for-loop and
in every instance of a command stretched over multiple lines.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54226 from adamemerson/wip-build-experiment

build: Fix ARM build and make install-deps.sh more robust

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54216 from adamemerson/wip-fmt-felicity

build: fmt felicity

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: add --progress flag to git submodule update commands

Ceph has lots of submodules that needs to be cloned before building
binaries from the repository. Seeing the progress when these submodules
are being cloned is useful, especially when developers/users have a
network issue or a slow network.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #54272 into main

* refs/pull/54272/head:
cmake: do not issue CMAKE_BUILD_TYPE warning if set

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53720 from ljflores/bz_2241104

osd: fix logic in check_pg_upmaps

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: do not issue CMAKE_BUILD_TYPE warning if set

Fixes: b8c6ba01b262d33084b9c607d829503ac8ae25c5
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54238 from zdover23/wip-doc-2023-10-30-rados-config-osd-config-ref-scrubbing

doc/rados: improve "scrubbing" explanation

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Update default nvmeof gw version to v0.0.5

Signed-off-by: barakda <barak.davidov@gmail.com>

Merge pull request #50821 from cbodley/wip-cmake-uring

cmake: fix WITH_SYSTEM_LIBURING and make uring available outside of src/blk

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: improve "scrubbing" explanation

Improve the English describing the operation of scrubbing PGs in
doc/rados/configuration/osd-config-ref.rst.

Co-authored-by: Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54239 from zdover23/wip-doc-2023-10-30-glossary-bluestore

doc/glossary: improve "BlueStore" entry

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary: improve "BlueStore" entry

Initially s/backend/back end/ but then I added a little more information
about BlueStore's use of RocksDB to map object names to block locations
on disk.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54247 from rhcs-dashboard/fix-no-multisite-error

mgr/dashboard: fix rgw multisite error when no rgw entity is present

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54240 from rhcs-dashboard/fix-63347-main

mgr/dashboard: Block Ui fails in angular with target es2022

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54142 from YiteGu/add-bluestore-fragmentation-to-metrics

os/bluestore: add bluestore fragmentation micros to prometheus

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #53993 from shminjs/fix-out-and-down-osd-daemon-report

mgr: remove out&down osd from mgr daemons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53689 from leonid-s-usov/mds-raise

common/admin_socket: add a command to raise a signal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53563 from ifed01/wip-ifed-pg-osd-stats

osd: introduce dump_osd_pg_stats admin socket command.

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54177 from ronen-fr/wip-rf-map-msg

osd: fix use-after-move in build_incremental_map_msg()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #54198 from ceph/wip-yuriw-release-process-main

docs: revised Distro Codemap, removed `octopus`, added `reef`

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #54252 from rhcs-dashboard/quincy-release-notes

doc: Release notes 17.2.7 with updated dashboard info

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

doc: Release notes 17.2.7 with updated dashboard info

Signed-off-by: Nizamudeen A <nia@redhat.com>

build: Fix `WITH_FMT_HEADER_ONLY`

Previously it was still linking in the submodule's static libraries
and ldd showed it linking to the system fmt's shared libraries.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

mgr/rbd_support: remove CreateSnapshotRequests __del__()

There is no need for CreateSnapshotRequests.__del__() that calls
CreateSnapshotRequests.wait_for_pending().
MirrorSnapshotScheduleHandler.shutdown() already calls
CreateSnapshotRequests.wait_for_pending().

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/rbd_support: fix recursive locking on CreateSnapshotRequests lock

The MirrorSnapshotScheduleHandler's run thread issues asynchronous
create snapshot requests using a CreateSnapshotRequests instance. When
the thread invokes a CreateSnapshotRequests instance's get_ioctx(),
the instance's class variable lock is acquired. With the class
variable lock held, the garbage collection of a CreateSnapshotRequests
instance may race in the thread. The thread would then call
CreateSnapshotRequests __del__() that tries to acquire the class
variable lock that the thread already holds. Fix this
recursive deadlock by converting the CreateSnapshotRequests lock from
a class variable to an instance variable. There is no need to share
the lock across CreateSnapshotRequests instances.

Also convert MirrorSnapshotScheduleHandler, PerfHandler and
TrashPurgeScheduleHandler class variables to instance variables
that don't need to be shared across the instances.

Fixes: https://tracker.ceph.com/issues/62994
Signed-off-by: Ramana Raja <rraja@redhat.com>
Co-Authored-By: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54246 from ceph/epuertat-patch-1

create-backport-trackers: pin to Python <3.12

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Update default nvmeof gw version to v0.0.5

Signed-off-by: barakda <barak.davidov@gmail.com>

mgr/dashboard: fix rgw multisite error when no rgw entity is present

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #54241 from rhcs-dashboard/fix-63348-main

mgr/dashboard: update rgw multisite import form helper info

Reviewed-by: Nizamudeen A <nia@redhat.com>

build: Add `WITH_SYSTEM_FMT` option

The `WITH_SYSTEM_FMT` option builds against system fmt and will fail
if a suitable version is not found. It is defaulted to `OFF`.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54153 from ceph/wip-yuriw-release-17.2.7-main

doc: 17.2.7 Release Notes

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

build: Fix fmt version check

Currently, when attempting to build ceph on a system with fmt
installed, we try to build against it whatever the version. This
constantly breaks people's builds, since newer versions of fmt often
change the API.

This change specifies that versions must be below 10 as well as at or
above 8.1.1, so that on systems with a new format, we fall back to
using the submodule.

It also removes the `Findfmt.cmake` module, as that does not check
the installed version. Instead, we use the cmake config file installed by
the system package of fmt and does support version checking.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54151 from rkachach/fix_issue_63291

mgr/rook: fix the namespaces used in rook and removes hard-coded references

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@ibm.com>

create-backport-trackers: Python >= 3.12 deprecated distutils

See https://docs.python.org/3.10/whatsnew/3.10.html#distutils-deprecated.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #50593 from pkalever/fix-feature-disable

rbd-nbd: fix stuck with disable request

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: update rgw multisite import form helper info

Change 'To obtain the token, generate it from your secondary Ceph cluster' to 'To obtain the token, generate it from your primary Ceph cluster' in rgw multisite import form helper

Fixes: https://tracker.ceph.com/issues/63348
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Block Ui fails in angular with target es2022

The BlockUi element is failing in angular with target es2022..change it to es2020

Fixes: https://tracker.ceph.com/issues/63347
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #54232 from zdover23/wip-doc-2023-10-29-ceph-volume-lvm-activate-2

doc/ceph-volume: improve front matter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54231 from zdover23/wip-doc-2023-10-29-ceph-volume-lvm-activate

doc/ceph-volume: explain idempotence

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/ceph-volume: improve front matter

Simplify and improve the readability of the front matter of
doc/ceph-volume/lvm/activate.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53848 from kchheda3/wip-topic-policy

rgw/notification: Add support to set topic policy while topic creation

reviewed-by: yuvalif, anthonyeleven

Merge pull request #54230 from zdover23/wip-doc-2023-10-29-install-cephadm

doc/install: simplify cephadm docs

doc/ceph-volume: explain idempotence

Improve the grammar of a note explaining that the execution of the call
that activates ceph-volume is idempotent.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/install: simplify cephadm docs

Simplify the explanation of cephadm.

Signed-off-by: Zac Dover <zac.dover@proton.me>

build: Don't kill install-deps.sh if we can't get ceph-libboost

We can still use the submodule.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54082 from pdvian/wip-qa-suites-fix

qa/smoke,orch,perf-basic: add POOL_APP_NOT_ENABLED to ignorelist

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge branch 'main' into wip-topic-policy

Merge pull request #53003 from alimaredia/wip-rgw-perf-counters-cache

Perf counters cache + Labeled RGW Op counters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54218 from zdover23/wip-doc-2023-10-27-rados-troubleshooting-troubleshooting-pg

doc/rados: edit troubleshooting-pg.rst

Merge pull request #54219 from zdover23/wip-doc-2023-10-27-rados-ops-pools-cache-tiering-removal

doc/rados: remove cache-tiering-related keys

Merge pull request #53976 from phlogistonjohn/jjm-cephadm-gdb-extd-doc

cephadm: document extended gdb debugging techniques

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/rados: remove cache-tiering-related keys

Remove information related to cache-tiering-related keys from
doc/rados/operations/pools.rst. Cache-tiering is deprecated in Reef.
This PR is suitable for backporting to the Reef release branch, but not
to release branches prior to Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/librbd/fsx: wait for resize to propagate in krbd_resize()

With this changes resize request will not be blocked until the resize is
completed. Because of this the fsx test fails as it assumes that the
request to resize immediately implies changes on the device size.

Hence we have to add a wait in resize handler of fsx for the device to
actually get resized.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

doc/rados: edit troubleshooting-pg.rst

s/placement group/pool/ in a sentence that, prior to this change, was
confusing. Suitable for backports prior to Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54200 from zdover23/wip-doc-2023-10-26-hitset-remove-for-reef-and-later

doc/rados: remove HitSet-related key information

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados: remove HitSet-related key information

Remove HitSet-related key information from
doc/rados/operations/pools.rst. HitSet-related keys are relevant only to
releases of Ceph that support cache tiering. Only Quincy and earlier
(inclusive) releases of Ceph support cache tiering. Backport this commit
from main to Reef, but not to Quincy or to release branches earlier than
Quincy.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: 17.2.7 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53909 from aclamk/aclamk-fix-63121

os/kv_test: Fix estimate functions

docs: revised Distro Codemap, removed `octopus`, added `reef`

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #54155 from jrchyang/fix-divide-zero

os/bluestore: fix crash caused by dividing by 0

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #53807 from rhcs-dashboard/fix-63088-main

mgr/dashboard: Consider null values as zero in grafana panels

Reviewed-by: Nizamudeen A <nia@redhat.com>

osd: fix use-after-move in build_incremental_map_msg()

Fixes: https://tracker.ceph.com/issues/63310
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

doc/cephadm: extend container debugging to cover live processes

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/cephadm: clean up debugging section

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add a --dry-run option to cephadm shell

Instead of creating the shell, the --dry-run option prints the container
command that would be used. This can be used as a starting point for
creating custom container commands similar to what cephadm shell would
generate but with tweaks.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

crimson/osd/pg: rename submit_error_log returned future

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: add comments around do_osd_ops_execute parameters

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/rook: fixing the namespaces used to access some resources/config

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/rook: remove all the hardcoded refs to the 'rook-ceph' namespace
Fixes: https://tracker.ceph.com/issues/63291
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #54148 from ivoalmeida/fix-edit-user-bucket-quota

mgr/dashboard: fixed edit user quota form error

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

os/bluestore: fix crash caused by dividing by 0

In Allocator::create() => ZonedAllocator::ZonedAllocator()
num_zones is calculated by size / zone_size,
causing a crash when zone_size is set to 0

Signed-off-by: Jrchyang Yu <yuzhiqiang_yewu@cmss.chinamobile.com>

rgw/notification: Add support to set topic policy while topic creation

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

mgr/dashboard: fixed edit user quota form error

Fixes: https://tracker.ceph.com/issues/63287
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge PR #54030 into main

* refs/pull/54030/head:
client: always refresh mds feature bits on session open

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54025 from yuvalif/wip-yuval-fix-63205

rgw/notifications/test: fix rabbitmq and kafka issues in centos9

reviewed-by: cbodley

cephadm: add deployment test for osd

Add a deployment test case for OSD. OSD has some special properties that
we have extra assertions for.

Part of a series of commits to increase coverage of deployment
path features with regards to container engine options, env vars
and mounts. This will serve future refactoring efforts.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add test assertions for nvmeof options, mount

Part of a series of commits to increase coverage of deployment
path features with regards to container engine options, env vars
and mounts. This will serve future refactoring efforts.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add test assertions for iscsi options, mounts

Part of a series of commits to increase coverage of deployment
path features with regards to container engine options, env vars
and mounts. This will serve future refactoring efforts.

Signed-off-by: John Mulligan <jmulligan@redhat.com>