Merge PR #55758 into main

* refs/pull/55758/head:
doc: update 'journal reset' command with --yes-i-really-really-mean-it
qa: fix cephfs-journal-tool command options and make fs inactive
cephfs-journal-tool: Add warning messages during 'journal reset' and prevent execution on active fs

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57291 into main

* refs/pull/57291/head:
pybind/mgr/mirroring: Fix KeyError: 'directory_count' in daemon status

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57694 from zhscn/fix-avl-alloc

crimson/os/seastore/rbm/avlallocator: return enough regions when request size is greater than max_alloc_size

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57753 from zdover23/wip-doc-2024-05-29-doc-dev-perf-counters

doc/dev: add note about intro of perf counters

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57729 from idryomov/wip-66232

qa/suites/rbd: override extra_system_packages directly on install task

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #57688 into main

* refs/pull/57688/head:
tools/first-damage: Don't skip stray directory object

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57579 into main

* refs/pull/57579/head:
mds/quiesce: disable quiesce root debug parameters by default
mds/quiesce-agt: never send a synchronous ack
mds/quiesce-agt: add test for a rapid async ack
mds/quiesce: always abort fragmenting asynchronously to prevent reentrancy
mds/quiesce: overdrive an export if it hasn't frozen the tree yet
mds/quiesce: quiesce_inode should not hold on to remote auth pins
qa/cephfs: check that a completed quiesce doesn't hold remote auth pins
mds: add `--lifetime` parameter to the `lock path` asok command
mds/quiesce: accept a regular file as the quiesce root
mds: command_quiesce_path: rename `--wait` to `--await` for consistency
mds: command_quiesce_path: do not block the asok thread and return an adequate rc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev: add note about intro of perf counters

Add a note to the "perf counter" section of doc/dev/perf_counters.rst
that explains that this feature was introduced in the Reef release of
Ceph. This note will prevent us from accidentally backporting
perf-counter-related PRs to Quincy.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57462 from rhcs-dashboard/rgw-policy-group

mgr/dashboard: rgw policy group management api

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #57655 from AliMasarweh/wip-alimasa-bn-correct-reload

RGW|Bucket notification: reload realm correctly

Reviewed-by: yuvalif<ylifshit@redhat.com>

Merge pull request #57732 from rishabh-d-dave/doc-dev-teuth

doc/developer_guide: update doc about installing teuthology

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/developer_guide: update doc about installing teuthology

There are 2 more ways to install teuthology. Approach with boostrap
script is easier and more convenient while other approach is more
elaborate but manual, document both of them. Don't delete the currently
documented approach because it lets users install teuthology
conveniently in a custom virtual environment. So, keep all three.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57070 from rhcs-dashboard/fix-read-only-landing-page

mgr/dashboard: fix readonly landingpage

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57720 from zdover23/wip-doc-2024-05-27-cephfs-fs-volumes

doc/cephfs: s/subvolumegroups/subvolume groups

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57638 from yuvalif/wip-yuval-66033

test/rgw/notifications: allow http server to buffer more requests

Reviewed-By: Casey Bodley <cbodley@ibm.com>

mgr/dashboard: manage flow and pipes apis

Fixes: https://tracker.ceph.com/issues/66238
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: rgw policy group management api

Fixes: https://tracker.ceph.com/issues/66238
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57700 from tchaikov/wip-mgr-prometheus-packaging

mgr/prometheus: s/pkg_resources.packaging/packaging/

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

qa/suites/rbd: override extra_system_packages directly on install task

[1] and [2] added support for applying extra_system_packages overrides
directly on install task, but at the same time broke our long standing
workaround where we sneaked extra_system_packages directive in through
an override on ceph task.  This is likely getting addressed in [3], but
it's better to not rely on this odd feature in the first place.

[1] https://github.com/ceph/teuthology/pull/1941
[2] https://github.com/ceph/teuthology/pull/1943
[3] https://github.com/ceph/teuthology/pull/1942

Fixes: https://tracker.ceph.com/issues/66232
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57671 from idryomov/wip-labeler-qa

.github: expand tests label to all files under qa

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #57628 from joscollin/wip-B66173-doc-per-client-metrics

doc: document metrics exported by CephFS

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: s/subvolumegroups/subvolume groups

Use the term "subvolume groups" instead of "subvolumegroups" where the
term appears in plain English. The string "subvolumegroups" is correct
in commands, and remains unchanged.

Also add formatting to command output, to make clearer that the output
is indeed output.

Fixes: https://github.com/ceph/ceph/pull/57574#discussion_r1606988766
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: document metrics exported by CephFS

Fixes: https://tracker.ceph.com/issues/66173
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #57584 from rhcs-dashboard/dashboard-pendingreleasenotes

doc: Update pendingreleasenotes for dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

mds/quiesce: disable quiesce root debug parameters by default

Fixes: https://tracker.ceph.com/issues/66225
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-agt: never send a synchronous ack

Defer to the agent thread to perform all acking.
This avoids race conditions between the updating thread
and the acking thread.

Fixes: https://tracker.ceph.com/issues/66219
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-agt: add test for a rapid async ack

In this scenario, the agent thread is able to run and generate an ack
before the db_update call returns to the caller.

Fixes: https://tracker.ceph.com/issues/66219
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: always abort fragmenting asynchronously to prevent reentrancy

Fixes: https://tracker.ceph.com/issues/66208
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: overdrive an export if it hasn't frozen the tree yet

Just like with the fragmenting, we should abort an ongoing export
if a quiesce is attempted for the directory.

To minimize the stress for the system, we only allow the abort
if the export hasn't yet managed to freeze the tree. If that is the case,
then quiesce will have to wait for the export to finish.

Fixes: https://tracker.ceph.com/issues/66123
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: quiesce_inode should not hold on to remote auth pins

1. avoid taking a remote authpin for the quiesce lock
2. drop remote authpins that were taken because of other locks

We should not be forcing a mustpin when taking quiesce lock.
This creates unnecessary overhead due to the distributed nature
of the quiesce: all ranks will execute quiesce_inode, including
the auth rank, which will authpin the inode.

Auth pinning on the auth rank is important to synchronize quiesce
with operations that are managed by the auth, like fragmenting
and exporting.

If we let a remote quiesce process take a foreign authpin then
it may block freezing on the auth, which will stall quiesce locally.
This wouldn't be a problem if the quiesce that is blocked on the auth
and the quiesce that's holding a remote authpin from the replica side
were unrelated, but in our case it may be the same logical quiesce
that effectively steps on its own toes. This creates an opportunity
for a deadlock.

Fixes: https://tracker.ceph.com/issues/66152
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

qa/cephfs: check that a completed quiesce doesn't hold remote auth pins

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: add `--lifetime` parameter to the `lock path` asok command

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: accept a regular file as the quiesce root

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: command_quiesce_path: rename `--wait` to `--await` for consistency

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: command_quiesce_path: do not block the asok thread and return an adequate rc

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57393 into main

* refs/pull/57393/head:
doc/man: add missing long option switches

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge pull request #57609 from Svelar/asan_rbd_mirror

test/rbd-mirror: free remote_journaler in PrepareRemoteImageRequest tests

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #56033 from yuvalif/wip-yuval-kafka-cleanup

rgw/kafka: do not destroy the connection on errors

Reviewed-By: Krunal Chheda <kchheda3@bloomberg.net>, Casey Bodley <cbodley@ibm.com>

Merge pull request #57701 from zdover23/wip-doc-2024-05-25-cephfs-fs-volumes

doc/cephfs: disambiguate two sentences

doc/cephfs: disambiguate two sentences

Rewrite two sentences so that they are unambgiuously in the indicative
mood and not, as they previously seemed, in the imperative mood.

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/rbd: free remote_journaler in PrepareRemoteImageRequest tests

When sanitizer is enabled, unittest_rbd_mirror shows

```
=================================================================
==1377627==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2 byte(s) in 2 object(s) allocated from:
    #0 0xaaaac462d088 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rbd_mirror+0x30cd088) (BuildId: 054d3b6699c622daad91c7f70e36616220fbd5ad)
    #1 0xaaaac522f13c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::get_client() /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:148:24
    #2 0xaaaac522e60c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int) /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:120:5
    #3 0xaaaac5232d1c in librbd::util::detail::C_CallbackAdapter<rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>, &(rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int))>::finish(int) /root/ceph/src/librbd/Utils.h:63:5
    #4 0xaaaac4863428 in Context::complete(int) /root/ceph/src/include/Context.h:99:5
    #5 0xaaaac489ac30 in librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()::operator()() const /root/ceph/src/librbd/asio/ContextWQ.h:31:12
    #6 0xaaaac489aaf4 in boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>::operator()() /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:60:5
    #7 0xaaaac489aaac in void boost::asio::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3
    #8 0xaaaac489aa4c in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #9 0xaaaac489a9ec in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>*) /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:111:3
    #10 0xaaaac4899c00 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #11 0xaaaac489c814 in boost::asio::detail::executor_op<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7
    #12 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5
    #13 0xaaaac489f074 in boost::asio::detail::strand_executor_service::run_ready_handlers(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>&) /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.ipp:150:8
    #14 0xaaaac489ed2c in boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.hpp:136:5
    #15 0xaaaac489ebd4 in void boost::asio::asio_handler_invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3
    #16 0xaaaac489e284 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #17 0xaaaac48a3e08 in boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7
    #18 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5
    #19 0xaaaac6766b1c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:493:12
    #20 0xaaaac67660a4 in boost::asio::detail::scheduler::run(boost::system::error_code&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:210:10
    #21 0xaaaac7562d48 in boost::asio::io_context::run() /root/ceph/build/boost/include/boost/asio/impl/io_context.ipp:64:24
    #22 0xaaaac7562bcc in ceph::async::io_context_pool::start(short)::'lambda'()::operator()() const /root/ceph/src/common/async/context_pool.h:69:16
    #23 0xaaaac7562b18 in void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #24 0xaaaac7562960 in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #25 0xaaaac7562804 in std::invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/functional:97:14
    #26 0xaaaac75627dc in auto std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)::operator()<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) const /root/ceph/src/common/Thread.h:79:10
    #27 0xaaaac7562728 in ceph::async::io_context_pool::start(short)::'lambda'() std::__invoke_impl<void, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)&&, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #28 0xaaaac75624ec in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #29 0xaaaac756231c in void std::thread::_Invoker<std::tuple<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13

SUMMARY: AddressSanitizer: 2 byte(s) leaked in 2 allocation(s).
```

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #57693 from tchaikov/cmake-shec_utils

cmake: : link shec_utils against legacy-option-headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/prometheus: s/pkg_resources.packaging/packaging/

instead of relying on the internal implementation of pkg_resources,
let's import packaging directly.

in this change, we also add the dependency to the packaging module
in the packaging recipes.

See also https://github.com/pypa/setuptools/issues/4385

Fixes https://tracker.ceph.com/issues/66201
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #57686 into main

* refs/pull/57686/head:
script/ceph-backport: update Ceph PR id

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57689 from kotreshhr/fix_unpack_first_damage

tools/first-damage: Fix struct unpack

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/rbm/avlallocator: return enough regions when request size is greater than max_alloc_size

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

cmake: : link shec_utils against legacy-option-headers

in c24a6ffe20, we tried to link all target dependent on legacy option
headers against legacy-option-headers, but we missed some of them.
in our CI, we spotted build failure like:
```
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/erasure-code/ErasureCode.cc:26:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/osd/osd_types.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config_proxy.h:6:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config.h:27:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config_values.h:59:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/options/legacy_config_opts.h:1:10: fatal error: 'global_legacy_options.h' file not found
         ^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
so in this change, we link the shec_utils to `legacy-option-headers`
as well to fulfill the build dependency.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tools/first-damage: Fix struct unpack

The first field is a 64 bit integer, the script
was unpacking it as 32 bit integer. Fixed the
same.

Also printing the 'first' value in debug

Signed-off-by: Kotresh HR <khiremat@redhat.com>

tools/first-damage: Don't skip stray directory object

Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #57654 from zhscn/fix-error

crimson/os/seastore: fix compile error in release build

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57656 from Matan-B/wip-matanb-crimson-get-param

test/crimson/seastore/transaction_manager_test_state: fix compilation…

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

script/ceph-backport: update Ceph PR id

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57188 from cbodley/wip-boost-asio-yield-group

common/async: add primitives for structured concurrency with optional_yield

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57630 from cbodley/wip-boost-asio-spawn-unlock

rgw/pubsub: Waiter unlocks before suspend

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

.github: expand tests label to all files under qa

The test job definition under qa/suites is an integral part of almost
any test.  Often, the test logic is split between the task or workunit
and respective snippet(s) under qa/suites.

Other files under qa are less used, but still related to nothing but
testing, so just add the label on all of it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

RGW|Bucket notification: disable notif v2 to reload realm correctly

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #57554 from zdover23/wip-doc-2024-05-20-cephfs-fs-volumes-fs-subvolumes-sections

doc/cephfs: separate commands into sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57567 from zdover23/wip-doc-2024-05-20-cephfs-fs-volumes-cloning-snapshots

doc/cephfs: edit "Cloning Snapshots" in fs-volumes.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57574 from zdover23/wip-doc-2024-05-21-cephfs-fs-volumes-pinning-subvolumes

doc/cephfs: edit "Pinning Subvolumes..."

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

common/async: add max_concurrent_for_each() algorithm

inspired by seastar's max_concurrent_for_each(), implemented for
optional_yield in terms of yield_throttle

can also be overloaded for co_await() and co_throttle (not part of this
branch)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix readonly landingpage

Fixes: https://tracker.ceph.com/issues/65643
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

test/crimson/seastore/transaction_manager_test_state: fix compilation error

Introduced in: https://github.com/ceph/ceph/commit/898efa78e2cde90a2bdf8e6f7eda1d4def736705

Followup: https://github.com/ceph/ceph/pull/53433

```
FAILED: src/crimson/tools/CMakeFiles/perf-staged-fltree.dir/perf_staged_fltree.cc.o
 error: ‘GetParam’ was not declared in this scope
  287 |     if (std::get<1>(GetParam()) == integrity_check_t::FULL_CHECK) {
      |                     ^~~~~~~~
/home/yogisha/ceph/src/test/crimson/seastore/transaction_manager_test_state.h: In member function ‘virtual seastar::future<> SeaStoreTestState::_init()’:
/home/yogisha/ceph/src/test/crimson/seastore/transaction_manager_test_state.h:439:21: error: ‘GetParam’ was not declared in this scope
  439 |     if (std::get<1>(GetParam()) == integrity_check_t::FULL_CHECK) {
      |                     ^~~~~~~~
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57639 from yuvalif/wip-yuval-skip-reload-tests

test/rgw/notifications: skip reload tests when there is no realm

Reviewed-By: ali.masarwa@ibm.com

crimson/os/seastore: fix compile error in release build

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

doc: Update pendingreleasenotes for dashboard

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57633 from mcv21/patch-1

Docs: clarify use of location: in host spec

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge PR #57493 into main

* refs/pull/57493/head:
qa/cephfs: pass MDS name, not FS name, to "ceph mds fail" cmd

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57469 into main

* refs/pull/57469/head:
mds: set dispatcher order
mds: use regular dispatch for processing beacons
msg: add priority to dispatcher invocation order
mds: note when dispatcher is called

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57392 into main

* refs/pull/57392/head:
common/Formatter: use CachedStackStringStream for efficiency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #57374 into main

* refs/pull/57374/head:
common/Formatter: trivial cons/des should be default

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #57215 into main

* refs/pull/57215/head:
doc: document new --output-file switch
test/cli: ignore tmp_file_template
qa/workunits: add --output-file test in cephtool workunit
common,ceph: add output file switch to dump json to
common/options: add configs for temporary files made by daemons
common/Formatter: write the pending string on flush

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge PR #57170 into main

* refs/pull/57170/head:
client: set LIBMOUNT_FORCE_MOUNT2=always

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57081 into main

* refs/pull/57081/head:
messages/MClientMetrics: increase priority ahead of regular requests
mds: use regular dispatch for processing metrics

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57642 from dmick/wip-release-doc

doc/dev/release-process.rst: note new 'project' arguments

doc/dev/release-process.rst: note new 'project' arguments

Support added to the release scripts (from ceph-build.git) to
work for ceph-iscsi, so 'project' must be passed to these scripts,
and will appear in the prerelease pathnames.  See also
https://github.com/ceph/ceph-build/pull/2243 and
https://github.com/ceph/ceph-container/pull/2210

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #57575 from cbodley/wip-rgw-maintainers-ali

rgw: remove Ali Maredia from MAINTAINERS.md

Reviewed-by: Adam Emerson <aemerson@redhat.com>

common/async: add spawn_throttle for bounded concurrency with optional_yield

a primitive for structured concurrency with stackful coroutines from
boost::asio::spawn(). this relies on spawn()'s support for per-op
cancellation to guarantee that the lifetime of child coroutines won't
exceed the lifetime of their spawn_throttle, making it safe for children
to access memory from their parent's stack

by taking optional_yield in the constructor, spawn_throttle transparently
supports synchronous execution (where optional_yield is empty) and
asynchronous execution within a stackful coroutine (where optional_yield
contains the parent's yield_context)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/notifications: skip reload tests when there is no realm

this does not impact teuthology, only local runs.
in teuthology all bucket notification tests are defined with a realm

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notifications: allow http server to buffer more requests

python doc:
https://docs.python.org/3/library/socketserver.html#socketserver.ThreadingMixIn
say tha the default is 5

Fixes: https://tracker.ceph.com/issues/66033
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/kafka: do not destroy the connection on errors

as well as other simplifications:
* do not store temporary configuration in the connection object. just
  use as a local variable
* do not create a connection without a producer

other improvements:
* copy to a local list before publishing
* convert internal error codes to errno

Fixes: https://tracker.ceph.com/issues/66017
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

common/async: add cancel_on_error enum

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add service template for execution_context shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add yield_waiter template

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: clarify use of location: in host spec

It wasn't clear that you can specify more than one element of the CRUSH hierarchy in a spec file, nor that it might be useful to do so (e.g. to ensure the host ends up beneath the default root).

So update the text to make it clearer, and similarly the example.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Merge pull request #57616 from Matan-B/wip-matanb-fix-python-tests

test/pybind/test_rados.py: fix test_list_objects

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw/pubsub: Waiter unlocks before suspend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/pybind/test_rados.py: fix test_list_objects

Commit 6662e6bc215c2d0bee16bc3674b07be1e239b4c0 accidentally removed a line from this test.

```
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:=================================== FAILURES ===================================
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:_________________________ TestIoctx.test_list_objects __________________________
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:self = <test_rados.TestIoctx object at 0x7fddcc6924c0>
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:    def test_list_objects(self):
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:        self.ioctx.write('b', b'foo')
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:        self.ioctx.write_full('c', b'bar')
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:        self.ioctx.append('d', b'jazz')
2024-05-21T14:06:10.464 INFO:tasks.workunit.client.0.smithi031.stdout:        object_names = [obj.key for obj in self.ioctx.list_objects()]
2024-05-21T14:06:10.464 INFO:tasks.workunit.client.0.smithi031.stdout:>       eq(sorted(object_names), ['a', 'b', 'c', 'd'])
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55592 from cbodley/wip-boost-asio-spawn

rgw: switch back to boost::asio for spawn() and yield_context

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #57152 from sajibreadd/wip-61887

s3: placement target is added in GetBucketLocation api response as header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57409 from Suyashd999/fix-uam1

current_status is being used after moving.

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #57249 from cbodley/wip-65668

rgw/notify: decouple add/remove_persistent_topic() from Manager

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #56014 from badone/wip-tracker-63591-pyyaml-cython_sources

install-deps: Update Pyyaml version

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57529 from rishabh-d-dave/ptl-no-ceph-ci-push

ptl-tool: allow not pushing branch to ceph-ci

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #55831 from xxhdx1985126/wip-seastore-lba-pointer-consolidation

crimson/os/seastore: lba pointer consolidation

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57262 from xxhdx1985126/wip-seastore-clone-remap-refcount

crimson/os/seastore/lba_manager: don't increase intermediate mappings' refcount if LBAManager::clone_mapping() is called to remap mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #57552 into main

* refs/pull/57552/head:
mds/queisce-db: collect acks while bootstrapping

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57332 into main

* refs/pull/57332/head:
mds/quiesce: drop remote authpins before waiting for the quiesce lock
qa/cephfs/test_quiesce: test proper handling of remote authpins
mds: don't clear `AUTHPIN_FROZEN` until `FROZEN` in rename_prep
mds: enhance the `lock path` asok command
mds/quiesce: overdrive fragmenting that's still freezing
revert: mds: provide a mechanism to authpin while freezing
qa/cephfs/test_quiesce: enhance the fragmentation test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57583 from tchaikov/cmake-legacy-headers

cmake: link rados_snap_set_diff_obj and krbd against legacy-option-headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: link rados_snap_set_diff_obj and krbd against legacy-option-headers

in c24a6ffe20, we tried to link all target dependent on legacy option
headers against legacy-option-headers, but we missed some of them.
in our CI, we spotted build failure like:
```
FAILED: src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o
/usr/bin/ccache /usr/bin/clang++-14 -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_NO_TS_EXECUTORS -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/home/jenkins-build/build/workspace/ceph-api/build/src/include -I/home/jenkins-build/build/workspace/ceph-api/src -isystem /opt/ceph/include -isystem /home/jenkins-build/build/workspace/ceph-api/build/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/api/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/ext/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/sdk/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/xxHash -isystem /home/jenkins-build/build/workspace/ceph-api/src/fmt/include -g -Werror -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-unused-private-field -Wno-address-of-packed-member -Wno-unused-function -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register -DCEPH_DEBUG_MUTEX -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++20 -MD -MT src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o -MF src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o.d -o src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o -c /home/jenkins-build/build/workspace/ceph-api/src/librados/snap_set_diff.cc
In file included from /home/jenkins-build/build/workspace/ceph-api/src/librados/snap_set_diff.cc:7:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/ceph_context.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_proxy.h:6:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config.h:27:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_values.h:59:
/home/jenkins-build/build/workspace/ceph-api/src/common/options/legacy_config_opts.h:7:10: fatal error: 'osd_legacy_options.h' file not found
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[111/1748] Generating immutable-object-cache_options.cc, ../../../include/immutable-object-cache_legacy_options.h
[112/1748] Building CXX object src/CMakeFiles/krbd.dir/krbd.cc.o
FAILED: src/CMakeFiles/krbd.dir/krbd.cc.o
/usr/bin/ccache /usr/bin/clang++-14 -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_NO_TS_EXECUTORS -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/home/jenkins-build/build/workspace/ceph-api/build/src/include -I/home/jenkins-build/build/workspace/ceph-api/src -isystem /opt/ceph/include -isystem /home/jenkins-build/build/workspace/ceph-api/build/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/api/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/ext/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/sdk/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/xxHash -isystem /home/jenkins-build/build/workspace/ceph-api/src/fmt/include -g -Werror -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-unused-private-field -Wno-address-of-packed-member -Wno-unused-function -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register -DCEPH_DEBUG_MUTEX -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++20 -MD -MT src/CMakeFiles/krbd.dir/krbd.cc.o -MF src/CMakeFiles/krbd.dir/krbd.cc.o.d -o src/CMakeFiles/krbd.dir/krbd.cc.o -c /home/jenkins-build/build/workspace/ceph-api/src/krbd.cc
In file included from /home/jenkins-build/build/workspace/ceph-api/src/krbd.cc:44:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/mon/MonMap.h:28:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/mon/mon_types.h:20:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/include/Context.h:19:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/dout.h:29:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/ceph_context.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_proxy.h:6:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config.h:27:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_values.h:59:
/home/jenkins-build/build/workspace/ceph-api/src/common/options/legacy_config_opts.h:11:10: fatal error: 'rgw_legacy_options.h' file not found
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```

so in this change, we link the related targets to
`legacy-option-headers` as well to fulfill the build dependency.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mds/quiesce: drop remote authpins before waiting for the quiesce lock

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/65802