test/rgw/notification: verify network access to http server

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57070 from rhcs-dashboard/fix-read-only-landing-page

mgr/dashboard: fix readonly landingpage

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57720 from zdover23/wip-doc-2024-05-27-cephfs-fs-volumes

doc/cephfs: s/subvolumegroups/subvolume groups

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57638 from yuvalif/wip-yuval-66033

test/rgw/notifications: allow http server to buffer more requests

Reviewed-By: Casey Bodley <cbodley@ibm.com>

Merge pull request #57700 from tchaikov/wip-mgr-prometheus-packaging

mgr/prometheus: s/pkg_resources.packaging/packaging/

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #57671 from idryomov/wip-labeler-qa

.github: expand tests label to all files under qa

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #57628 from joscollin/wip-B66173-doc-per-client-metrics

doc: document metrics exported by CephFS

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: s/subvolumegroups/subvolume groups

Use the term "subvolume groups" instead of "subvolumegroups" where the
term appears in plain English. The string "subvolumegroups" is correct
in commands, and remains unchanged.

Also add formatting to command output, to make clearer that the output
is indeed output.

Fixes: https://github.com/ceph/ceph/pull/57574#discussion_r1606988766
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: document metrics exported by CephFS

Fixes: https://tracker.ceph.com/issues/66173
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #57584 from rhcs-dashboard/dashboard-pendingreleasenotes

doc: Update pendingreleasenotes for dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #57393 into main

* refs/pull/57393/head:
doc/man: add missing long option switches

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge pull request #57609 from Svelar/asan_rbd_mirror

test/rbd-mirror: free remote_journaler in PrepareRemoteImageRequest tests

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #56033 from yuvalif/wip-yuval-kafka-cleanup

rgw/kafka: do not destroy the connection on errors

Reviewed-By: Krunal Chheda <kchheda3@bloomberg.net>, Casey Bodley <cbodley@ibm.com>

Merge pull request #57701 from zdover23/wip-doc-2024-05-25-cephfs-fs-volumes

doc/cephfs: disambiguate two sentences

doc/cephfs: disambiguate two sentences

Rewrite two sentences so that they are unambgiuously in the indicative
mood and not, as they previously seemed, in the imperative mood.

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/rbd: free remote_journaler in PrepareRemoteImageRequest tests

When sanitizer is enabled, unittest_rbd_mirror shows

```
=================================================================
==1377627==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2 byte(s) in 2 object(s) allocated from:
    #0 0xaaaac462d088 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rbd_mirror+0x30cd088) (BuildId: 054d3b6699c622daad91c7f70e36616220fbd5ad)
    #1 0xaaaac522f13c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::get_client() /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:148:24
    #2 0xaaaac522e60c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int) /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:120:5
    #3 0xaaaac5232d1c in librbd::util::detail::C_CallbackAdapter<rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>, &(rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int))>::finish(int) /root/ceph/src/librbd/Utils.h:63:5
    #4 0xaaaac4863428 in Context::complete(int) /root/ceph/src/include/Context.h:99:5
    #5 0xaaaac489ac30 in librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()::operator()() const /root/ceph/src/librbd/asio/ContextWQ.h:31:12
    #6 0xaaaac489aaf4 in boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>::operator()() /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:60:5
    #7 0xaaaac489aaac in void boost::asio::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3
    #8 0xaaaac489aa4c in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #9 0xaaaac489a9ec in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>*) /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:111:3
    #10 0xaaaac4899c00 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #11 0xaaaac489c814 in boost::asio::detail::executor_op<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7
    #12 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5
    #13 0xaaaac489f074 in boost::asio::detail::strand_executor_service::run_ready_handlers(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>&) /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.ipp:150:8
    #14 0xaaaac489ed2c in boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.hpp:136:5
    #15 0xaaaac489ebd4 in void boost::asio::asio_handler_invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3
    #16 0xaaaac489e284 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #17 0xaaaac48a3e08 in boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7
    #18 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5
    #19 0xaaaac6766b1c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:493:12
    #20 0xaaaac67660a4 in boost::asio::detail::scheduler::run(boost::system::error_code&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:210:10
    #21 0xaaaac7562d48 in boost::asio::io_context::run() /root/ceph/build/boost/include/boost/asio/impl/io_context.ipp:64:24
    #22 0xaaaac7562bcc in ceph::async::io_context_pool::start(short)::'lambda'()::operator()() const /root/ceph/src/common/async/context_pool.h:69:16
    #23 0xaaaac7562b18 in void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #24 0xaaaac7562960 in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #25 0xaaaac7562804 in std::invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/functional:97:14
    #26 0xaaaac75627dc in auto std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)::operator()<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) const /root/ceph/src/common/Thread.h:79:10
    #27 0xaaaac7562728 in ceph::async::io_context_pool::start(short)::'lambda'() std::__invoke_impl<void, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)&&, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #28 0xaaaac75624ec in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #29 0xaaaac756231c in void std::thread::_Invoker<std::tuple<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13

SUMMARY: AddressSanitizer: 2 byte(s) leaked in 2 allocation(s).
```

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #57693 from tchaikov/cmake-shec_utils

cmake: : link shec_utils against legacy-option-headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/prometheus: s/pkg_resources.packaging/packaging/

instead of relying on the internal implementation of pkg_resources,
let's import packaging directly.

in this change, we also add the dependency to the packaging module
in the packaging recipes.

See also https://github.com/pypa/setuptools/issues/4385

Fixes https://tracker.ceph.com/issues/66201
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #57686 into main

* refs/pull/57686/head:
script/ceph-backport: update Ceph PR id

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57689 from kotreshhr/fix_unpack_first_damage

tools/first-damage: Fix struct unpack

Reviewed-by: Venky Shankar <vshankar@redhat.com>

cmake: : link shec_utils against legacy-option-headers

in c24a6ffe20, we tried to link all target dependent on legacy option
headers against legacy-option-headers, but we missed some of them.
in our CI, we spotted build failure like:
```
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/erasure-code/ErasureCode.cc:26:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/osd/osd_types.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config_proxy.h:6:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config.h:27:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config_values.h:59:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/options/legacy_config_opts.h:1:10: fatal error: 'global_legacy_options.h' file not found
         ^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
so in this change, we link the shec_utils to `legacy-option-headers`
as well to fulfill the build dependency.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tools/first-damage: Fix struct unpack

The first field is a 64 bit integer, the script
was unpacking it as 32 bit integer. Fixed the
same.

Also printing the 'first' value in debug

Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #57654 from zhscn/fix-error

crimson/os/seastore: fix compile error in release build

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57656 from Matan-B/wip-matanb-crimson-get-param

test/crimson/seastore/transaction_manager_test_state: fix compilation…

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

script/ceph-backport: update Ceph PR id

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57188 from cbodley/wip-boost-asio-yield-group

common/async: add primitives for structured concurrency with optional_yield

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57630 from cbodley/wip-boost-asio-spawn-unlock

rgw/pubsub: Waiter unlocks before suspend

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

.github: expand tests label to all files under qa

The test job definition under qa/suites is an integral part of almost
any test.  Often, the test logic is split between the task or workunit
and respective snippet(s) under qa/suites.

Other files under qa are less used, but still related to nothing but
testing, so just add the label on all of it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57554 from zdover23/wip-doc-2024-05-20-cephfs-fs-volumes-fs-subvolumes-sections

doc/cephfs: separate commands into sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57567 from zdover23/wip-doc-2024-05-20-cephfs-fs-volumes-cloning-snapshots

doc/cephfs: edit "Cloning Snapshots" in fs-volumes.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57574 from zdover23/wip-doc-2024-05-21-cephfs-fs-volumes-pinning-subvolumes

doc/cephfs: edit "Pinning Subvolumes..."

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

common/async: add max_concurrent_for_each() algorithm

inspired by seastar's max_concurrent_for_each(), implemented for
optional_yield in terms of yield_throttle

can also be overloaded for co_await() and co_throttle (not part of this
branch)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix readonly landingpage

Fixes: https://tracker.ceph.com/issues/65643
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

test/crimson/seastore/transaction_manager_test_state: fix compilation error

Introduced in: https://github.com/ceph/ceph/commit/898efa78e2cde90a2bdf8e6f7eda1d4def736705

Followup: https://github.com/ceph/ceph/pull/53433

```
FAILED: src/crimson/tools/CMakeFiles/perf-staged-fltree.dir/perf_staged_fltree.cc.o
 error: ‘GetParam’ was not declared in this scope
  287 |     if (std::get<1>(GetParam()) == integrity_check_t::FULL_CHECK) {
      |                     ^~~~~~~~
/home/yogisha/ceph/src/test/crimson/seastore/transaction_manager_test_state.h: In member function ‘virtual seastar::future<> SeaStoreTestState::_init()’:
/home/yogisha/ceph/src/test/crimson/seastore/transaction_manager_test_state.h:439:21: error: ‘GetParam’ was not declared in this scope
  439 |     if (std::get<1>(GetParam()) == integrity_check_t::FULL_CHECK) {
      |                     ^~~~~~~~
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57639 from yuvalif/wip-yuval-skip-reload-tests

test/rgw/notifications: skip reload tests when there is no realm

Reviewed-By: ali.masarwa@ibm.com

crimson/os/seastore: fix compile error in release build

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

doc: Update pendingreleasenotes for dashboard

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57633 from mcv21/patch-1

Docs: clarify use of location: in host spec

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge PR #57493 into main

* refs/pull/57493/head:
qa/cephfs: pass MDS name, not FS name, to "ceph mds fail" cmd

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57469 into main

* refs/pull/57469/head:
mds: set dispatcher order
mds: use regular dispatch for processing beacons
msg: add priority to dispatcher invocation order
mds: note when dispatcher is called

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57392 into main

* refs/pull/57392/head:
common/Formatter: use CachedStackStringStream for efficiency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #57374 into main

* refs/pull/57374/head:
common/Formatter: trivial cons/des should be default

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #57215 into main

* refs/pull/57215/head:
doc: document new --output-file switch
test/cli: ignore tmp_file_template
qa/workunits: add --output-file test in cephtool workunit
common,ceph: add output file switch to dump json to
common/options: add configs for temporary files made by daemons
common/Formatter: write the pending string on flush

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge PR #57170 into main

* refs/pull/57170/head:
client: set LIBMOUNT_FORCE_MOUNT2=always

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57081 into main

* refs/pull/57081/head:
messages/MClientMetrics: increase priority ahead of regular requests
mds: use regular dispatch for processing metrics

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57642 from dmick/wip-release-doc

doc/dev/release-process.rst: note new 'project' arguments

doc/dev/release-process.rst: note new 'project' arguments

Support added to the release scripts (from ceph-build.git) to
work for ceph-iscsi, so 'project' must be passed to these scripts,
and will appear in the prerelease pathnames.  See also
https://github.com/ceph/ceph-build/pull/2243 and
https://github.com/ceph/ceph-container/pull/2210

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #57575 from cbodley/wip-rgw-maintainers-ali

rgw: remove Ali Maredia from MAINTAINERS.md

Reviewed-by: Adam Emerson <aemerson@redhat.com>

common/async: add spawn_throttle for bounded concurrency with optional_yield

a primitive for structured concurrency with stackful coroutines from
boost::asio::spawn(). this relies on spawn()'s support for per-op
cancellation to guarantee that the lifetime of child coroutines won't
exceed the lifetime of their spawn_throttle, making it safe for children
to access memory from their parent's stack

by taking optional_yield in the constructor, spawn_throttle transparently
supports synchronous execution (where optional_yield is empty) and
asynchronous execution within a stackful coroutine (where optional_yield
contains the parent's yield_context)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/notifications: skip reload tests when there is no realm

this does not impact teuthology, only local runs.
in teuthology all bucket notification tests are defined with a realm

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notifications: allow http server to buffer more requests

python doc:
https://docs.python.org/3/library/socketserver.html#socketserver.ThreadingMixIn
say tha the default is 5

Fixes: https://tracker.ceph.com/issues/66033
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/kafka: do not destroy the connection on errors

as well as other simplifications:
* do not store temporary configuration in the connection object. just
  use as a local variable
* do not create a connection without a producer

other improvements:
* copy to a local list before publishing
* convert internal error codes to errno

Fixes: https://tracker.ceph.com/issues/66017
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

common/async: add cancel_on_error enum

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add service template for execution_context shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add yield_waiter template

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: clarify use of location: in host spec

It wasn't clear that you can specify more than one element of the CRUSH hierarchy in a spec file, nor that it might be useful to do so (e.g. to ensure the host ends up beneath the default root).

So update the text to make it clearer, and similarly the example.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Merge pull request #57616 from Matan-B/wip-matanb-fix-python-tests

test/pybind/test_rados.py: fix test_list_objects

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw/pubsub: Waiter unlocks before suspend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/pybind/test_rados.py: fix test_list_objects

Commit 6662e6bc215c2d0bee16bc3674b07be1e239b4c0 accidentally removed a line from this test.

```
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:=================================== FAILURES ===================================
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:_________________________ TestIoctx.test_list_objects __________________________
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:self = <test_rados.TestIoctx object at 0x7fddcc6924c0>
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:    def test_list_objects(self):
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:        self.ioctx.write('b', b'foo')
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:        self.ioctx.write_full('c', b'bar')
2024-05-21T14:06:10.463 INFO:tasks.workunit.client.0.smithi031.stdout:        self.ioctx.append('d', b'jazz')
2024-05-21T14:06:10.464 INFO:tasks.workunit.client.0.smithi031.stdout:        object_names = [obj.key for obj in self.ioctx.list_objects()]
2024-05-21T14:06:10.464 INFO:tasks.workunit.client.0.smithi031.stdout:>       eq(sorted(object_names), ['a', 'b', 'c', 'd'])
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55592 from cbodley/wip-boost-asio-spawn

rgw: switch back to boost::asio for spawn() and yield_context

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #57152 from sajibreadd/wip-61887

s3: placement target is added in GetBucketLocation api response as header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57409 from Suyashd999/fix-uam1

current_status is being used after moving.

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #57249 from cbodley/wip-65668

rgw/notify: decouple add/remove_persistent_topic() from Manager

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #56014 from badone/wip-tracker-63591-pyyaml-cython_sources

install-deps: Update Pyyaml version

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57529 from rishabh-d-dave/ptl-no-ceph-ci-push

ptl-tool: allow not pushing branch to ceph-ci

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #55831 from xxhdx1985126/wip-seastore-lba-pointer-consolidation

crimson/os/seastore: lba pointer consolidation

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57262 from xxhdx1985126/wip-seastore-clone-remap-refcount

crimson/os/seastore/lba_manager: don't increase intermediate mappings' refcount if LBAManager::clone_mapping() is called to remap mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #57552 into main

* refs/pull/57552/head:
mds/queisce-db: collect acks while bootstrapping

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57332 into main

* refs/pull/57332/head:
mds/quiesce: drop remote authpins before waiting for the quiesce lock
qa/cephfs/test_quiesce: test proper handling of remote authpins
mds: don't clear `AUTHPIN_FROZEN` until `FROZEN` in rename_prep
mds: enhance the `lock path` asok command
mds/quiesce: overdrive fragmenting that's still freezing
revert: mds: provide a mechanism to authpin while freezing
qa/cephfs/test_quiesce: enhance the fragmentation test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57583 from tchaikov/cmake-legacy-headers

cmake: link rados_snap_set_diff_obj and krbd against legacy-option-headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: link rados_snap_set_diff_obj and krbd against legacy-option-headers

in c24a6ffe20, we tried to link all target dependent on legacy option
headers against legacy-option-headers, but we missed some of them.
in our CI, we spotted build failure like:
```
FAILED: src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o
/usr/bin/ccache /usr/bin/clang++-14 -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_NO_TS_EXECUTORS -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/home/jenkins-build/build/workspace/ceph-api/build/src/include -I/home/jenkins-build/build/workspace/ceph-api/src -isystem /opt/ceph/include -isystem /home/jenkins-build/build/workspace/ceph-api/build/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/api/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/ext/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/sdk/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/xxHash -isystem /home/jenkins-build/build/workspace/ceph-api/src/fmt/include -g -Werror -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-unused-private-field -Wno-address-of-packed-member -Wno-unused-function -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register -DCEPH_DEBUG_MUTEX -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++20 -MD -MT src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o -MF src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o.d -o src/CMakeFiles/rados_snap_set_diff_obj.dir/librados/snap_set_diff.cc.o -c /home/jenkins-build/build/workspace/ceph-api/src/librados/snap_set_diff.cc
In file included from /home/jenkins-build/build/workspace/ceph-api/src/librados/snap_set_diff.cc:7:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/ceph_context.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_proxy.h:6:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config.h:27:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_values.h:59:
/home/jenkins-build/build/workspace/ceph-api/src/common/options/legacy_config_opts.h:7:10: fatal error: 'osd_legacy_options.h' file not found
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[111/1748] Generating immutable-object-cache_options.cc, ../../../include/immutable-object-cache_legacy_options.h
[112/1748] Building CXX object src/CMakeFiles/krbd.dir/krbd.cc.o
FAILED: src/CMakeFiles/krbd.dir/krbd.cc.o
/usr/bin/ccache /usr/bin/clang++-14 -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_NO_TS_EXECUTORS -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/home/jenkins-build/build/workspace/ceph-api/build/src/include -I/home/jenkins-build/build/workspace/ceph-api/src -isystem /opt/ceph/include -isystem /home/jenkins-build/build/workspace/ceph-api/build/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/api/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/ext/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/jaegertracing/opentelemetry-cpp/sdk/include -isystem /home/jenkins-build/build/workspace/ceph-api/src/xxHash -isystem /home/jenkins-build/build/workspace/ceph-api/src/fmt/include -g -Werror -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-unused-private-field -Wno-address-of-packed-member -Wno-unused-function -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register -DCEPH_DEBUG_MUTEX -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++20 -MD -MT src/CMakeFiles/krbd.dir/krbd.cc.o -MF src/CMakeFiles/krbd.dir/krbd.cc.o.d -o src/CMakeFiles/krbd.dir/krbd.cc.o -c /home/jenkins-build/build/workspace/ceph-api/src/krbd.cc
In file included from /home/jenkins-build/build/workspace/ceph-api/src/krbd.cc:44:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/mon/MonMap.h:28:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/mon/mon_types.h:20:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/include/Context.h:19:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/dout.h:29:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/ceph_context.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_proxy.h:6:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config.h:27:
In file included from /home/jenkins-build/build/workspace/ceph-api/src/common/config_values.h:59:
/home/jenkins-build/build/workspace/ceph-api/src/common/options/legacy_config_opts.h:11:10: fatal error: 'rgw_legacy_options.h' file not found
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```

so in this change, we link the related targets to
`legacy-option-headers` as well to fulfill the build dependency.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mds/quiesce: drop remote authpins before waiting for the quiesce lock

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/65802

qa/cephfs/test_quiesce: test proper handling of remote authpins

When a request is blocked on the quiesce lock, it should release
all remote authpins, especially those that make an inode AUTHPIN_FROZEN

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: don't clear `AUTHPIN_FROZEN` until `FROZEN` in rename_prep

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: enhance the `lock path` asok command

* when the quiesce lock is taken by this op, don't consider the inode `quiesced`
* drop all locks taken during traversal
* drop all local authpins after the locks are taken
* add --await functionality that will block the command until locks are taken or an error is encountered
* return the RC that represents the operation result. 0 if the operation was scheduled and hasn't failed so far
* add authpin control flags
** --ap-freeze - to auth_pin_freeze the target inode
** --ap-dont-block - to pass auth_pin_nonblocking when acquiring the target inode locks

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: overdrive fragmenting that's still freezing

Quiesce requires revocation of capabilities,
which is not working for a freezing/frozen nodes.
Since it is best effort, abort an ongoing fragmenting
for the sake of a faster quiesce.

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/65716

revert: mds: provide a mechanism to authpin while freezing

This is a functional revert of a9964a7ccc4394f923fb0f1c76eb8fa03fe8733d
git revert was giving too many conflicts, as the code has changed
too much since the original commit.

The bypass freezing mechanism lead us into several deadlocks,
and when we found out that a freezing inode defers reclaiming
client caps, we realized that we needed to try a different approach.
This commit removes the bypass freezing related changes to clear way
for a different approach to resolving the conflict between quiesce
and freezing.

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

qa/cephfs/test_quiesce: enhance the fragmentation test

Repeatedly quiesce under a heavy balancer load

Fixes: https://tracker.ceph.com/issues/65716
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

rgw: remove Ali Maredia from MAINTAINERS.md

Ali has moved on to another project

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/cephfs: edit "Pinning Subvolumes..."

Edit the section "Pinning Subvolumes and Subvolume Groups" in
doc/cephfs/fs-volumes.rst. This is merely a grammar edit.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57414 from neha-ojha/wip-milestone-squid

.github/milestone.yml: add squid

Reviewed-by: Laura Flores <lflores@redhat.com>

doc/cephfs: edit "Cloning Snapshots" in fs-volumes.rst

Edit the "Cloning Snapshots" section in doc/cephfs/fs-volumes.rst. This
commit represents only a grammar pass. A future commit (and future PR)
will separate this section into subsections by command.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: separate commands into sections

Separate commands so that each command has its own subsection in the
section "FS Subvolumes" in the file doc/cephfs/fs-volumes.rst.
Previously, the list of commands for manipulating subvolumes was one
long, unbroken list and the beginning of one section could easily be
mistaken for the end of the previous section.

Signed-off-by: Zac Dover <zac.dover@proton.me>

src/ptl-tool: allow not pushing branch to ceph-ci

Sometimes we need a branch but don't want to launch builds on shaman for
it. For such cases, provide an option that allows not pushing the branch
to ceph-ci.

Creating a branch that'll only be passed to "teuthology-suite" option
"--suite-branch" is an example of such a case.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

crimson/os/seastore/btree: drop FixedKVNode::child_pos_for_key()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: better UT checks

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: correct lba pointer comments

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cached_extent: EXIST_MUTATION_PENDING also
considered as mutatione_pending

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cached_extent: adjust
CachedExtent::on_replace_prior()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: misc consolidations

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: drop unnecessary friend declarations

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: add CachedExtent::on_rewrite() and
CachedExtent::is_rewrite()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba_manager: extents pass to LBALeafNode::update()
must either has a null parent tracker or point to the current leaf node

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/lba_manager: hide incref_extent(t, addr, delta)

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

mds/queisce-db: collect acks while bootstrapping

Keeping the acks that come in will allow processing them
immediately after the bootstrap is over, avoiding unnecessary
set state transitions.

Fixes: https://tracker.ceph.com/issues/66119
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57011 from tchaikov/wip-cmake-legacy-option-header

cmake: link against legacy-option-headers instead of depending on it

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57542 from zdover23/wip-doc-2024-05-18-ceph-fs-volumes-subvolume-quiesce-squid-note

doc/cephfs: Squid and later - subvolume quiesce

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57383 from athanatos/sjust/wip-65867-user-version

crimson: fix user_version handling

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #56114 from xxhdx1985126/wip-seastore-teuthology-max-test-attr-len

src/test/TestRados: add max-attr-len to control the max length of attributes sent to OSDs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55008 from NitzanMordhai/wip-nitzan-clear-data-digest-for-crc-check

osd: full-object read CRC mismatch due to 'truncate' modifying oi.size w/o clearing 'data_digest'