spawn: remove spawn submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: switch back to boost::asio for spawn() and yield_context

a fork of boost::asio::spawn() was introduced in 2020 with spawn::spawn() from #31580. this fork enabled rgw to customize how the coroutine stacks are allocated in order to avoid stack overflows in frontend request coroutines. this customization was based on a StackAllocator concept from the boost::context library

in boost 1.80, that same StackAllocator overload was added to boost::asio::spawn(), along with other improvements like per-op cancellation. now that boost has everything we need, switch back and drop the spawn submodule

this required switching a lot of async functions from async_completion<> to async_initiate<>. similar changes were necessary to enable the c++20 coroutine token boost::asio::use_awaitable

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/aio: YieldingAioThrottle::async_wait() uses async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/dmclock: async_request() uses async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/d3n: async_read() uses async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/http: async_wait() uses async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/pubsub: Waiter uses async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librados/asio: functions use async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: SharedMutex uses async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crypto/qat: use async_initiate and any_completion_handler

the qat async initiator functions were based on async_completion<> and
its completion_handler member, but the updated boost::asio::yield_context
doesn't provide a completion_handler. switch to the updated
async_initate() method which does work with boost::asio::yield_context

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/d4n: SSDDriver uses async_initiate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: context_pool satisfies ExecutionContext

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56979 from yuvalif/wip-yuval-65337

rgw/notification: start/stop endpoint managers in notification manager

Reviewed-By: cbodley@ibm.com , kchheda3@bloomberg.net

Merge PR #56148 into main

* refs/pull/56148/head:
cephfs_mirror: check m_instance_watcher and m_mirror_watcher before using them

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56699 into main

* refs/pull/56699/head:
qa: ignore `Invalid tag char` warning
qa: ignore `object missing on disk` warning

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #56525 into main

* refs/pull/56525/head:
mgr/stats: initialize mx_last_updated in FSPerfStats

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56135 into main

* refs/pull/56135/head:
cephfs_mirror: increment sync_failures when sync_perms() and sync_snaps() fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55923 from NitzanMordhai/wip-nitzan-add-deprecate-cls-gather

objclass: deprecate cls_cxx_gather

Merge pull request #56983 from NitzanMordhai/wip-nitzan-thrash-erasure-code-crush-4-nodes-8-6

suites/ec-rados-plugin=jerasure-k=8-m=6-crush: roles set

test/rgw/notifications: make sure that http handler is running

fail test if not. to indicate this is a test issue
and not a product bug

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: prevent flooding the http server

in tests that require retries

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: avoid BlockingIOError when logging errors

for example. job: 7697397
in test: yuvalif-2024-05-08_09:55:02-rgw:notifications-wip-yuval-65337-distro-default-smithi

also reduce the side of the error log by sending less objects to the
test_ps_s3_persistent_topic_stats test

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notifications: fix lifecycle tests

* tests were passing only because they were not performings their asserts
* tests are now separated with their own attribute
* their topics are now marked "persistent" to workaround the issue in:
  https://tracker.ceph.com/issues/65645

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: start/stop endpoint managers in notification manager

Fixes: https://tracker.ceph.com/issues/65337
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rados/test: Remove cls_remote_reade since gather deprecated

https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

rgw/notifications: cleanup includes and unused parameters

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: enable notifications debug logs in vstart

as well as in:
- multisite tests (used for notification v2 migration tests)
- the qa suites running notifications

enable lifecycle logs in notification tests: for the lc notification test cases

this is needed after: 429967917b4f9b5f71214fa3c7718e780728249b

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57421 from zdover23/wip-doc-2024-05-12-dev-developer-guide-dash-devel

doc/dev: s/to asses/to assess/

Merge pull request #57420 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x-followup

doc/cephfs: edit fs-volumes.rst (1 of x) followup

doc/dev: s/to asses/to assess/

s/to asses/to assess/ in doc/deve/developer_guide/dash-devel.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: edit fs-volumes.rst (1 of x) followup

Include the suggestions for improving doc/cephfs/fs-volumes.rst made by
Anthony D'Atri here
https://github.com/ceph/ceph/pull/57415#discussion_r1597362110

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57415 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x

doc/cephfs: edit fs-volumes.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit fs-volumes.rst (1 of x)

Edit doc/cephfs/fs-volumes to the section "FS Subvolumes" (but not
including the section "FS Subvolumes").

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57214 from adk3798/cephadm-ignore-stray-on-upgrade

qa/cephadm: ignore CEPHADM_STRAY_DAEMON in upgrade tests

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57205 from SidGilad/bug/65739_support_cgroup_split_removal_for_adopt

mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56735 from samarahu/wip-rgw-d4n-long-descriptions

d4n: Add more in-depth config descriptions and improve naming

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Samarah Uriarte <samarah.uriarte@ibm.com>

Merge pull request #56585 from ivancich/wip-upgrade-arrow+flight-15.0.2

rgw: upgrade Apache Arrow submodule to 15.0.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57259 from pdvian/wip-fix-unfoundev-progress

mgr/cephadm: Fix unfound progress events

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57182 from adk3798/nvmeof-config-fix

mgr/cephadm:  update default NVMEoF container image version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57181 from adk3798/cleanup-iscsi-nvmeof-keyring

mgr/cephadm: cleanup iscsi and nvmeof keyrings upon daemon removal

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57096 from phlogistonjohn/jjm-smb-output-fmt

mgr/smb: disable JSON/YAML key sorting in favor of ordered dicts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57089 from adk3798/test-cephadm-timeout-ignore-refresh-failed

qa/cephadm: ignore CEPHADM_REFRESH_FAILED on timeout test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57080 from phlogistonjohn/jjm-teuth-pull-img

qa/tasks/cephadm: fix pulling containers from private registries

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57042 from adk3798/host-drain-test-ignore-stray-host

qa/cephadm: ignore stray warnings on host drain test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56970 from adk3798/cephadm-ignore-orch-paused

qa/cephadm: ignore CEPHADM_PAUSED on test_orch_cli test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56963 from adk3798/nfs-cluster-ls-json-pretty

mgr/pybind/object_format: fix json-pretty being marked invalid

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56957 from adk3798/no-stray-recent-removal

 mgr/cephadm: don't mark daemons created/removed in the last minute as stray

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56823 from adk3798/autotune-testing

mgr/cephadm: additional debug logging for autotuner

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56802 from clwluvw/rgw-mdsearch

rgw: allow disabling mdsearch APIs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57372 from ceph/wip-lusov-quiesce-docs

doc/cephfs: expand the role of quiesce in consistent backups

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57277 from tchaikov/test_election-free-tracker

test/mon/test_election: free ConnectionTracker if blocked

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #56493 from igomon-bloomberg/wip_kafka_sasl_username_password_attrs

rgw/s3-notifications: use user-name/password topic attributes for SASL authentication

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57075 from cbodley/wip-rgw-ldap-derr

rgw: don't print error when ldap isn't configured

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #57083 from ivancich/wip-optimize-bucket-check-mulitparts

rgw: make incomplete multipart upload part of bucket check efficient

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57098 from kchheda3/wip-dont-log-passwd

rgw/notification: Do not log endpoint as it could contain broker user-id & password.

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw: update s3select submodule addressing arrow 15.0.0 compilation issues

Update arrow submodule.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/cephfs: expand the role of quiesce in consistent backups

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/65881

Merge pull request #57331 from ronen-fr/wip-rf-tier0

tests/librados: do not use '0' as a string

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57155 from mkogan1/wip-rgw-ssl-ec

rgw/beast: fix crash observed in SSL stream.async_shutdown()

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57360 from cbodley/wip-neorados-rm-completion.h

neorados: remove unused symlink to completion.h

Reviewed-by: Adam Emerson <aemerson@redhat.com>

objclass: deprecate cls_cxx_gather

cls_cxx_gather is not maintained and having issues with retry.
since there is no current use of it, we will deprecate it.

Fixes: https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #55773 from weirdwiz/main

exporter: handle exceptions gracefully

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #56865 from zmc/cmake-sccache-max-jobs

cmake: Intelligently set job limits for sccache

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

neorados: remove unused symlink to completion.h

this header is unused after ea67f3dee2a3f8fcdcbb0bc0e80e38ec70378f05
replaced uses of ceph::async::Completion with
boost::asio::any_completion_handler

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55437 from tchaikov/ehh-upstart

debian: do not use upstart to start/stop services

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56238 from SMIL-Infra/journald-fix

common/Journald: fix alias and mis-align

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55560 from tchaikov/systemd-sans-clsuter-name

systemd: do not pass --cluster option

Reviewed-by: John Mulligan <jmulligan@redhat.com>

exporter: handle exceptions gracefully

ceph exporter crahes, and fails to handle exceptions in
dump_asok_metrics(). add try and catch blocks to handle the exceptions gracefully.

Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>

Merge pull request #55007 from shminjs/fix-wrong-osd-op-flag-name

osd/osd_types: fix spelling of osd_op_flag_name

Merge PR #56810 into main

* refs/pull/56810/head:
common/StackStringStream: update pointer to newly allocated memory in overflow()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #57330 from ronen-fr/wip-rf-iam-rgw0

tests/rgw: fix ints returned where a string is expected

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57258 from cbodley/wip-qa-rgw-d4n-tests

qa/rgw/d4n: run ceph_test_d4n_ tests

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge PR #57192 into main

* refs/pull/57192/head:
PendingReleaseNotes: add note on the client incompatibility health warning and feature bit
doc/cephfs: add client_mds_auth_caps client feature bit
doc/cephfs: add missing client feature bits
doc/cephfs: document MDS_CLIENTS_BROKEN_ROOTSQUASH health error
qa: add tests for MDS_CLIENTS_BROKEN_ROOTSQUASH
mds: raise health warning if client lacks feature for root_squash
mon/MDSMonitor: add note about missing metadata inclusion
mds: check relevant caps for fs include root_squash
mds: refactor out fs_name match in MDSAuthCaps
qa: test for root_squash with multiple caps
qa: pass kwargs to mount from remount
qa: simplify update_attrs and only update relevant keys
client: allow overriding client features

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57298 from zdover23/wip-doc-2024-05-07-mgr-dashboard-overview

doc/mgr: edit "Overview" in dashboard.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #57166 into main

* refs/pull/57166/head:
qa: make quiesce ops dump world readable
qa: use specific ops/cache dump file names

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57165 into main

* refs/pull/57165/head:
qa: ignore variation of PG_DEGRADED health warning

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57226 from ronen-fr/wip-rf-unused-cache

common: mark assert-only variables as unused

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tests/librados: do not use '0' as a string

not even as a default string parameter to a
function.

C++23 does not allow int-to-string conversions,

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

tests/rgw: fix ints returned where a string is expected

Modifying test_rgw_iam_policy.cc to avoid C++23
compilation errors. C++23 does not allow int-to-string
conversions, and '0' cannot be returned where a string
is expected.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

PendingReleaseNotes: add note on the client incompatibility health warning and feature bit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: add client_mds_auth_caps client feature bit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: add missing client feature bits

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: document MDS_CLIENTS_BROKEN_ROOTSQUASH health error

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add tests for MDS_CLIENTS_BROKEN_ROOTSQUASH

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: raise health warning if client lacks feature for root_squash

Rather than evict all clients lacking this feature bit, raise a health error
that pushes the administrator to address it. This avoids the surprise of having
all affected clients suddenly evicted in the cluster.

Fixes: https://tracker.ceph.com/issues/65733
Fixes: 954ed30
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MDSMonitor: add note about missing metadata inclusion

There is a "client_count" metadata on the health warning that apparently was
intended to be used for aggregating warnings but never was. Add a TODO item for
that.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: check relevant caps for fs include root_squash

When denying client reconnects because the MDS caps include root_squash and the
client features do not include CEPHFS_FEATURE_MDS_AUTH_CAPS_CHECK, ensure those
caps are only for the file system the MDS is joined to.

Fixes: https://tracker.ceph.com/issues/65733
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: refactor out fs_name match in MDSAuthCaps

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test for root_squash with multiple caps

Where the client has root_squash for one cap but not for another. The fs
without root_squash should not necessarily reject the client.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: pass kwargs to mount from remount

So we can pass mntargs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: simplify update_attrs and only update relevant keys

So we can just pass the caller's kwargs to update_attrs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/mgr: edit "Overview" in dashboard.rst

Edit and update the "Overview" section in doc/mgr/dashboard.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56943 from AliMasarweh/wip-alimasa-bn-test

RGW: test for exact match if the test is not persistent

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge PR #54433 into main

* refs/pull/54433/head:
CephContext: acquire _fork_watchers_lock in notify_post_fork()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa: make quiesce ops dump world readable

Fixes: https://tracker.ceph.com/issues/65701
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use specific ops/cache dump file names

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cmake: Intelligently set job limits for sccache

If we are correctly configured for distributed mode, use the cluster's CPU count
instead of ours. If we are configured for sccache but without distributed mode,
inform the user but continue with normal job limits.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #56515 from wanglinke521/wlk-dev

os/bluestore: fix the problem that _estimate_log_size_N calculates the log size incorrectly

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56421 from rhcs-dashboard/fix-regex-rgw-sync-metrics

exporter: fix regex for rgw sync metrics

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

client: allow overriding client features

For testing purposes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>