common/async: spawn_throttle wraps call to asio::spawn()

cancellation of the parent must immediately cancel its children, which
only works if the children are on the same executor as the parent

prohibit child coroutines from being spawned on a different executor by
wrapping the call to asio::spawn() in a new spawn_throttle::spawn()
interface

expose an overload for asio::spawn()'s optional StackAllocator argument

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58705 from gitkenan/crimsonDocChanges

doc/dev/crimson: making it more user-friendly, re-arranging chapters and fixing typos

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58275 from NitzanMordhai/wip-nitzn-host-thraser-fix-min-in-checks

suites: host thrasher should check min_in before thrashing host

Merge pull request #58393 from NitzanMordhai/wip-nitzan-old-client-thrasher-slow-ops-ignorelist

suites: thrash old client missing SLOW_OPS in the ignore-list

Merge pull request #58277 from NitzanMordhai/wip-nitzan-upgrade-suites-quincy-ignore-list-adding

suites/upgrade/quincy-x: update the ignore list

Merge pull request #58600 from zdover23/wip-doc-2024-07-16-dev-guide-basic-workflow

doc/dev: add formatting to basic workflow

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58725 from zdover23/wip-doc-2024-07-23-dev-dev-guide-basic-workflow

doc/dev_guide: add needs-upgrade-testing label info

Reviewed-by: Laura Flores <lflores@ibm.com>

doc/dev_guide: add needs-upgrade-testing label info

Add an explanation of the "needs-upgrade-testing" label and explain when
to use it.

Laura is the true author of this text.

Co-authored-by: Laura Flores <lflores@ibm.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/dev/crimson: clarify and rearrange for userability

Signed-off-by: gitkenan <kenan.mth@gmail.com>

Merge pull request #56517 from gukaifeng/doc-amend

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #58353 from pereman2/bad-collection-objectstore

tools/objectstore: check for wrong coll open_collection

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57002 from aclamk/wip-aclamk-bs-storetest-expand-synthetic

Improved structure for objectstore unit tests.

doc/start/os-recommendations: remove 16.2.z support for CentOS 7

Merge pull request #58639 from zdover23/wip-doc-2024-07-17-start-os-recommendations

doc/start: add tested container host oses

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: add tested container host oses

Add a list of OSes that have been tested as container hosts. This commit
is added in response to a request made by Josh Durgin on Slack.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58452 from Matan-B/wip-matanb-seastar-july7

seastar: update submodule

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58256 from NitzanMordhai/wip-nitzan-wait-osd-admin-command

crimson/osd: adding osdmap subscribe

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58700 from kshtsk/wip-fix-unused-in-msg-async-rdma

msg/async/rdma: cleanup unused variables

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #58688 from idryomov/wip-67051

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Reviewed-by: Mykola Golub <mgolub@suse.com>

msg/async/rdma: cleanup unused variables

Fixes: 2fb8be43fa2a758cc7934f7a82a48ad2d47d1112
Signed-off-by: kshtsk <kyrylo.shatskyy@clyso.com>

Merge pull request #58695 from pecastro/correct_mailing_list_instructions

doc: Amend dev mailing list subscribe instructions.

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc: Amend dev mailing list subscribe instructions.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

Merge pull request #58591 from idryomov/wip-aio-completion-prep-for-atomic-wait

librbd: fix inconsistency between AioCompletion is_complete() and wait_for_complete()

Reviewed-by: N Balachandran <nibalach@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58180 from guojidan/format

rbd: add the validate of the format and clone_format

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Commit 40f6f5224bce ("qa/workunits/rbd: fix issues in
luks-encryption.sh") did the right thing for reads, which solved
most of the issue.  However, it actually made a step in the opposite
direction for writes -- depending on the RBD cache settings, rbd-nbd
virtual devices can behave as physical devices with a volatile write
cache, so fsync is required.

While at it, involving O_DIRECT for reads isn't needed outside of
test_encryption_format().

Fixes: https://tracker.ceph.com/issues/67051
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52639 from ljflores/wip-tracker-58925

qa: suppress Leak_StillReachable mon leak in centos 9 jobs

Merge pull request #58671 from zdover23/wip-doc-2024-07-19-start-get-involved

doc/start: update mailing list links

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #58678 from cbodley/wip-doc-cephadm-quay-ci

doc/cephadm/upgrade: ceph-ci containers are hosted at quay.ceph.io

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

doc/cephadm/upgrade: ceph-ci containers are hosted by quay.ceph.io

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57737 from Svelar/asan_rgw_amqp

rgw/amqp: lock erase and create connection before emplace

Merge pull request #58662 from yuvalif/wip-yuval-64690-again

test/rgw/amqp: skip TestAMQP.MaxConnections test

Merge pull request #58656 from rhcs-dashboard/rgw-sync-policy-routing-fix

mgr/dashboard: Rgw sync policy routing fix

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56331 from rhcs-dashboard/update-mon-stacks

mgr/cephadm: bump monitoring stacks versions

doc/start: update mailing list links

Update the User and Dev mailing list subscription links. This commit is
made in response to a request from Dan van der Ster.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58636 from afreen23/wip-nvmeof-namespace

mgr/dashboard: Add namespaces views in dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58519 from smanjara/wip-fix-copy-obj-replication

rgw/multisite: don't retain RGW_ATTR_OBJ_REPLICATION_TRACE attr on copy_object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56937 from Svelar/asan_bluestore_type

bluestore/bluestore_types: avoid heap-buffer-overflow in another way to keep code uniformity

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56352 from ifed01/wip-ifed-many-many-extents-read

blk/aio: fix long batch (64+K entries) submission.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #54988 from Matan-B/wip-matanb-mapgap-fix

osd/OSD: rewrite track_pools_and_pg_num_changes logic

Reviewed-by: Samuel Just <sjust@redhat.com>

qa: suppress Leak_StillReachable mon leaks

These leaks came about when we changed the distro
from centos 8 to centos 9.

Fixes: https://tracker.ceph.com/issues/61774
Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #57817 from phlogistonjohn/jjm-smb-sqlite

mgr/smb: add sqlite internal store backend for smb mgr module

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58518 from anoopcs9/smb-mgr-updates

mgr/smb: improvements to smb mgr module and docs

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: Add namespaces views in dashboard

Fixes https://tracker.ceph.com/issues/66990

- list namespaces
- create namespaces
- edit namespaces
- delete namespaces
- update NQN regex

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #58644 from rhcs-dashboard/fix-66994-main

mgr/dashboard: Add a new chart for replication delta per shard in rgw sync overview grafana dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58649 from cbodley/wip-64173

rgw/rados: set_attrs() falls back to existing attrs for index update

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: rgw sync policy routing fixes

Fixes: https://tracker.ceph.com/issues/66977
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58173 from ronen-fr/wip-rf-targets-j13

osd/scrub: no shared scrub-job ownership between PGs and the scrub queue

Reviewed-by: Samuel Just <sjust@redhat.com>

test/rgw/amqp: skip TestAMQP.MaxConnections test

not really a fix, mainly a workaround until we do
the amqp refactoring: https://tracker.ceph.com/issues/67011

Fixes: https://tracker.ceph.com/issues/64690
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #58555 from rhcs-dashboard/rgw-multisite-sync-pipe

 mgr/dashboard: RGW multisite sync pipe

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #58648 from afreen23/wip-nvmeof-listener

Allow listeners creation and deletion

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58551 from xxhdx1985126/wip-66924

crimson/os/seastore/transaction_manager: consider inconsistency between backrefs and lbas acceptable when cleaning segments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58441 from rhcs-dashboard/add-nfs-export-route

mgr/dashboard: add NFS export button for subvolume/ grp

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57591 from aclamk/wip-aclamk-denc-compat-check

common, os/bluestore: Fix lack of checking for compat on DENC_START

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Create and delete listeners

Fixes https://tracker.ceph.com/issues/66996

- list listener under subsystems
- delete listener
- create listener

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: RGW multisite sync pipe

Fixes: https://tracker.ceph.com/issues/66926
Signed-off-by: Naman Munet <nmunet@redhat.com>

rgw/rados: set_attrs() falls back to existing attrs for index update

set_attrs() needs the ACLOwner, etag, content-type and storage class for
the bucket index update. if those attrs aren't being updated, consult
their existing attrs in state->attrset

Fixes: https://tracker.ceph.com/issues/64173
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: add NFS export route from subvolume/subvolume grp

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: Add a new chart for replication delta per shard in rgw sync overview grafana dashboard

Fixes: https://tracker.ceph.com/issues/66994
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #57722 from sajibreadd/wip-62500

os/bluestore: Warning added for slow operations and stalled read

librbd: get rid of AIO_STATE_CALLBACK in AioCompletion

After commit 002afa0fe375 ("librbd: avoid using lock within AIO
completion where possible"), the only method whose behavior would
change if AIO_STATE_CALLBACK is removed is is_complete() and it
actually needs fixing anyway: because of state != AIO_STATE_PENDING
test, is_complete() returns true both for AIO_STATE_CALLBACK and
AIO_STATE_COMPLETE, while wait_for_complete() still blocks on
AIO_STATE_CALLBACK and returns only on AIO_STATE_COMPLETE.  These
methods back public APIs, so this inconsistency is exposed to users.

If we move to setting state to AIO_STATE_COMPLETE at the top of
mark_complete_and_notify() (i.e. before event socket notification), the
transient state for callbacks can be eliminated entirely and the
inconsistency goes away.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58571 from zdover23/wip-doc-2024-07-13-cephfs-mount-using-fuse

doc/cephfs: improve mount-using-fuse

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: improve mount-using-fuse

Improve the first twenty percent of doc/cephfs/mount-using-fuse.rst.
This commit changes the text so that it is clearer that FUSE is an
alternative to mounting CephFS with the kernel driver. This commit also
links to the kernel driver page from the "FUSE-mount" page.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/transaction_manager: consider inconsistency between
backrefs and lbas acceptable when cleaning segments

Consider the following scene:
1. Trans.A writes the final record of Segment S, in which it overwrite
   another extent E in the same segment S;
2. Before Trans.A "complete_commit", Trans.B tries to rewrite new
   records and roll the segments, which closes Segment S;
3. Before Trans.A "complete_commit", a new cleaner Transaction C tries
   to clean the segment;

In this scenario, C might see a part of extent E's laddr space mapped
to another location within the same segment S. This is actually a valid
case.

Fixes: https://tracker.ceph.com/issues/66924
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57169 from jagombar/fixCpatch

cpatch: Search base image when patching files.

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #57906 from kamoltat/wip-ksirivad-stretch-mode-netsplit-test

mon/ElectionLogic: tie-breaker mon ignore proposal from marked down mon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #57592 from Svelar/asan_ec_shec_t

ec: free plugin memory when it destructs and repeated loads

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #57201 from NitzanMordhai/wip-nitzan-standalone-pg-split-merge-daemon-commands

test: ceph daemon command with asok path

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56982 from NitzanMordhai/wip-nitzan-neorados-watch-notify-test-reconnect

test/neorados: timeout test won't reconnect at timeout

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #56605 from myoungwon/wip-65241

osd: fix use-after-free before calling finish_unreadable_object()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58354 from ivancich/wip-appropriate-min-objs-shard

rgw: make sure min objs per shard is appropriate

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58442 from clwluvw/getlocation-contenttype

rgw: add missing content-type for RGWGetBucketLocation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57640 from cbodley/wip-rgw-beast-realm-reload

rgw/beast: realm reload closes connections so requests drain quickly

Reviewed-by: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #58567 from ivancich/wip-reshard-cleanups

rgw: add reshard status to bucket stats plus minor misc items

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

mgr/dashboard: nvmf apis broken for insecure channel

because when no certificate, it raises an exception which stops the api call
abruptly.

Signed-off-by: Nizamudeen A <nia@redhat.com>

osd/scrub: fix job requeue conditions

Previous commits handled the following two cases correctly:
- requeueing a scrub job while the OSD is still the primary, and
- not restoring the scrub job to the queue if the PG is not there;
Here we handle the missed scenario: the PG is there (we were able to
lock it), but is no longer the primary.

Also - a configuration change must not cause a re-queue of a
scrub-job for a PG that is in the middle of scrubbing.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

test/osd: delete test_scrub_sched (for now)

removing this unit-test, as it must be rewritten to match the
changes in the implementation of the scrub scheduling mechanism,
and that implementation is still in flux.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: making osd_scrub_sched.cc compatible with Clang-14

Removing the use of C++ ranges, as
Clang versions below 16 are buggy when libstd++ ranges are used.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/standalone/scrub: fix osd-scrub-test.sh

following changes in scrub code

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: scheduling the next scrub following scrub completion

or after an aborted scrub.

To note: one of the important changes in this commit:
merging the functionality of adjust_target_time() &
update_schedule() into a single function - adjust_schedule().

Regarding the handling of aborts:

Most of the time - all that is required following a scrub abort is to
requeue the scrub job - the one that triggered the aborted scrub -
with just a delay added to its n.b..

But we must take into account scenarios where "something" caused the
parameters prepared for the *next* scrub to show higher urgency or
priority. "Something" - as in an operator command requiring immediate
scrubbing, or a change in the pool/cluster configuration.
In such cases - the current requested flags and the parameters of
the aborted scrub must be merged.

Note that the current implementation is a temporary solution, to be
replaced by a per-level updating of the relevant target.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: improve update_scrub_job()

...and on_scrub_schedule_input_change()
by accepting a parameter to control whether the scheduled time for
periodic scrubs that are already due should be updated.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: passing the scrub-job copy through the scrubber

moving the scrubbed sjob copy thru the scrubber - from scrub
session initiation to its termination (or abort - where we use the
handed "old version" of the sjob to update the new one).

Note that in this version - not all the information that was used to
determine the specifics of the initiated scrub is passed to the
scrubber and back. In this half-baked stage of the refactoring,
the resulting implementation handling of corner cases, still using
the "planned scrub" flags, is less than optimal.

The next step (dual targets, replacing the 'planned scrub' flags with
specific attributes in the scheduling target) fixes this.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: fix adjust_target_time()

... to use populate_config_params() output

Also: fix determine_scrub_time() to use the same already-available
data. determine_scrub_time() renamed to determine_initial_schedule(),
to better reflect its purpose.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: modify scrub registration implementation details

following the change to the queue into holding a copy of the
ScrubJob, the registration process - initiated by
schedule_scrub_with_osd() - can now be simplified.

adjust_target_time() is relocated as is. It will be refactored
in the next commit.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: the scrub queue now holds a copy of the ScrubJob

... and not a shared pointer to it.
The PG's entry in the scrub queue and the scrub-job object that
is part of the scrubber are now separate entities, that can be
modified separately - each with its own locking mechanism.

the interaction between OsdScrub (the OSD's scrub scheduler
object) and the scrub queue during scrub initiation is simplified:
instead of fetching from the queue a list of scrub targets
possibly ready to be scrubbed, the OsdScrub now picks only one
candidate - the top of the queue.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: Modify osd_scrub_sched.h high-level diagram

to convey the change in the scheduling data ownership: no longer
a scrub-job object shared between the Scrubber and the OSD
scrub queue. Instead - the scrub queue holds a copied snapshot
of the scheduling data.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

include/denc.h: Review fixes

- Added note to release-checklist
- Created specialized DENC_START_OSD_REQID
- Use CEPH_RELEASE_SQUID (19) value

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #57120 from NUABO/tancz

os/bluestore: allow use BtreeAllocator

Merge pull request #58544 from rhcs-dashboard/rgw-multisite-sync-pipe-and-flow

mgr/dashboard: RGW multisite sync flow

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

tools/objectstore: check for wrong coll open_collection

Fixes: https://tracker.ceph.com/issues/66802
Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #58334 from aclamk/wip-aclamk-patch-bluerocksenv

os/bluestore: Fix BlueRocksEnv attempts to use POSIX

mgr/dashboard: RGW multisite sync flow

Fixes: https://tracker.ceph.com/issues/66915
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #57715 from rhcs-dashboard/add-rgw-multisite-wizard

mgr/dashboard: add a wizard to setup rgw multisite replication

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58596 from batrick/superuser-doc

doc: clarify superuser note for ceph-fuse

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55265 from vshankar/wip-58878

osdc/Journaler: derr log when header update get (possibly) overwritten out-of-order

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #58517 from phlogistonjohn/jjm-cephadm-ranked-by-spec

mgr/cephadm: allow services to be conditionally ranked

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58514 from phlogistonjohn/jjm-smb-fix-show-no-shares

mgr/smb: fix ceph smb show when a cluster has not associated shares

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #58379 from adk3798/no-cgroups-split-bootstrap

cephadm: turn off cgroups_split setting  when bootstrapping with --no-cgroups-split

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57853 from rhcs-dashboard/cache-subvol-smb

mgr/smb: cache subvolume (& subvolume group) path lookups

Reviewed-by: John Mulligan <jmulligan@redhat.com>

doc/dev: add formatting to basic workflow

Add some formatting to a few sections near the end of
doc/dev/developer_guide_basic-workflow.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58566 from ljflores/wip-tracker-66881

qa/workunits/cephtool: add extra privileges to cephtool script