crimson: Support mclock for crimson

The patch is trying to sync mclock source of crimson
similar to classic osds. Currently the feature is use
by crimson only for background recovery operations but
later we will use it for other OSD operations also.To use
the same user need to configure crimson_osd_scheduler_concurrency
parameter for osd and the schedule is enabled for the recovery
related operations.

Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>

Merge pull request #62176 from anthonyeleven/weofferedthereleasesorder

doc/releases: Add ordering comment to releases.yml

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61224 from CookiePLMonster/patch-22

dokan: fix a Y2038 bug by replacing `Int32x32To64` with multiplication

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #61753 from aainscow/flat_maps

common: Introduce efficient map and set structure useful for EC.

Merge pull request #62074 from ronen-fr/wip-rf-more4-keys

mon,mgr,auth,client: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62133 from ronen-fr/wip-rf-more6-keys

tests: replace obsolete get_tracked_conf_keys()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62173 from guits/fix-70363

ceph-volume: allow zapping partitions on multipath devices

Merge pull request #61669 from badone/wip-remove-openstack.yaml

qa: Remove openstack.yaml file which is no longer relevant

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Laura Flores <lflores@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

doc/releases: Add ordering comment to releases.yml

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #61624 from VallariAg/wip-nvmeof-ceph-status

mon: Add nvmeof group/gateway name in  "ceph -s"

Merge pull request #62150 from ifed01/wip-ifed-fix-discard-op

blk/kerneldevice: count ops not bytes in discard_op perf counter

Reviewed-by: guyite@bytedance.com

ceph-volume: allow zapping partitions on multipath devices

ceph-volume refuses to zap a device if it is a partition on a multipath
device due to an overly strict condition. This change ensures that only
full mapper devices (excluding partitions) are blocked from being zapped,
allowing partitions on multipath devices to be processed correctly.

Fixes: https://tracker.ceph.com/issues/70363
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #61651 from anthonyeleven/improve-radosgw-layout

doc/radosgw: Improve layout.rst

Merge pull request #62158 from dang/wip-dang-68336

RGW - Temporarily disable List2Inotify1 in the posix bucket cache uni…

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

common: bitset_set and mini_flat_map

The bitset_set is compile-time-fixed size bitmap, which can be accessed using a std::set-like iterator.

The mini_flat_map is similar to a boost::flat_map, except with more restrictions on the size and key, allowing it to make
better use of vectors.

Both have the restriction that the key must be unambiguously castable to/from a small integer. The mini_flat_map has the additional restriction, that the size must be known at construction time.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61933 from NitzanMordhai/wip-nitzan-logmonitor-forward-msg-noreply

LogMonitor: set no_reply for forward MLog commands

Merge pull request #61934 from NitzanMordhai/wip-nitzan-remove-tier-cache-tests-rados-suite

suites/rados: cache tier deprecated, no need to keep the tests for it

Merge pull request #61404 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_plugin_support

common/io_exerciser: ceph_test_rados_io_sequence plugin/profile support

Merge pull request #61348 from Naveenaidu/naveen/wip-tracker-65450

qa/tasks/thrashosds-health: whitelist PG_BACKFILL_FULL

RGW - Temporarily disable List2Inotify1 in the posix bucket cache unit tests, until we can find why inotify fails on CI machines

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #61659 from Matan-B/wip-matanb-cmake-crimson-boost-asio

crimson/CMakeLists: cleanup BOOST_ASIO_DISABLE_CONCEPTS

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61569 from soumyakoduri/wip-skoduri-tier-config

rgw/cloudtier: Correct option ordering in RGWZoneGroupPlacementTier

Reviewed-by: J. Eric Ivancich (ivancich@redhat.com)

Merge PR #62108 into main

* refs/pull/62108/head:
cmake/cephfs: fix options to enable client and dependencies

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #62107 into main

* refs/pull/62107/head:
pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z
pybind/cephfs: increment ref before calling out to c++

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62028 from Matan-B/wip-matanb-crimson-mgrc-con

crimson/mgr/client: fix _send_report empty conn usage

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61647 from mohit84/crimson_throttle

crimson: Trigger a wake operation via acquire_throttle

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62084 from xxhdx1985126/wip-70202

crimson/os/seastore/linked_tree_node: set reserved_ptr to an 8-byte aligned address

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

blk/kerneldevice: count ops not bytes in discard_op perf counter

Fixes: https://tracker.ceph.com/issues/70335
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #62026 from guits/cv-api-lvm-refact

ceph-volume: Refactor LVM object handling

Merge pull request #62012 from VallariAg/wip-vallari-subsystem-limit-alert

monitoring: fix NVMeoFSubsystemNamespaceLimit

Merge pull request #62020 from cbodley/wip-70191

rgw: use object ARN for InitMultipart permissions

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

doc/radosgw: Improve layout.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #62016 from zmc/bwc-for-pipeline

build-with-container: Updates to support ceph-dev-pipeline

build-with-container.py: Add --rpmbuild-extra-args

Signed-off-by: Zack Cerza <zack@cerza.org>

Dockerfile.build: Install sccache

Note that it is not used automatically.

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Add --ceph-version

So that we can build from an extracted tarball, as opposed to a git repo.

Signed-off-by: Zack Cerza <zack@cerza.org>

make-srpm.sh: Skip make-dist if tarball is present

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Add --env-file

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #61939 from phlogistonjohn/jjm-bwc-updates-20250219

build-with-container: fixes and enhancements

cmake/cephfs: fix options to enable client and dependencies

invoking cmake with -DWITH_LIBCEPHFS=OFF fails to configure the client target:

CMake Error at src/client/CMakeLists.txt:13 (target_link_libraries):
  Target "client" links to:

    Boost::locale

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

because the client target is not conditional on WITH_LIBCEPHFS in src/CMakeLists.txt:

add_subdirectory(client)

if(WITH_LIBCEPHFS)

because client is also needed for ceph-fuse, make the client and its
dependencies depend on WITH_LIBCEPHFS OR WITH_FUSE

Signed-off-by: Casey Bodley <cbodley@redhat.com>

pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z

For some newer versions of cython, it appears it requires explicitly specifying
noexcept but old versions of Cython 0.29.Z do not understand that attribute.

See: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#exception-values-and-noexcept

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/cephfs: increment ref before calling out to c++

At the time this construction seemed safe since the caller should have a
reference but it could conveivably be the only ref. We don't want the ref count
to reach 0.

Additionally, catch errors so this callback is genuinely noexcept.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/osd: Code formatting changes for plugin support in ceph_test_rados_io_support

Modified variables used in ceph_test_rados_io_sequence to use snake_case instead of camelCase to be consistent with prior work.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

test/osd: Added support for new plugins

Added new command line options for specifying plugins along with optional readers for them. Added arguments to JSON structures for some optional arguments to encode and decode more information about profiles. Added more choices for random selections of different plugin values and changed structure so profiles are created as part of a pool specifically, encapsulating them.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/test/osd: Split reader classes for program options into a seperate class

Split the program option readers out into a seperate class from the main logic controlling the operationg of ceph_test_rados_io_sequence

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/test/osd: Move ceph_test_rados_io_sequence to its own subfolder

Move files for setting up the cpeh_test_rados_io_sequence to their own subfolder of test/osd so they can be grouped together as more files are added in the future.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

erasure-code: Add minimum granularity interface to EC plugins

Add get_minimum_granularity() function to the EC plugin interface,
which returns the minimum size in bytes that can be used for partial writes.
This is based on the minimum size of coding chunk update
that the particular technique supports.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

Merge pull request #61771 from aainscow/new_io_sequencer_sequences

test: Add new sequences to the IO sequencer.

tests: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61263 from aainscow/legacy_ec

OSD: Refactor EC to have two versions: Legacy and New EC

Merge pull request #62075 from ronen-fr/wip-rf-more5-keys

bluestore, librados, msg: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/mgr/client: add Client::retry_interval()

Could be useful in the future, for now a mere refactor.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/mgr/client:Introduce Client::send()

Client::reconnect nullifies the connection used by the mgr client
before setting a new one.

In this time, we might re-use the nullptr connection due to tasks
that are being run in the background (See: dispatch_in_background).

To avoid this, we had multiple `if (!conn)` checks, some methods
even checked this condition twice to reduce the possibilty of using
undefined the connection.

Instead of introducing an additional check in Client::_send_report,
Introduce Client::send which would be responsible for:
a) Veryfing the connection is set
b) Trying to get a shared access to conn_lock

Client::reconnect will lock conn_lock exclusivly until the
connection is set. If we send is called while reconnecting,
sending will be dropped - same as before.

Fixes: https://tracker.ceph.com/issues/70179
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62009 from Jayaprakash-ibm/wip-bdev-skip-aio-zero-size

blk/kernel : skip AIO thread for zero-sized devices

Merge pull request #59429 from rhcs-dashboard/carbon-forms-rgw

mgr/dashboard: carbonize rgw user and bucket form

test: ceph_test_rados_io_sequence - seq 14 - write object filling gaps

New sequence - write object initially with gaps and fill in the holes
until the object is fully populated. Permutate the step (gap between
writes), start at randomized offsets to produce gaps at the start of
the object as well as between writes

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence - scripting enhancements

Scripting enhancements for interactive mode - ignore lines
begining with #, add a sleep command

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence - add truncate

Add support for truncate I/O in interactive mode

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence - add append with gaps sequence

New I/O sequence - appends to objects by writing past the old
object length so there is a gap before the new written data.
Tests permutations of write length and gap length.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence append sequences

Sequence 11 - sequential appends of fixed size
Sequence 12 - sequentiial overwrite+append of fixed size and different offsets

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence support appends

1. Add append I/O to extend size of object
2. Allow write I/Os to extend size of object
3. Make interactive mode handle EOF gracefully

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

mgr/dashboard: carbonize rgw user and bucket form

Fixes: https://tracker.ceph.com/issues/70119
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #61744 from rhcs-dashboard/mgr-api-test-fixes

qa/tests: retry the api call after making the request

Merge pull request #61533 from clwluvw/prepare-op-cleanup

rgw: remove unused log op info from prepare_op

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61942 from 9401adarsh/wip-adashoka-51855

rgw: add parameter for clients to time out incase they hit wrong url

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #61944 from kchheda3/wip-fix-opslog-crash

rgw/log: Fix crash during shutdown with ops-log enable.

Reviewed-By: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #62018 from cbodley/wip-rgw-admin-threads

radosgw-admin: lower default thread pool size

Reviewed-By: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #61907 from cbodley/wip-70084

rgw: /admin/account APIs accept negative values

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os/seastore/linked_tree_node: set reserved_ptr to an 8-byte
aligned address

Maybe we shouldn't be using an instant integer as the invalid pointer.

Fixes: https://tracker.ceph.com/issues/70202
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #61158 from avanthakkar/fence-old-ranks-smb

cephadm: fence old ranks for SMB service

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #62103 from rhcs-dashboard/revert-pr-61089

mgr/dashboard: Revert PR 61089

Merge pull request #60844 from VinayBhaskar-V/main

librbd: add rbd_diff_iterate3() API to take source snapshot by ID

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #62072 from idryomov/wip-70190

librbd: fix a deadlock on image_lock caused by Mirror::image_disable()

Reviewed-by: Ramana Raja <rraja@redhat.com>

rgw/cloudtier: Correct option ordering in RGWZoneGroupPlacementTier

Two tier-config options (related to `cloud-restore`) were incorrectly added
in the middle of the encoding and decoding methods of RGWZoneGroupPlacementTier.
This modification can cause compatibility issues with older decoders when
attempting to read v2-encoded REST objects.

The fix is to correct the option order and update the decode() function to
properly interpret the structure based on the encoded version.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #61930 from thotz/s3-tests-verify-restore

qa/rgw: add new teuthology for cloud restore s3 tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #61364 from ifed01/wip-ifed-no-compression-plugin-reload

os/bluestore: do cache locally compressor engines ever used.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #61908 from Jayaprakash-ibm/wip-time-measure-fix

os/bluestore : Assigning a named variable to ceph::time_guard to prevent immediate destruction

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@redhat.com>

Merge pull request #61882 from liu-chunmei/fault_range

os/bluestore: fix fault_range for _do_write_v2

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #61843 from ifed01/wip-ifed-fix-expand

os/bluestore: fix bdev expansion and more

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #61502 from Jayaprakash-ibm/wip-refactor-allocator-class

os/bluestore : Refactoring Allocator Class

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Revert "pybind/mgr: Protobuffer file generation for grpc deps."

This reverts commit 0ff89b041dcdb1a3322294a714630859412631d5.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Revert "pybind/mgr: Generate pb2 file in src/pybind/mgr/dashboard/services/proto using the protobuf file from the gateway submodule."

This reverts commit 7f2e1910af05803e5d19c993344a5b7f13040259.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Revert "Move proto generation code block out of if(WITH_MGR_DASHBOARD_FRONTEND)."

This reverts commit 37e633384b7c18893bc0e30a8f3967f6496da00b.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #61596 from cbodley/wip-69741

qa/rgw: fix user cleanup in s3tests task

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/mgr/client: partial move to coroutines

Avoid moving capturing lambdas to coroutines.
Specifically lambdas which are passed to dispatch_in_background.

See: https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rcoro-capture

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/mgr/client: update logger

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/orch: add a test to verify ctdb_status doesn't include GONE node

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge PR #60726 into main

* refs/pull/60726/head:
qa: add test checking 'journal import' from empty dump file
cephfs-journal-tool: fix segfault during 'journal import' from invalid dump file

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60938 from joscollin/wip-fix-try-lock

cephfs_mirror: do not log the previous error

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #61073 from ljflores/tracker-69098

qa/suites/fs/upgrade/mds_upgrade_sequence/tasks: set require-osd-release

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #61076 from joscollin/wip-B69232-test_cephfs_mirror_stats_fail

qa: fix test_cephfs_mirror_stats failure

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/rgw: add new teuthology for cloud restore s3 tests

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

test/pybind/rbd: fix read offset in write zeroes tests

Random data is written and write zeroes is invoked on 0~256, but the
read is done on 256~256.  This means that if write zeroes malfunctions
the test wouldn't catch it (especially in the thick provision case).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: add rbd_diff_iterate3() API to take source snapshot by ID

Allow a diff to start from a non-user snapshot.  This would be used by
"rbd du" command to account for non-user snapshots which are currently
just skipped potentially resulting in underreported space usage and in
other places.

Fixes: https://tracker.ceph.com/issues/65720
Co-authored-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Vinay Bhaskar Varada <vvarada@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61433 from smanjara/wip-fix-tenant-replication

rgw/multisite: fix forwarded requests for tenanted buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61421 from clwluvw/replication-conf

rgw: handle destination bucket as an ARN in ReplicationConfiguration

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #60990 from umesh-mv/wip-umeshmv-67933

rgw/rgw_zone : fixing inline-data default value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61650 from cbodley/wip-rgw-delete-full-try

rgw/rados: enable object deletion at rados pool quota

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

mon: track 'mon_cluster_log_to_file'

'mon_cluster_log_to_file' changes were handled in
LogMonitor::handle_conf_change(), but the LogMonitor was not
registered to track it. This PR adds the registration.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

auth,client: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>