Revert "os/bluestore: Add debug hooks"

This reverts commit cbfc24dd7e9b3d13b76c340b60d8d56f13f75ae6.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Deferred writes unit test, replicates corruption"

This reverts commit b2086ef0c96362b6bab4666e635cf26e3f01b40d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Modify _deferred_replay"

This reverts commit c10a794f72613ffdccdd64e24840dfce93174f30.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Split _deferred_replay into 1) apply IO and 2) remove keys"

This reverts commit f2b33bc9719fa5f7bafbbd0a4e591ae350fc85af.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "test/unittest_deferred: debug code"

This reverts commit fff713965ecc5ce4ae80a299d1975cb23b31c3e9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "more debug code"

This reverts commit e1e840628d3842e6664d2f1305a821abf6b631d2.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "fixup"

This reverts commit 5935a0f295a5762bedbe8d64e7d94b88ac81c2cb.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60753 from aclamk/wip-aclamk-more-deferred-overwrite-fix

os/bluestore: Fix problem with deferred writes replay
This PR had gone through testing, including teuthology, but not in its final form.
But the issue its fixed is urgent.
This PR is merged before final teuthology run.

Merge pull request #62146 from Matan-B/wip-matanb-crimson-wip-69439-v2

crimson: Handle peer replies handling once received

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62118 from rhcs-dashboard/update-proto-files-dashboard

mgr/dashboard: update dashboard nvmeof proto files

Merge pull request #62176 from anthonyeleven/weofferedthereleasesorder

doc/releases: Add ordering comment to releases.yml

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61224 from CookiePLMonster/patch-22

dokan: fix a Y2038 bug by replacing `Int32x32To64` with multiplication

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #61753 from aainscow/flat_maps

common: Introduce efficient map and set structure useful for EC.

Merge pull request #62074 from ronen-fr/wip-rf-more4-keys

mon,mgr,auth,client: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62133 from ronen-fr/wip-rf-more6-keys

tests: replace obsolete get_tracked_conf_keys()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62173 from guits/fix-70363

ceph-volume: allow zapping partitions on multipath devices

Merge pull request #61669 from badone/wip-remove-openstack.yaml

qa: Remove openstack.yaml file which is no longer relevant

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Laura Flores <lflores@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

mgr/dashboard: update dashboard nvmeof proto files

Fixes: https://tracker.ceph.com/issues/70330
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/releases: Add ordering comment to releases.yml

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #61624 from VallariAg/wip-nvmeof-ceph-status

mon: Add nvmeof group/gateway name in  "ceph -s"

Merge pull request #62150 from ifed01/wip-ifed-fix-discard-op

blk/kerneldevice: count ops not bytes in discard_op perf counter

Reviewed-by: guyite@bytedance.com

ceph-volume: allow zapping partitions on multipath devices

ceph-volume refuses to zap a device if it is a partition on a multipath
device due to an overly strict condition. This change ensures that only
full mapper devices (excluding partitions) are blocked from being zapped,
allowing partitions on multipath devices to be processed correctly.

Fixes: https://tracker.ceph.com/issues/70363
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #61651 from anthonyeleven/improve-radosgw-layout

doc/radosgw: Improve layout.rst

Merge pull request #62158 from dang/wip-dang-68336

RGW - Temporarily disable List2Inotify1 in the posix bucket cache uni…

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

common: bitset_set and mini_flat_map

The bitset_set is compile-time-fixed size bitmap, which can be accessed using a std::set-like iterator.

The mini_flat_map is similar to a boost::flat_map, except with more restrictions on the size and key, allowing it to make
better use of vectors.

Both have the restriction that the key must be unambiguously castable to/from a small integer. The mini_flat_map has the additional restriction, that the size must be known at construction time.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61933 from NitzanMordhai/wip-nitzan-logmonitor-forward-msg-noreply

LogMonitor: set no_reply for forward MLog commands

Merge pull request #61934 from NitzanMordhai/wip-nitzan-remove-tier-cache-tests-rados-suite

suites/rados: cache tier deprecated, no need to keep the tests for it

Merge pull request #61404 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_plugin_support

common/io_exerciser: ceph_test_rados_io_sequence plugin/profile support

Merge pull request #61348 from Naveenaidu/naveen/wip-tracker-65450

qa/tasks/thrashosds-health: whitelist PG_BACKFILL_FULL

RGW - Temporarily disable List2Inotify1 in the posix bucket cache unit tests, until we can find why inotify fails on CI machines

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #61659 from Matan-B/wip-matanb-cmake-crimson-boost-asio

crimson/CMakeLists: cleanup BOOST_ASIO_DISABLE_CONCEPTS

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61569 from soumyakoduri/wip-skoduri-tier-config

rgw/cloudtier: Correct option ordering in RGWZoneGroupPlacementTier

Reviewed-by: J. Eric Ivancich (ivancich@redhat.com)

Merge PR #62108 into main

* refs/pull/62108/head:
cmake/cephfs: fix options to enable client and dependencies

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #62107 into main

* refs/pull/62107/head:
pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z
pybind/cephfs: increment ref before calling out to c++

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62028 from Matan-B/wip-matanb-crimson-mgrc-con

crimson/mgr/client: fix _send_report empty conn usage

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/pg_backend: rep_op_ret_t to not return acked_peers

Cleanup, no longer used.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: Handle peer replies handling once received

Instead of updating last complete on disk and calling complete_write
in PG::submit_transacion (chained to all_completed).

Move the completion handling earlier as soon as all the peers acked.
This essentially means that we move the handling to
ReplicatedBackend::get_rep_op_reply, unless replication size is 1 -
then we would mark completion in ReplicatedBackend::submit_transaction.

Fixes: https://tracker.ceph.com/issues/69439
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/replicated_backend: Introduce pending_shart_t::last_complete

This will be used (next commit) once we'll call complete_write from
ReplicatedBackend and not from PG as we do now.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: Introduce PG peering helpers

This will be used once we'll call complete_write from
ReplicatedBackend and not from PG as we do now.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: use rep_op_fut_t

rep_op_fut_t is already used in mutiple files.
Instead of intoducing common types in each relevant file,
future work shoud move this type (and similar) to "crimson_types" file.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #61647 from mohit84/crimson_throttle

crimson: Trigger a wake operation via acquire_throttle

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62084 from xxhdx1985126/wip-70202

crimson/os/seastore/linked_tree_node: set reserved_ptr to an 8-byte aligned address

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

blk/kerneldevice: count ops not bytes in discard_op perf counter

Fixes: https://tracker.ceph.com/issues/70335
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #62026 from guits/cv-api-lvm-refact

ceph-volume: Refactor LVM object handling

Merge pull request #62012 from VallariAg/wip-vallari-subsystem-limit-alert

monitoring: fix NVMeoFSubsystemNamespaceLimit

Merge pull request #62020 from cbodley/wip-70191

rgw: use object ARN for InitMultipart permissions

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

doc/radosgw: Improve layout.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #62016 from zmc/bwc-for-pipeline

build-with-container: Updates to support ceph-dev-pipeline

build-with-container.py: Add --rpmbuild-extra-args

Signed-off-by: Zack Cerza <zack@cerza.org>

Dockerfile.build: Install sccache

Note that it is not used automatically.

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Add --ceph-version

So that we can build from an extracted tarball, as opposed to a git repo.

Signed-off-by: Zack Cerza <zack@cerza.org>

make-srpm.sh: Skip make-dist if tarball is present

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Add --env-file

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #61939 from phlogistonjohn/jjm-bwc-updates-20250219

build-with-container: fixes and enhancements

cmake/cephfs: fix options to enable client and dependencies

invoking cmake with -DWITH_LIBCEPHFS=OFF fails to configure the client target:

CMake Error at src/client/CMakeLists.txt:13 (target_link_libraries):
  Target "client" links to:

    Boost::locale

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

because the client target is not conditional on WITH_LIBCEPHFS in src/CMakeLists.txt:

add_subdirectory(client)

if(WITH_LIBCEPHFS)

because client is also needed for ceph-fuse, make the client and its
dependencies depend on WITH_LIBCEPHFS OR WITH_FUSE

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/osd/osd_operation/client_request: remove CompletionEvent from
instance_handle_t::pg_tracking_events

The CompletionEvent is not a rerunnable event

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z

For some newer versions of cython, it appears it requires explicitly specifying
noexcept but old versions of Cython 0.29.Z do not understand that attribute.

See: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#exception-values-and-noexcept

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/cephfs: increment ref before calling out to c++

At the time this construction seemed safe since the caller should have a
reference but it could conveivably be the only ref. We don't want the ref count
to reach 0.

Additionally, catch errors so this callback is genuinely noexcept.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/osd: Code formatting changes for plugin support in ceph_test_rados_io_support

Modified variables used in ceph_test_rados_io_sequence to use snake_case instead of camelCase to be consistent with prior work.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

test/osd: Added support for new plugins

Added new command line options for specifying plugins along with optional readers for them. Added arguments to JSON structures for some optional arguments to encode and decode more information about profiles. Added more choices for random selections of different plugin values and changed structure so profiles are created as part of a pool specifically, encapsulating them.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/test/osd: Split reader classes for program options into a seperate class

Split the program option readers out into a seperate class from the main logic controlling the operationg of ceph_test_rados_io_sequence

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/test/osd: Move ceph_test_rados_io_sequence to its own subfolder

Move files for setting up the cpeh_test_rados_io_sequence to their own subfolder of test/osd so they can be grouped together as more files are added in the future.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

erasure-code: Add minimum granularity interface to EC plugins

Add get_minimum_granularity() function to the EC plugin interface,
which returns the minimum size in bytes that can be used for partial writes.
This is based on the minimum size of coding chunk update
that the particular technique supports.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

Merge pull request #61771 from aainscow/new_io_sequencer_sequences

test: Add new sequences to the IO sequencer.

tests: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61263 from aainscow/legacy_ec

OSD: Refactor EC to have two versions: Legacy and New EC

Merge pull request #62075 from ronen-fr/wip-rf-more5-keys

bluestore, librados, msg: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/mgr/client: add Client::retry_interval()

Could be useful in the future, for now a mere refactor.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/mgr/client:Introduce Client::send()

Client::reconnect nullifies the connection used by the mgr client
before setting a new one.

In this time, we might re-use the nullptr connection due to tasks
that are being run in the background (See: dispatch_in_background).

To avoid this, we had multiple `if (!conn)` checks, some methods
even checked this condition twice to reduce the possibilty of using
undefined the connection.

Instead of introducing an additional check in Client::_send_report,
Introduce Client::send which would be responsible for:
a) Veryfing the connection is set
b) Trying to get a shared access to conn_lock

Client::reconnect will lock conn_lock exclusivly until the
connection is set. If we send is called while reconnecting,
sending will be dropped - same as before.

Fixes: https://tracker.ceph.com/issues/70179
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62009 from Jayaprakash-ibm/wip-bdev-skip-aio-zero-size

blk/kernel : skip AIO thread for zero-sized devices

Merge pull request #59429 from rhcs-dashboard/carbon-forms-rgw

mgr/dashboard: carbonize rgw user and bucket form

test: ceph_test_rados_io_sequence - seq 14 - write object filling gaps

New sequence - write object initially with gaps and fill in the holes
until the object is fully populated. Permutate the step (gap between
writes), start at randomized offsets to produce gaps at the start of
the object as well as between writes

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence - scripting enhancements

Scripting enhancements for interactive mode - ignore lines
begining with #, add a sleep command

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence - add truncate

Add support for truncate I/O in interactive mode

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence - add append with gaps sequence

New I/O sequence - appends to objects by writing past the old
object length so there is a gap before the new written data.
Tests permutations of write length and gap length.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence append sequences

Sequence 11 - sequential appends of fixed size
Sequence 12 - sequentiial overwrite+append of fixed size and different offsets

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

test: ceph_test_rados_io_sequence support appends

1. Add append I/O to extend size of object
2. Allow write I/Os to extend size of object
3. Make interactive mode handle EOF gracefully

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

mgr/dashboard: carbonize rgw user and bucket form

Fixes: https://tracker.ceph.com/issues/70119
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #61744 from rhcs-dashboard/mgr-api-test-fixes

qa/tests: retry the api call after making the request

Merge pull request #61533 from clwluvw/prepare-op-cleanup

rgw: remove unused log op info from prepare_op

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61942 from 9401adarsh/wip-adashoka-51855

rgw: add parameter for clients to time out incase they hit wrong url

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #61944 from kchheda3/wip-fix-opslog-crash

rgw/log: Fix crash during shutdown with ops-log enable.

Reviewed-By: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #62018 from cbodley/wip-rgw-admin-threads

radosgw-admin: lower default thread pool size

Reviewed-By: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #61907 from cbodley/wip-70084

rgw: /admin/account APIs accept negative values

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os/seastore/linked_tree_node: set reserved_ptr to an 8-byte
aligned address

Maybe we shouldn't be using an instant integer as the invalid pointer.

Fixes: https://tracker.ceph.com/issues/70202
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #61158 from avanthakkar/fence-old-ranks-smb

cephadm: fence old ranks for SMB service

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #62103 from rhcs-dashboard/revert-pr-61089

mgr/dashboard: Revert PR 61089

Merge pull request #60844 from VinayBhaskar-V/main

librbd: add rbd_diff_iterate3() API to take source snapshot by ID

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #62072 from idryomov/wip-70190

librbd: fix a deadlock on image_lock caused by Mirror::image_disable()

Reviewed-by: Ramana Raja <rraja@redhat.com>

rgw/cloudtier: Correct option ordering in RGWZoneGroupPlacementTier

Two tier-config options (related to `cloud-restore`) were incorrectly added
in the middle of the encoding and decoding methods of RGWZoneGroupPlacementTier.
This modification can cause compatibility issues with older decoders when
attempting to read v2-encoded REST objects.

The fix is to correct the option order and update the decode() function to
properly interpret the structure based on the encoded version.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #61930 from thotz/s3-tests-verify-restore

qa/rgw: add new teuthology for cloud restore s3 tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #61364 from ifed01/wip-ifed-no-compression-plugin-reload

os/bluestore: do cache locally compressor engines ever used.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #61908 from Jayaprakash-ibm/wip-time-measure-fix

os/bluestore : Assigning a named variable to ceph::time_guard to prevent immediate destruction

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@redhat.com>

Merge pull request #61882 from liu-chunmei/fault_range

os/bluestore: fix fault_range for _do_write_v2

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #61843 from ifed01/wip-ifed-fix-expand

os/bluestore: fix bdev expansion and more

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #61502 from Jayaprakash-ibm/wip-refactor-allocator-class

os/bluestore : Refactoring Allocator Class

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Revert "pybind/mgr: Protobuffer file generation for grpc deps."

This reverts commit 0ff89b041dcdb1a3322294a714630859412631d5.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Revert "pybind/mgr: Generate pb2 file in src/pybind/mgr/dashboard/services/proto using the protobuf file from the gateway submodule."

This reverts commit 7f2e1910af05803e5d19c993344a5b7f13040259.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Revert "Move proto generation code block out of if(WITH_MGR_DASHBOARD_FRONTEND)."

This reverts commit 37e633384b7c18893bc0e30a8f3967f6496da00b.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #61596 from cbodley/wip-69741

qa/rgw: fix user cleanup in s3tests task

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>