os/bluestore: Improve bluestore_blob_t::add_tail

It could happen that target blob already had empty mapping at the end.
In such case, do not add more empty mappings, just expand exising one.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Make write_v2 not need to use compress_extents, addendum

There was possible case that affected rightmost changed extent.
It could be that 2 mergeable extents were next to each other.
This change melds them together.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Make write_v2 calculate affected range

This reduces size for dirty_range and need_reshard range.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Make write_v2 not need to use compress_extents

Remove need to call compress_extents.
The only place that could produce compressable extents are
_try_reuse_allocated_[l/r] functions.

Create _place_extent_in_blob() function that adds mapping to blob.
If possible, it expands the extent; if not, creates the new one.
The function is used whenever we need new mapping to an existing Blob.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

os/bluestore: Add fault_range_ex

Add fault_range_ex, a sibling to fault_range.
The new function does exactly the same as the original.
The difference is the returned range <start, end> that reflects
range encompassed by shards affected.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #60714 from aclamk/wip-aclamk-bluefs-unittest-async-compaction

os/bluestore: New unit test for BlueFS async compaction

Merge pull request #61810 from Jayaprakash-ibm/wip-bluestore-onode-test-enhancement

os/bluestore : Fill onode test instances with sample data

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #61900 from aclamk/wip-aclamk-bs-fix-get-unused-mask

os/bluestore: implemented bluestore_blob_t::get_unused_mask

Reviewed-by: Chunmei Liu <chunmei.liu@ibm.com>

Merge pull request #61829 from rkachach/fix_issue_69971

mgr/cephadm: Add virtual_ip as a param for mgmt-gateway apply cmd

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #62085 from afreen23/wip-sso

mgr/dashboard: Improve sso role mapping

Merge pull request #60530 from aclamk/wip-aclamk-mute-deferred-test

os/bluestore: Disable invoking unittest_deferred

Merge pull request #62214 from Matan-B/wip-matanb-revert-60753

Revert "os/bluestore: Fix problem with deferred writes replay"

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #61476 from piyushagarwal1411/pool

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in pool.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61474 from piyushagarwal1411/osd

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in osd.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61463 from piyushagarwal1411/host

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in host.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61210 from piyushagarwal1411/cephfs

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in cephfs.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #62126 from vshankar/wip-70287

mds: include deleted file dentry in snapdiff readdir result

Revert "os/bluestore: Add debug hooks"

This reverts commit cbfc24dd7e9b3d13b76c340b60d8d56f13f75ae6.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Deferred writes unit test, replicates corruption"

This reverts commit b2086ef0c96362b6bab4666e635cf26e3f01b40d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Modify _deferred_replay"

This reverts commit c10a794f72613ffdccdd64e24840dfce93174f30.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Split _deferred_replay into 1) apply IO and 2) remove keys"

This reverts commit f2b33bc9719fa5f7bafbbd0a4e591ae350fc85af.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "test/unittest_deferred: debug code"

This reverts commit fff713965ecc5ce4ae80a299d1975cb23b31c3e9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "more debug code"

This reverts commit e1e840628d3842e6664d2f1305a821abf6b631d2.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "fixup"

This reverts commit 5935a0f295a5762bedbe8d64e7d94b88ac81c2cb.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61963 from myoungwon/wip-qa-seastore-rbm-test

crimson/os/seastore: add QA test for seastore with RBM under rados_simple

Merge pull request #61783 from adk3798/cephadm-nvmeof-one-daemon-per-node

mgr/cephadm: block deploying nvmeof daemons of different groups on same host

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61887 from ShwetaBhosale1/fix_issue_70060_port_4721_is_not_opened_while_deploying_agent

mgr/cephadm: Open port 4721 while deployling cephadm agent

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #62189 from idryomov/wip-66419

qa/workunits/rbd: wait for resize to be applied in rbd-nbd

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #62013 from Matan-B/wip-matanb-crimson-coroutine-fixes

crimson: revert coroutine workarounds

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60403 from trociny/wip-68206

rgw: trigger resharding of versioned buckets sooner

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/dashboard: Improve sso role mapping

Fixes https://tracker.ceph.com/issues/70366

- adds support for ISV
- using jmespath expression for fetching roles from payload
- added roles_path parameter in sso enable command as optional argument
- modified roles mapper for sso login

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60753 from aclamk/wip-aclamk-more-deferred-overwrite-fix

os/bluestore: Fix problem with deferred writes replay
This PR had gone through testing, including teuthology, but not in its final form.
But the issue its fixed is urgent.
This PR is merged before final teuthology run.

Merge pull request #62146 from Matan-B/wip-matanb-crimson-wip-69439-v2

crimson: Handle peer replies handling once received

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62118 from rhcs-dashboard/update-proto-files-dashboard

mgr/dashboard: update dashboard nvmeof proto files

Merge pull request #62176 from anthonyeleven/weofferedthereleasesorder

doc/releases: Add ordering comment to releases.yml

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61224 from CookiePLMonster/patch-22

dokan: fix a Y2038 bug by replacing `Int32x32To64` with multiplication

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

qa/workunits/rbd: wait for resize to be applied in rbd-nbd

Implement the same logic as in commit 6f3d0f570f1a ("test/librbd/fsx:
wait for resize to propagate in krbd_resize()").

Fixes: https://tracker.ceph.com/issues/66419
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61753 from aainscow/flat_maps

common: Introduce efficient map and set structure useful for EC.

Merge pull request #62074 from ronen-fr/wip-rf-more4-keys

mon,mgr,auth,client: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62133 from ronen-fr/wip-rf-more6-keys

tests: replace obsolete get_tracked_conf_keys()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62173 from guits/fix-70363

ceph-volume: allow zapping partitions on multipath devices

Merge pull request #61669 from badone/wip-remove-openstack.yaml

qa: Remove openstack.yaml file which is no longer relevant

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Laura Flores <lflores@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

mgr/dashboard: update dashboard nvmeof proto files

Fixes: https://tracker.ceph.com/issues/70330
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/releases: Add ordering comment to releases.yml

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #61624 from VallariAg/wip-nvmeof-ceph-status

mon: Add nvmeof group/gateway name in  "ceph -s"

Merge pull request #62150 from ifed01/wip-ifed-fix-discard-op

blk/kerneldevice: count ops not bytes in discard_op perf counter

Reviewed-by: guyite@bytedance.com

ceph-volume: allow zapping partitions on multipath devices

ceph-volume refuses to zap a device if it is a partition on a multipath
device due to an overly strict condition. This change ensures that only
full mapper devices (excluding partitions) are blocked from being zapped,
allowing partitions on multipath devices to be processed correctly.

Fixes: https://tracker.ceph.com/issues/70363
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #61651 from anthonyeleven/improve-radosgw-layout

doc/radosgw: Improve layout.rst

Merge pull request #62158 from dang/wip-dang-68336

RGW - Temporarily disable List2Inotify1 in the posix bucket cache uni…

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

common: bitset_set and mini_flat_map

The bitset_set is compile-time-fixed size bitmap, which can be accessed using a std::set-like iterator.

The mini_flat_map is similar to a boost::flat_map, except with more restrictions on the size and key, allowing it to make
better use of vectors.

Both have the restriction that the key must be unambiguously castable to/from a small integer. The mini_flat_map has the additional restriction, that the size must be known at construction time.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61933 from NitzanMordhai/wip-nitzan-logmonitor-forward-msg-noreply

LogMonitor: set no_reply for forward MLog commands

Merge pull request #61934 from NitzanMordhai/wip-nitzan-remove-tier-cache-tests-rados-suite

suites/rados: cache tier deprecated, no need to keep the tests for it

Merge pull request #61404 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_plugin_support

common/io_exerciser: ceph_test_rados_io_sequence plugin/profile support

Merge pull request #61348 from Naveenaidu/naveen/wip-tracker-65450

qa/tasks/thrashosds-health: whitelist PG_BACKFILL_FULL

crimson/os/seastore/rbm: add do_with() to prevent accessing unavailable reference

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore/rbm: create a device file if it doesn't exist

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

qa/suite/crimson-rados: add seastore test with RBM

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

os/bluestore: implemented bluestore_blob_t::get_unused_mask

The function was used only by writer v2, but it was returning 0 always.
Now it properly returns the mask of used and unused blob regions.
Changed returned type unused_t (16 bits) -> uint64_t.

Made write_v2 path now properly mark unused.

+ new unittest bluestore_blob_t.get_unused_mask
+ fixed unittest ExtentMapFixture.rain

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

RGW - Temporarily disable List2Inotify1 in the posix bucket cache unit tests, until we can find why inotify fails on CI machines

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #61659 from Matan-B/wip-matanb-cmake-crimson-boost-asio

crimson/CMakeLists: cleanup BOOST_ASIO_DISABLE_CONCEPTS

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61569 from soumyakoduri/wip-skoduri-tier-config

rgw/cloudtier: Correct option ordering in RGWZoneGroupPlacementTier

Reviewed-by: J. Eric Ivancich (ivancich@redhat.com)

Merge PR #62108 into main

* refs/pull/62108/head:
cmake/cephfs: fix options to enable client and dependencies

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #62107 into main

* refs/pull/62107/head:
pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z
pybind/cephfs: increment ref before calling out to c++

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62028 from Matan-B/wip-matanb-crimson-mgrc-con

crimson/mgr/client: fix _send_report empty conn usage

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/pg_backend: rep_op_ret_t to not return acked_peers

Cleanup, no longer used.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: Handle peer replies handling once received

Instead of updating last complete on disk and calling complete_write
in PG::submit_transacion (chained to all_completed).

Move the completion handling earlier as soon as all the peers acked.
This essentially means that we move the handling to
ReplicatedBackend::get_rep_op_reply, unless replication size is 1 -
then we would mark completion in ReplicatedBackend::submit_transaction.

Fixes: https://tracker.ceph.com/issues/69439
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/replicated_backend: Introduce pending_shart_t::last_complete

This will be used (next commit) once we'll call complete_write from
ReplicatedBackend and not from PG as we do now.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: Introduce PG peering helpers

This will be used once we'll call complete_write from
ReplicatedBackend and not from PG as we do now.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: use rep_op_fut_t

rep_op_fut_t is already used in mutiple files.
Instead of intoducing common types in each relevant file,
future work shoud move this type (and similar) to "crimson_types" file.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #61647 from mohit84/crimson_throttle

crimson: Trigger a wake operation via acquire_throttle

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62084 from xxhdx1985126/wip-70202

crimson/os/seastore/linked_tree_node: set reserved_ptr to an 8-byte aligned address

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

blk/kerneldevice: count ops not bytes in discard_op perf counter

Fixes: https://tracker.ceph.com/issues/70335
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #62026 from guits/cv-api-lvm-refact

ceph-volume: Refactor LVM object handling

test/libcephfs: include inode type change test for snapdiff

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: include deleted file dentry in snapdiff readdir result

Fixes: http://tracker.ceph.com/issues/70287
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #62012 from VallariAg/wip-vallari-subsystem-limit-alert

monitoring: fix NVMeoFSubsystemNamespaceLimit

Merge pull request #62020 from cbodley/wip-70191

rgw: use object ARN for InitMultipart permissions

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

doc/radosgw: Improve layout.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #62016 from zmc/bwc-for-pipeline

build-with-container: Updates to support ceph-dev-pipeline

build-with-container.py: Add --rpmbuild-extra-args

Signed-off-by: Zack Cerza <zack@cerza.org>

Dockerfile.build: Install sccache

Note that it is not used automatically.

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Add --ceph-version

So that we can build from an extracted tarball, as opposed to a git repo.

Signed-off-by: Zack Cerza <zack@cerza.org>

make-srpm.sh: Skip make-dist if tarball is present

Signed-off-by: Zack Cerza <zack@cerza.org>

build-with-container.py: Add --env-file

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #61939 from phlogistonjohn/jjm-bwc-updates-20250219

build-with-container: fixes and enhancements

cmake/cephfs: fix options to enable client and dependencies

invoking cmake with -DWITH_LIBCEPHFS=OFF fails to configure the client target:

CMake Error at src/client/CMakeLists.txt:13 (target_link_libraries):
  Target "client" links to:

    Boost::locale

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

because the client target is not conditional on WITH_LIBCEPHFS in src/CMakeLists.txt:

add_subdirectory(client)

if(WITH_LIBCEPHFS)

because client is also needed for ceph-fuse, make the client and its
dependencies depend on WITH_LIBCEPHFS OR WITH_FUSE

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/osd/osd_operation/client_request: remove CompletionEvent from
instance_handle_t::pg_tracking_events

The CompletionEvent is not a rerunnable event

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z

For some newer versions of cython, it appears it requires explicitly specifying
noexcept but old versions of Cython 0.29.Z do not understand that attribute.

See: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#exception-values-and-noexcept

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

pybind/cephfs: increment ref before calling out to c++

At the time this construction seemed safe since the caller should have a
reference but it could conveivably be the only ref. We don't want the ref count
to reach 0.

Additionally, catch errors so this callback is genuinely noexcept.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

test/osd: Code formatting changes for plugin support in ceph_test_rados_io_support

Modified variables used in ceph_test_rados_io_sequence to use snake_case instead of camelCase to be consistent with prior work.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

test/osd: Added support for new plugins

Added new command line options for specifying plugins along with optional readers for them. Added arguments to JSON structures for some optional arguments to encode and decode more information about profiles. Added more choices for random selections of different plugin values and changed structure so profiles are created as part of a pool specifically, encapsulating them.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/test/osd: Split reader classes for program options into a seperate class

Split the program option readers out into a seperate class from the main logic controlling the operationg of ceph_test_rados_io_sequence

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/test/osd: Move ceph_test_rados_io_sequence to its own subfolder

Move files for setting up the cpeh_test_rados_io_sequence to their own subfolder of test/osd so they can be grouped together as more files are added in the future.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

erasure-code: Add minimum granularity interface to EC plugins

Add get_minimum_granularity() function to the EC plugin interface,
which returns the minimum size in bytes that can be used for partial writes.
This is based on the minimum size of coding chunk update
that the particular technique supports.

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

rgw/rados: calculate_preferred_shards takes bool is_versioned

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: scale shards based on versioned() to cover suspended buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: trigger resharding of versioned buckets sooner

Versioned buckets require more keys per object.

Fixes: https://tracker.ceph.com/issues/68206
Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

Merge pull request #61771 from aainscow/new_io_sequencer_sequences

test: Add new sequences to the IO sequencer.

tests: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61263 from aainscow/legacy_ec

OSD: Refactor EC to have two versions: Legacy and New EC