ceph-volume: migrate unit tests from 'mock' to 'unittest.mock'
unit tests in ceph-volume was still using the external 'mock' library
for unit tests, which is unnecessary since 'unittest.mock' is part
of the Python standard library (available since Python 3.3).
This commit updates all imports to use 'unittest.mock' instead,
ensuring better maintainability and removing the need for an extra
dependency.
This refactors `get_physical_osds()`.
The calculation of `data_slots` is now more concise. The handling of
`dev_size`, `rel_data_size`, and `abs_size` is standardized.
The initialization of `free_size` is moved outside the loop
for clarity. Redundant checks and assignments are removed to simplify
the code.
ceph-volume: support splitting db even on collocated scenario
This change enables ceph-volume to create OSDs where the DB is
explicitly placed on a separate LVM partition, even in collocated
scenarios (i.e., block and DB on the same device).
This helps mitigate BlueStore fragmentation issues.
Given that ceph-volume can't automatically predict a proper default size for the db device,
the idea is to use the `--block-db-size` parameter:
Passing `--block-db-size` and `--db-devices` makes ceph-volume create db devices
on dedicated devices (current implementation):
```
Total OSDs: 2
Type Path LV Size % of device
----------------------------------------------------------------------------------------------------
data /dev/vdb 200.00 GB 100.00%
block_db /dev/vdd 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 200.00 GB 100.00%
block_db /dev/vdd 4.00 GB 2.00%
```
Passing `--block-db-size` without `--db-devices` makes ceph-volume create a separate
LV for db device on the same device (new behavior):
```
Total OSDs: 2
Type Path LV Size % of device
----------------------------------------------------------------------------------------------------
data /dev/vdb 196.00 GB 98.00%
block_db /dev/vdb 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 196.00 GB 98.00%
block_db /dev/vdc 4.00 GB 2.00%
```
This new behavior is supported with the `--osds-per-device` parameter:
```
Total OSDs: 4
Type Path LV Size % of device
----------------------------------------------------------------------------------------------------
data /dev/vdb 96.00 GB 48.00%
block_db /dev/vdb 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdb 96.00 GB 48.00%
block_db /dev/vdb 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 96.00 GB 48.00%
block_db /dev/vdc 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 96.00 GB 48.00%
block_db /dev/vdc 4.00 GB 2.00%
```
Nizamudeen A [Thu, 20 Feb 2025 09:10:28 +0000 (14:40 +0530)]
Merge pull request #61225 from rhcs-dashboard/rgw-ratelimit-integration1
mgr/dashboard: Rgw ratelimit feature for user and bucket
Reviewed-by: Aashish Sharma <aasharma@redhat.com> Reviewed-by: Ankush Behl <cloudbehl@gmail.com> Reviewed-by: Mark Nelson <mnelson@redhat.com> Reviewed-by: Nizamudeen A <nia@redhat.com> Reviewed-by: Naman Munet <nmunet@redhat.com>
Ronen Friedman [Thu, 20 Feb 2025 08:01:44 +0000 (10:01 +0200)]
Merge pull request #61812 from ronen-fr/wip-rf-fstab
qa/standalone: do not use /etc/fstab as an always-there bytes source Reviewed-by: Patrick Donnelly <pdonnell@redhat.com> Reviewed-by: Ilya Dryomov <idryomov@gmail.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Ronen Friedman [Thu, 13 Feb 2025 16:53:53 +0000 (10:53 -0600)]
qa/standalone: do not use /etc/fstab as an always-there bytes source
Multiple tests use /etc/fstab when a small data file is
required as input. After all, as some comments say:
# something that is always there
Alas - it's not always there. Not in containers.
Replacing with a newly-created temporary file filled with
random bytes.
For completeness - replacing similar references to
/etc/resolv.conf (as a source for random objects) in
the standalone tests, too.
See: https://github.com/llvm/llvm-project/commit/f46d1463b835560d90ad3ac02b63c771e4ebe566
```
common/interruptible_future.h:1182:16: error: a template argument list is expected after a name prefixed by the template keyword [-Wmissing-template-arg-list-after-template-kw]
1182 | ).template handle_interruption(std::move(efunc));
| ^
1 error generated.
```
Matan Breizman [Mon, 17 Feb 2025 10:00:34 +0000 (10:00 +0000)]
crimson/common/operation: fix decltype usage
```
ceph/src/crimson/common/operation.cc
In file included from /home/matan/ceph/src/crimson/common/operation.cc:4:
/home/matan/ceph/src/crimson/common/operation.h:312:31: error: no member named 'events' in 'AggregateBlockingEvent<T>'
312 | typename decltype(event.events)::iterator iter;
| ~~~~~ ^
1 error generated.
```
Zac Dover [Tue, 18 Feb 2025 13:44:15 +0000 (23:44 +1000)]
doc/mgr: edit grammar and formatting of rgw.rst
Improve the grammar and correct the formatting of the "Upgrading root ca
certificates" procedure that was added to the documentation in https://github.com/ceph/ceph/pull/61867
Naman Munet [Wed, 22 Jan 2025 10:59:20 +0000 (16:29 +0530)]
mgr/dashboard: Add confirmation textbox for resource name on delete action
Before:
=====
User was able to delete a single or multiple critical resources like ( images, snapshots, subvolumes, subvolume-groups, pools, hosts , OSDs, buckets, file system, services ) by just clicking on a checkbox.
After:
=====
User now has to type the resource name that they are deleting in the textbox on the delete modal, and then only they will be able to delete the critical resource.
Also from now onwards multiple selection for deletions of critical resources is not possible. Hence, user can delete only single resource at a time. On the other side, non-critical resources can be deleted in one go.
The BucketCacheFixtureInotify1.List2Inotify1 test was failing due to the
count being wrong. Fix the count, and flesh out the test case to test
the cache for deleting and adding. Also, move to per-test fixture, so
tests can't interfere with eachother due to inotify latency.
Fixes: https://tracker.ceph.com/issues/68336 Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>
Ilya Dryomov [Mon, 17 Feb 2025 11:41:51 +0000 (12:41 +0100)]
test/rbd_mirror: flush watch/notify callbacks in TestImageReplayer
TestImageReplayer establishes its own (i.e. outside of the SUT code)
watch on the header of the remote image to be able to synchronize the
execution of the test with certain notifications. This watch is
established before the remote image is opened and is teared down until
after the remote image is closed but while the image replayer is still
running. The flush that is part of image close sequence thus isn't
guaranteed to cover all callbacks, especially for snapshot-based
mirroring where UnlinkPeerRequest spawned from Replayer::unlink_peer()
generates a notification on the remote image for each completed unlink.
Since TestImageReplayer further immediately deletes C_WatchCtx, pretty
much any test can segfault when C_WatchCtx::handle_notify() is invoked
by TestWatchNotify infrastructure. Because it's a virtual method, the
segfault often involves a completely bogus instruction pointer:
Ronen Friedman [Tue, 11 Feb 2025 12:52:38 +0000 (06:52 -0600)]
osd/scrub: clean-up scrub-store IDs generation
The Scrub Store IDs (used to identify OMAP entries used for storing
object-specific error data) are generated by attaching some prefix to
the text representation of virtual (invented) hobjects.
This commit cleans up the code by moving the name generation
process (creating temporary un-needed objects, then
extracting the to_str() representation of their identity)
into separate functions.
N Balachandran [Sat, 15 Feb 2025 13:26:31 +0000 (18:56 +0530)]
rbd-mirror: fix possible recursive lock of ImageReplayer::m_lock
If periodic status update (LambdaContext which is queued from
handle_update_mirror_image_replay_status()) races with shutdown and
ends up being the last in-flight operation that shutdown was pending
on, we attempt to recursively acquire m_lock in shut_down() because
m_in_flight_op_tracker.finish_op() is called with m_lock (and also
m_threads->timer_lock) held. These locks are needed only for the call
to schedule_update_mirror_image_replay_status() and should be unlocked
immediately.
Fixes: https://tracker.ceph.com/issues/69978 Co-authored-by: Ilya Dryomov <idryomov@gmail.com> Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
Adam King [Fri, 7 Feb 2025 21:08:30 +0000 (16:08 -0500)]
qa/cephadm: stop cephadm mgr module during cleanup
We recommend this generally when tearing down a cluster
and do so in cephadm-ansible. Doing this avoids any
potential timings where the cephadm mgr module attempts
to undo any daemon removals as we're cleaning up
projects / ceph.git / log