Casey Bodley [Thu, 6 Mar 2025 18:13:36 +0000 (13:13 -0500)]
qa/vault: create_secrets() cleanup checks for orphaned keys
the vault task creates some keys on startup for use in s3-tests. on
cleanup, check that rgw has removed any temporary bucket keys that
were created during testing. fail with an assertion if the listed keys
differ
* refs/pull/62108/head:
cmake/cephfs: fix options to enable client and dependencies
Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net> Reviewed-by: Adam C. Emerson <aemerson@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z
For some newer versions of cython, it appears it requires explicitly specifying
noexcept but old versions of Cython 0.29.Z do not understand that attribute.
pybind/cephfs: increment ref before calling out to c++
At the time this construction seemed safe since the caller should have a
reference but it could conveivably be the only ref. We don't want the ref count
to reach 0.
Additionally, catch errors so this callback is genuinely noexcept.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Matan Breizman [Sun, 2 Mar 2025 14:33:07 +0000 (14:33 +0000)]
crimson/mgr/client:Introduce Client::send()
Client::reconnect nullifies the connection used by the mgr client
before setting a new one.
In this time, we might re-use the nullptr connection due to tasks
that are being run in the background (See: dispatch_in_background).
To avoid this, we had multiple `if (!conn)` checks, some methods
even checked this condition twice to reduce the possibilty of using
undefined the connection.
Instead of introducing an additional check in Client::_send_report,
Introduce Client::send which would be responsible for:
a) Veryfing the connection is set
b) Trying to get a shared access to conn_lock
Client::reconnect will lock conn_lock exclusivly until the
connection is set. If we send is called while reconnecting,
sending will be dropped - same as before.
New sequence - write object initially with gaps and fill in the holes
until the object is fully populated. Permutate the step (gap between
writes), start at randomized offsets to produce gaps at the start of
the object as well as between writes
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Fri, 29 Nov 2024 11:12:40 +0000 (11:12 +0000)]
test: ceph_test_rados_io_sequence - add append with gaps sequence
New I/O sequence - appends to objects by writing past the old
object length so there is a gap before the new written data.
Tests permutations of write length and gap length.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Soumya Koduri [Wed, 29 Jan 2025 16:17:45 +0000 (21:47 +0530)]
rgw/cloudtier: Correct option ordering in RGWZoneGroupPlacementTier
Two tier-config options (related to `cloud-restore`) were incorrectly added
in the middle of the encoding and decoding methods of RGWZoneGroupPlacementTier.
This modification can cause compatibility issues with older decoders when
attempting to read v2-encoded REST objects.
The fix is to correct the option order and update the decode() function to
properly interpret the structure based on the encoded version.
Ilya Dryomov [Mon, 3 Mar 2025 16:59:35 +0000 (17:59 +0100)]
test/pybind/rbd: fix read offset in write zeroes tests
Random data is written and write zeroes is invoked on 0~256, but the
read is done on 256~256. This means that if write zeroes malfunctions
the test wouldn't catch it (especially in the thick provision case).
VinayBhaskar-V [Tue, 26 Nov 2024 11:18:51 +0000 (16:48 +0530)]
librbd: add rbd_diff_iterate3() API to take source snapshot by ID
Allow a diff to start from a non-user snapshot. This would be used by
"rbd du" command to account for non-user snapshots which are currently
just skipped potentially resulting in underreported space usage and in
other places.
* refs/pull/61321/head:
qa: update require-osd-release to tentacle
tools/monmaptool: bump new cluster version to X
doc/dev/release-checklists: remove ceph-container task
script/ceph-release-notes: add squid/tentacle
doc/dev/release-checklists:: mark task complete
doc/dev/release-checklist: add nightlies task
doc/dev/release-checklists: update ceph-build for tentacle
doc/dev/release-checklists: note redmine is done
qa: update to tentacle
doc/dev/release-checklist: question telemetry tentacle test
osd/OSDMap: update to tentacle
qa/workunits/cephtool/test: update to tentacle
mon/OSDMonitor: update to tentacle
common/options/global.yaml.in: update for tentacle
mon/MgrMonitor: update for tentacle
qa/standalone/mon/misc: update for tentacle
doc: update compatset for tentacle
doc: no deprecated features
include/ceph_features: add SERVER_TENTACLE feature bit
cephadm,ceph-volume: update to tentacle
doc/dev/release-checklist: add backport-create-issue
script: update backport-resolve-issue to tentacle
*: add constants and release names
ceph_release: update to tentacle
librbd: bump version
CMakeLists.txt: update VERSION
doc: remove obsolete checklist item
doc: reset for tentacle
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com> Reviewed-by: Bill Scales <bill_scales@uk.ibm.com> Reviewed-by: Adam King <adking@redhat.com> Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
* refs/pull/60746/head:
client: skip unexpected command replies
mgr: indicate map message is acked instead of unhandled
osdc/Objecter: convert to ms_dispatch2 for ack
client: indicate maps are acked not processed
msg: add alternate statuses for ms_dispatch2 handling
tools/cephfs_mirror: do not process maps with fast dispatch
doc: add docs for volumes interface for charmap
qa: add tests for subvolume charmap settings
pybind/mgr/volumes: wire up charmap for subvol/subvolgroup
pybind/mgr: send MDS commands through cephfs client
pybind/cephfs: wire up mds_command2
mgr: add module method to send notifications
libcephfs: add mds_command2 for asynchronous commands
mgr: excise CephFS client from mgr C++ base
mgr: use std namespace
doc: add docs for CephFS charmap config
qa: add charmap tests
qa: add helpful exceptions for attr changes
qa: ignore libicu leaks
client: add wrappings for charmap manipuluation of dentry names
client: add dir_result_t::dentry::print
win32: add libicu Windows build
CMakeLists: add boost::locale dependency for client
install-deps: unconditionally install boost libraries
test/libcephfs: update root operation return values
client: refactor all path traversals through path_walk
test/libcephfs: test parallel creates
test/libcephfs: add test for lookup failure after readdir
client: init dentry shared_gen with invalid value
client: add _lookup debugging
client: remove redundant check
client: dump InodeStat from mds
mds: encode optmetadata in InodeStat sent to clients
mds: check client features for charmap
mds: add client feature bit for charmap
mds: wire up vxattr for changing charmap
mds: inherit charmap on mkdir
mds,include: add charmap optmetadata
mds,include: add inode_t optional metadata
client: hide alternate_name from API
client: move alternate_name once
client: optimize alternate_name passing to helper
client: relocate definition
client: print dentry with alternate_name on dump
client: move inode dump to print method
mds: add debugging for encoding lease stat
mds: make encode_lease a proper method
mds: add fscrypt metadata for inode stat size
client: use DentryRef for ref counting in MetaRequest
client: add DentryRef
client: add helper for determining if a perm check is necessary
client: cache client_permissions config
client: add debugging for conf changes
client: sort configs
client/UserPerm: add print method
client: note mount parameters in debug log
client: print stat mode in octal
common: add missing op string
include/filepath: add empty path check
ceph-volume: Refactor is_ceph_device to simplify error handling
Replace the try-except block with a direct get() call on lv.tags to
check for ceph.osd_id. This avoids catching AttributeError unnecessarily
and makes the logic more concise. Additionally, the warning message is
now logged when osd_id is None or 'null', ensuring consistency in
error handling.
Also, this fixes the unit test `api/test_lvm.py::TestVolume::test_is_not_ceph_device()`
as `api.lvm.is_ceph_device()` expects a `Volume` object.
ceph-volume: Introduce new `Lvm` base class to unify LVM object handling
This commit introduces a new `Lvm` base class to streamline LVM related objects
(`PVolume`, `VolumeGroup`, and `Volume`) by consolidating shared logic.
Key changes:
- `Lvm` centralizes common attributes like `name`, `tags`, `path`, and `binary_change`.
- `clear_tags`, `clear_tag`, `set_tag`, and `set_tags` are now defined in `Lvm`,
reducing code duplication.
- `PVolume`, `VolumeGroup`, and `Volume` inherit from `Lvm`,
simplifying their constructors.
- The redundant `_format_tag_args` and tag manipulation methods in
child classes are removed.
This refactor improves maintainability by reducing code duplication
while preserving the existing API behavior.
This replaces the previous approach using `list(sump(zip(repeat(op), tag_args), ()))` with a
more explicit loop-based implementation.
This makes the logic clearer while maintaining the same functionality.
Matan Breizman [Sun, 2 Mar 2025 08:42:45 +0000 (08:42 +0000)]
cmake/modules/BuildISAL.cmake: set no-integrated-as on clang only
this option is only relevant to clang, gcc will fail with:
```
CMake Error at ceph/build/src/erasure-code/isa/isal_ext-prefix/src/isal_ext-stamp/isal_ext-configure-Debug-impl.cmake:19 (message):
Command failed (77):
Ilya Dryomov [Sun, 2 Mar 2025 08:24:52 +0000 (09:24 +0100)]
librbd: fix a deadlock on image_lock caused by Mirror::image_disable()
With Mirror::image_disable() taking image_lock for write and calling
list_children() under it, the following deadlock is possible:
1. Mirror::image_disable() takes image_lock for write and calls
list_children()
2. AbstractWriteLog::periodic_stats() timer fires (it runs every
5 seconds) and ImageCacheState::write_image_cache_state() is called
under a global timer_lock
3. ImageCacheState::write_image_cache_state() successfully takes
owner_lock and blocks attempting to take image_lock for read because
it's already held for write by Mirror::image_disable()
4. list_children() blocks inside of a call to ImageState::close() on
a descendant image
5. The descendant image close can't proceed because TokenBucketThrottle
requires a global timer_lock to complete QosImageDispatch shutdown
6. safe_timer thread which is holding timer_lock can't proceed because
ImageCacheState::write_image_cache_state() is effectively blocked on
the descendant image close through Mirror::image_disable()
Until commit 281a64acf920 ("librbd: remove snapshot mirror image-meta
when disabling"), Mirror::image_disable() was taking image_lock only for
read meaning that this deadlock wasn't possible. The only other change
that commit 281a64acf920 made to the code block protected by image_lock
was using child_mirror_image_internal for cls_client::mirror_image_get()
call on descendant images instead of mirror_image_internal to preserve
the value of mirror_image_internal for later. Both are local variables
that have nothing to do with image_lock, so I'm going back and making
Mirror::image_disable() take image_lock only for read again.
J. Eric Ivancich [Fri, 28 Feb 2025 19:22:53 +0000 (14:22 -0500)]
doc/rgw: update dynamic resharding docs to reflect recent changes
The documentation on dynamic resharding is updated to include a) a
description of reducing the number of shards, b) related configuration
options, and c) the radosgw-admin sub-command to set a minimum number
of shards for a specific bucket.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
John Mulligan [Fri, 14 Feb 2025 19:51:03 +0000 (14:51 -0500)]
doc: document the new container build tool and link to it in README
Add a new markdown file in the root of the tree, ContainerBuild.md, that
can serve as a basic introduction to the new container build tools
recently merged to ceph.
Add a small 'breadcrumb' section to the project README.md to help find
this new document.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 20 Feb 2025 00:17:30 +0000 (19:17 -0500)]
script/build-with-container: add support for overlay dir
The source dir (aka homedir, default /ceph) is mounted in the container
read-write. This is needed as the various ceph build scripts expect to
write things into the tree - often this is in the build directory - but
not always. This can lead to small messes and/or situations that are
confusing to debug, especially if one is jumping between distros often.
Add an option to use an overlay volume for the homedir - by default we
enable a persistent overlay with a supplied "upper dir" where files that
were written will appear. One can also enable a temporary overlay that
forgets the writes when the container exits - maybe useful when doing
experiments in 'interactive' mode.
To use this option run the command with the `--overlay=<dir>` option.
For example: `./src/script/build-with-container.py -b build.inner
--overlay-dir build.ovr`. This will create a directory
`build.ovr/content` automatically and all new files will appear there.
For example the build directory will appear at
`build.ovr/content/build.inner`.
To use the temporary overlay use a `-` as the directory name. For
example: `./src/script/build-with-container.py -b build.inner
--overlay-dir -`
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 20 Feb 2025 14:50:49 +0000 (09:50 -0500)]
script/build-with-container: skip dnf cache dir volume mounts on docker
When using docker the --volume option is not available during build
(docker [buildx] build), unlike podman. Since passing these volumes must
be conditional on them being set up I see no way to handle this short of
just disabling the option on docker. Log the fact that it's being
skipped - the only other issue is that we pointlessly set up some dirs
and the build may be a bit slower.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
projects / ceph.git / log