Zack Cerza [Fri, 7 Mar 2025 20:53:23 +0000 (13:53 -0700)]
make-debs.sh: Optionally take debian version
Our existing CI builds have names like:
ceph-base_20.0.0-194-g6efaea33-1jammy_amd64.deb
Before this change, they are like:
ceph-base_20.0.0-158-gb0de3a42-1_amd64.deb
This way we can pass e.g. "jammy" to end up with names compatible with our CI
builds.
New sequence - write object initially with gaps and fill in the holes
until the object is fully populated. Permutate the step (gap between
writes), start at randomized offsets to produce gaps at the start of
the object as well as between writes
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Fri, 29 Nov 2024 11:12:40 +0000 (11:12 +0000)]
test: ceph_test_rados_io_sequence - add append with gaps sequence
New I/O sequence - appends to objects by writing past the old
object length so there is a gap before the new written data.
Tests permutations of write length and gap length.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Ilya Dryomov [Mon, 3 Mar 2025 16:59:35 +0000 (17:59 +0100)]
test/pybind/rbd: fix read offset in write zeroes tests
Random data is written and write zeroes is invoked on 0~256, but the
read is done on 256~256. This means that if write zeroes malfunctions
the test wouldn't catch it (especially in the thick provision case).
VinayBhaskar-V [Tue, 26 Nov 2024 11:18:51 +0000 (16:48 +0530)]
librbd: add rbd_diff_iterate3() API to take source snapshot by ID
Allow a diff to start from a non-user snapshot. This would be used by
"rbd du" command to account for non-user snapshots which are currently
just skipped potentially resulting in underreported space usage and in
other places.
* refs/pull/61321/head:
qa: update require-osd-release to tentacle
tools/monmaptool: bump new cluster version to X
doc/dev/release-checklists: remove ceph-container task
script/ceph-release-notes: add squid/tentacle
doc/dev/release-checklists:: mark task complete
doc/dev/release-checklist: add nightlies task
doc/dev/release-checklists: update ceph-build for tentacle
doc/dev/release-checklists: note redmine is done
qa: update to tentacle
doc/dev/release-checklist: question telemetry tentacle test
osd/OSDMap: update to tentacle
qa/workunits/cephtool/test: update to tentacle
mon/OSDMonitor: update to tentacle
common/options/global.yaml.in: update for tentacle
mon/MgrMonitor: update for tentacle
qa/standalone/mon/misc: update for tentacle
doc: update compatset for tentacle
doc: no deprecated features
include/ceph_features: add SERVER_TENTACLE feature bit
cephadm,ceph-volume: update to tentacle
doc/dev/release-checklist: add backport-create-issue
script: update backport-resolve-issue to tentacle
*: add constants and release names
ceph_release: update to tentacle
librbd: bump version
CMakeLists.txt: update VERSION
doc: remove obsolete checklist item
doc: reset for tentacle
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com> Reviewed-by: Bill Scales <bill_scales@uk.ibm.com> Reviewed-by: Adam King <adking@redhat.com> Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
* refs/pull/60746/head:
client: skip unexpected command replies
mgr: indicate map message is acked instead of unhandled
osdc/Objecter: convert to ms_dispatch2 for ack
client: indicate maps are acked not processed
msg: add alternate statuses for ms_dispatch2 handling
tools/cephfs_mirror: do not process maps with fast dispatch
doc: add docs for volumes interface for charmap
qa: add tests for subvolume charmap settings
pybind/mgr/volumes: wire up charmap for subvol/subvolgroup
pybind/mgr: send MDS commands through cephfs client
pybind/cephfs: wire up mds_command2
mgr: add module method to send notifications
libcephfs: add mds_command2 for asynchronous commands
mgr: excise CephFS client from mgr C++ base
mgr: use std namespace
doc: add docs for CephFS charmap config
qa: add charmap tests
qa: add helpful exceptions for attr changes
qa: ignore libicu leaks
client: add wrappings for charmap manipuluation of dentry names
client: add dir_result_t::dentry::print
win32: add libicu Windows build
CMakeLists: add boost::locale dependency for client
install-deps: unconditionally install boost libraries
test/libcephfs: update root operation return values
client: refactor all path traversals through path_walk
test/libcephfs: test parallel creates
test/libcephfs: add test for lookup failure after readdir
client: init dentry shared_gen with invalid value
client: add _lookup debugging
client: remove redundant check
client: dump InodeStat from mds
mds: encode optmetadata in InodeStat sent to clients
mds: check client features for charmap
mds: add client feature bit for charmap
mds: wire up vxattr for changing charmap
mds: inherit charmap on mkdir
mds,include: add charmap optmetadata
mds,include: add inode_t optional metadata
client: hide alternate_name from API
client: move alternate_name once
client: optimize alternate_name passing to helper
client: relocate definition
client: print dentry with alternate_name on dump
client: move inode dump to print method
mds: add debugging for encoding lease stat
mds: make encode_lease a proper method
mds: add fscrypt metadata for inode stat size
client: use DentryRef for ref counting in MetaRequest
client: add DentryRef
client: add helper for determining if a perm check is necessary
client: cache client_permissions config
client: add debugging for conf changes
client: sort configs
client/UserPerm: add print method
client: note mount parameters in debug log
client: print stat mode in octal
common: add missing op string
include/filepath: add empty path check
Matan Breizman [Sun, 2 Mar 2025 08:42:45 +0000 (08:42 +0000)]
cmake/modules/BuildISAL.cmake: set no-integrated-as on clang only
this option is only relevant to clang, gcc will fail with:
```
CMake Error at ceph/build/src/erasure-code/isa/isal_ext-prefix/src/isal_ext-stamp/isal_ext-configure-Debug-impl.cmake:19 (message):
Command failed (77):
Ilya Dryomov [Sun, 2 Mar 2025 08:24:52 +0000 (09:24 +0100)]
librbd: fix a deadlock on image_lock caused by Mirror::image_disable()
With Mirror::image_disable() taking image_lock for write and calling
list_children() under it, the following deadlock is possible:
1. Mirror::image_disable() takes image_lock for write and calls
list_children()
2. AbstractWriteLog::periodic_stats() timer fires (it runs every
5 seconds) and ImageCacheState::write_image_cache_state() is called
under a global timer_lock
3. ImageCacheState::write_image_cache_state() successfully takes
owner_lock and blocks attempting to take image_lock for read because
it's already held for write by Mirror::image_disable()
4. list_children() blocks inside of a call to ImageState::close() on
a descendant image
5. The descendant image close can't proceed because TokenBucketThrottle
requires a global timer_lock to complete QosImageDispatch shutdown
6. safe_timer thread which is holding timer_lock can't proceed because
ImageCacheState::write_image_cache_state() is effectively blocked on
the descendant image close through Mirror::image_disable()
Until commit 281a64acf920 ("librbd: remove snapshot mirror image-meta
when disabling"), Mirror::image_disable() was taking image_lock only for
read meaning that this deadlock wasn't possible. The only other change
that commit 281a64acf920 made to the code block protected by image_lock
was using child_mirror_image_internal for cls_client::mirror_image_get()
call on descendant images instead of mirror_image_internal to preserve
the value of mirror_image_internal for later. Both are local variables
that have nothing to do with image_lock, so I'm going back and making
Mirror::image_disable() take image_lock only for read again.
J. Eric Ivancich [Fri, 28 Feb 2025 19:22:53 +0000 (14:22 -0500)]
doc/rgw: update dynamic resharding docs to reflect recent changes
The documentation on dynamic resharding is updated to include a) a
description of reducing the number of shards, b) related configuration
options, and c) the radosgw-admin sub-command to set a minimum number
of shards for a specific bucket.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
John Mulligan [Fri, 14 Feb 2025 19:51:03 +0000 (14:51 -0500)]
doc: document the new container build tool and link to it in README
Add a new markdown file in the root of the tree, ContainerBuild.md, that
can serve as a basic introduction to the new container build tools
recently merged to ceph.
Add a small 'breadcrumb' section to the project README.md to help find
this new document.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 20 Feb 2025 00:17:30 +0000 (19:17 -0500)]
script/build-with-container: add support for overlay dir
The source dir (aka homedir, default /ceph) is mounted in the container
read-write. This is needed as the various ceph build scripts expect to
write things into the tree - often this is in the build directory - but
not always. This can lead to small messes and/or situations that are
confusing to debug, especially if one is jumping between distros often.
Add an option to use an overlay volume for the homedir - by default we
enable a persistent overlay with a supplied "upper dir" where files that
were written will appear. One can also enable a temporary overlay that
forgets the writes when the container exits - maybe useful when doing
experiments in 'interactive' mode.
To use this option run the command with the `--overlay=<dir>` option.
For example: `./src/script/build-with-container.py -b build.inner
--overlay-dir build.ovr`. This will create a directory
`build.ovr/content` automatically and all new files will appear there.
For example the build directory will appear at
`build.ovr/content/build.inner`.
To use the temporary overlay use a `-` as the directory name. For
example: `./src/script/build-with-container.py -b build.inner
--overlay-dir -`
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 20 Feb 2025 14:50:49 +0000 (09:50 -0500)]
script/build-with-container: skip dnf cache dir volume mounts on docker
When using docker the --volume option is not available during build
(docker [buildx] build), unlike podman. Since passing these volumes must
be conditional on them being set up I see no way to handle this short of
just disabling the option on docker. Log the fact that it's being
skipped - the only other issue is that we pointlessly set up some dirs
and the build may be a bit slower.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 19 Feb 2025 18:20:36 +0000 (13:20 -0500)]
script/build-with-container: remove default --volume arg from ctr build
On the original github pr #59841 user fayak kindly informed us that the
--volume option was not supported by docker build. Since this section
was a leftover from a previous way of constructing the builder image and
was no longer needed we simply removed it.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 19 Feb 2025 18:20:01 +0000 (13:20 -0500)]
script/build-with-container.py: build builder image with --pull=always
Construct the builder image using the --pull=always flag to initiate a
pull of the base image (centos, ubuntu, etc) in order to avoid using a
stale base image. Since the script automatically (by default) avoids
building if a matching tag is in local container storage it is handy to
use a fresh base when it *is* time to build something. Otherwise, you
end up in a situation like I sometimes do - using a months old base
unintentionally.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 14 Feb 2025 19:50:42 +0000 (14:50 -0500)]
script/build-with-container: add a common packages target
Add a `packages` target to build-with-container.py that requests a build
of packages, whatever package type is native to the distro selected.
For example `./src/script/build-with-container.py -d ubuntu22.04 -e
packages` will automatically select a deb packages build where
`./src/script/build-with-container.py -d centos9 -e packages` will
trigger rpm packages to be built. The underlying package-type specific
targets remain unchanged.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 14 Feb 2025 16:44:35 +0000 (11:44 -0500)]
script/build-with-container: support custom tag suffixes
Previously, one could use the `--tag` option to completely override the
container tag generated by the script. However, there are cases where
one may want to add information to the tag rather than override it.
Allow the tag value to start with a plus (+) character that indicates
that the remainder of the string is to be suffixed to the generated tag.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add a command line option --base-branch that allows the user to supply a
custom base branch name. git doesn't make determining this easy so we
always assume a base branch of 'main' by default - but this option lets
one change that.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 14 Feb 2025 16:24:29 +0000 (11:24 -0500)]
src/script: rename CEPH_BRANCH to CEPH_BASE_BRANCH for build container
Previously, we were passing build argument of CEPH_BRANCH, but that was
a bit misleading as we expect the current branch to vary a bit (as users
will be using branches to develop and test the code). What we actually
care about is the base branch ('main', 'squid', etc) as that is fed into
our bootstrap script and we want the option to simple variations based
on the name of said base branch.
Rename CEPH_BRANCH to CEPH_BASE_BRANCH for clarity.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add a new --current-branch argument that lets the user supply a name for
the current branch. This allows the automatic tag generation to avoid
calling git - something useful if the tree is not using a git checkout
(like a tarball). It also allows you to pull a temporary branch in git
but ignore it and act like the temporary branch is the base branch.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Tue, 11 Feb 2025 23:36:13 +0000 (18:36 -0500)]
script/build-with-container: add more distro aliases
Add a system to define distro name aliases and use that to define some
additional aliases, primarily to match ubuntu codenames rather than
version numbers. Requested by Zack.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
rgw: add radosgw-admin sub-command to set-min-shards for a bucket
There is now a mechansim to set the minimum number of shards when a
bucket is created, and dynamic resharding adheres to that
setting. This adds the ability to modify that minimum shard count that
exists within the bucket layout of the bucket instance
object. Example:
J. Eric Ivancich [Wed, 15 Jan 2025 16:26:59 +0000 (11:26 -0500)]
rgw: allow per-bucket minimum number of shards
Dynamic resharding can now reduce the number of shards. The code
currently has a hard-coded value of 11 as the minimum number of shards
dynamic resharding can reshard to. There may be cases where the user
wants to set an alternate minimum, such as when they have a sense of
how many objects the bucket will eventually hold.
This PR builds off of https://github.com/ceph/ceph/pull/61269 .
That PR allows the user to specify an initial number of shards during
bucket creation. This PR then takes that number to be the minimum and
saves it in the layout field of the bucket instance object
(RGWBucketInfo).
When dynamic resharding is triggered, it will use that stored value as
a minimum number of shards for resharing.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
projects / ceph.git / log