Kirill Nazarov [Sun, 26 Jan 2025 19:08:24 +0000 (22:08 +0300)]
rbd: add --estimated-size option for import from stdin
One issue with importing from stdin is that it's not easy to track
progress. The only feasible option is to process messages on the highest
log level looking for lines like
but when it comes to large images it takes a lot of effort.
This commit introduces --estimated-size option, that makes it possible
to print out progress in percents via the standard mechanism. Obviously,
it requires the knowledge of the amount of provided data in advance and
in case of an error nonsensical percents might be printed, but I don't
think it's that big of a deal.
Also use `estimated size` as the base image size, making resizing not
necessary in cases where we know the exact amount of data provided from
stdin.
When the cluster needs to be read, the completion is posted to ASIO.
However, in the two special cases (cluster DNE and zero cluster), the
completion is completed inline at the moment. This violates invariants
and can eventually lead to a lockup. For example, in a scenario of
a read from a clone image whose parent is under migration:
io::ObjectReadRequest::read_parent()
io::util::read_parent()
< image_lock is taken for read >
io::ImageDispatchSpec::send()
migration::ImageDispatch::read()
migration::QCOWFormat::ReadRequest::send()
...
migration::QCOWFormat::ReadRequest::read_clusters()
< cluster DNE >
migration::QCOWFormat::ReadRequest::handle_read_clusters()
io::AioCompletion::complete()
io::ObjectReadRequest::copyup()
is_copy_on_read()
< image_lock is taken for read >
copyup() expects to be called with no locks held, but going through
QCOWFormat in the "cluster DNE" case essentially maintains image_lock
taken in read_parent() and then it's taken again by the same thread in
is_copy_on_read(). Under pthreads, it's not a problem:
A thread may hold multiple concurrent read locks on rwlock (that is,
successfully call the pthread_rwlock_rdlock() function n times). If
so, the thread must perform matching unlocks (that is, it must call
the pthread_rwlock_unlock() function n times).
But according to C++ standard it's undefined behavior:
If lock_shared is called by a thread that already owns the mutex in
any mode (exclusive or shared), the behavior is undefined.
Other, longer and more elaborate, call chains are possible too and
there it may end up being a write lock, a tripped assertion, etc. To
avoid this, make the special cases in read_clusters() behave the same
as the main path.
Zac Dover [Wed, 25 Jun 2025 09:19:49 +0000 (19:19 +1000)]
doc/radosgw: line edit bucket_logging.rst
Edit doc/radosgw/bucket_logging.rst so that it is not solecistic and so
that its punctuation is corrected and its use of articles is corrected.
This file remains in my judgment demotic and maybe demotic enough to
warrant another editorial pass in the future.
Venky Shankar [Wed, 25 Jun 2025 06:39:39 +0000 (12:09 +0530)]
Merge PR #59435 into main
* refs/pull/59435/head:
mgr/volumes: Fix json.loads for test on mon caps
mgr/volumes: Add test for mon caps if auth key has remaining mds/osd caps
mgr/volumes: Keep mon caps if auth key has remaining mds/osd caps
Add comprehensive documentation for defining configuration options in
ceph-mgr modules, including all supported properties and their usage.
Previously, the documentation did not explain how to define ceph-mgr
module configuration options, despite subtle differences from other Ceph
components. This change documents all supported Option properties, their
types, and provides clear examples to help module developers properly
configure their options.
Kefu Chai [Wed, 25 Jun 2025 03:02:46 +0000 (11:02 +0800)]
doc: do not depend on typed-ast
the typed-ast project was marked end of life since July 2023, and
not maintained anymore. since we build the document using readthedocs'
service, and in .readtherdocs.yml we use python 3.9, which comes with
ast module included by its standard library.
the typed-ast dependency was originally added in 30d41597, but now that
we are using python 3.9, there is no need to use this module anymore.
Kefu Chai [Wed, 25 Jun 2025 03:50:24 +0000 (11:50 +0800)]
doc/dev/config: Document how to use :confval: directive for config options
Add comprehensive guide for documenting configuration options using the
:confval: directive, including naming conventions and cross-referencing.
Previously, the documentation lacked guidance on using the :confval:
directive and the important distinction between regular config options
and mgr module options (which require the mgr/<module>/ namespace
prefix). This change provides detailed examples and best practices for
properly documenting and referencing both types of configuration options.
Kefu Chai [Tue, 24 Jun 2025 14:38:13 +0000 (22:38 +0800)]
rbd: fix unused function warning when WITH_KRBD is disabled
Guard print_error_description() and get_unsupported_features() with
`#ifdef WITH_KRBD` to prevent compiler warnings when KRBD support is
not enabled.
These functions are only called by do_kernel_map(), which is itself
conditionally compiled. When WITH_KRBD is not defined, the compiler
generates unused function warnings for these helper functions.
Fixes warning:
```
/home/kefu/dev/ceph/src/tools/rbd/action/Kernel.cc:305:13: warning: ‘void rbd::action::kernel::print_error_description(const char*, const char*, const char*, const char*, int)’ defined but not used [-Wunused-function]
305 | static void print_error_description(const char *poolname,
| ^~~~~~~~~~~~~~~~~~~~~~~
```
Zac Dover [Mon, 23 Jun 2025 12:50:03 +0000 (22:50 +1000)]
doc/rados: clarify "upmap_max_deviation"
Clarify the threshold set by "upmap_max_deviation" and add the
information about this configurable that is currently in
src/pybind/mgr/balancer/module.py to src/common/options/global.yaml.in,
so that it will be accessible by means of ".. confval::" declarations.
Let PGShardManager::invoke_on_each_shard_seq pass the local shard_services
instance instead of using an additional helper.
The downside of dropping the generic sharded_map_seq helper is that it is
able to support *any* (seastar::)sharded object. However, as shard_services
is the only user of it - directly using the local instance without the
helper seems easier to read.
Matan Breizman [Sun, 22 Jun 2025 10:10:10 +0000 (10:10 +0000)]
crimson/common/smp_helpers: fix reactor_map_seq
Copy f into reactor_map_seq which would be kept alive
due to this method being a coroutine. That way, we can ensure
the lambdas passed to each core that are capturing f by
reference would be safe.
Alternatively, we can also copy f by using it's copy ctor and
pass a copy to each shard:
co_await crimson::submit_to(core, F(f))
However, avoiding the copy is possible here due to the sequential
traversal. Note, seastar's invoke_on_all do copy each callback to
every shard and is running the invocation in parallel.
The above would have fixed f's captures to be invalid and result
in a segfaults on diffrent shards.
Zac Dover [Mon, 23 Jun 2025 08:18:07 +0000 (18:18 +1000)]
doc/radosgw: remove "pubsub_event_lost"
Remove "pubsub_event_lost" from the list of "Notification Performance
Statistics" in doc/radosgw/notifications.rst. "pubsub_event_lost" is now
obsolete.
Shraddha Agrawal [Wed, 28 May 2025 05:56:26 +0000 (11:26 +0530)]
mon: add command osd pool clear-availability-status
This commit adds a new command to allow users to clear the
calculated availability score for a specified pool. This can be
done by issuing the command:
ceph osd pool clear-availability-status <pool_name>
Nitzan Mordechai [Wed, 21 May 2025 11:41:01 +0000 (11:41 +0000)]
src/mon/MgrStatMonitor: fix invalid iterator increment in calc_pool_availability()
Erasing entries from `pool_availability` inside a range-for
loop invalidated the hidden iterator, triggering an
“Invalid read” under Valgrind.
- Use `std::erase_if(pool_availability, predicate)` for
atomic removal.
- Refactor the stats-update loop to use structured bindings
and a clear `++it` for readability.
John Mulligan [Fri, 20 Jun 2025 23:03:22 +0000 (19:03 -0400)]
script/build-with-container: add rocky10 to built-in distros
Add "rocky10" (also aliased to "rockylinux10") to the known distro bases
so that the team can begin to experiment with the Rocky Linux 10 distro
for containerized builds.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 20 Jun 2025 23:46:16 +0000 (19:46 -0400)]
script/build-with-container: support --build-arg arguments
Allow passing --build-arg arguments to build-with-container.py
which are passed directly to the container build command.
This allows a developer to toggle certain features of the build
container, however this should not be used in CI.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 20 Jun 2025 23:34:45 +0000 (19:34 -0400)]
Dockerfile.build: make WITH_CRIMSON a build arg
We've chosen to enable crimson by default to match the CI, but that
is not always something a developer may want, so make WITH_CRIMSON
a build argument that can be toggled off if necessary.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Kefu Chai [Fri, 20 Jun 2025 23:00:01 +0000 (07:00 +0800)]
common/static_ptr: pass an integer to alignas to fix GCC-11 build failure
GCC-11 fails to compile `alignas(std::bit_ceil(Size))` despite std::bit_ceil()
being marked constexpr in libstdc++11. The compiler doesn't recognize it as a
constant expression, while GCC-12+ and Clang-14+ handle it correctly.
Define the alignment value as a separate constexpr variable before passing it
to alignas() to ensure compatibility with GCC-11.
Fixes compilation issue introduced in commit 73399b05 when std::aligned_storage_t
was replaced with alignas.
J. Eric Ivancich [Fri, 20 Jun 2025 20:00:54 +0000 (14:00 -0600)]
Merge pull request #63271 from rafaelweingartner/parameter_to_externalize_secret_key_ttl-upstream-2
rgw: Externalize Keystone secret key cache TTL
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com> Reviewed-by: Adam C. Emerson <aemerson@redhat.com> Reviewed-by: Tobias Urdin <tobias.urdin@binero.com>
Replace contains() + at() pattern with single find() call to avoid
double lookup overhead. Previously checked map.contains(key) then
used map.at(key), which performs two hash table lookups for the
same operation.
Now uses map.find() and checks the returned iterator against end(),
accessing the value directly from the iterator when found.
Kefu Chai [Fri, 20 Jun 2025 07:56:26 +0000 (15:56 +0800)]
osd/ECExtentCache: convert LRU::Key from class to struct
Change Key from a class with public members and constructor to a
plain struct. Since Key only holds data with no special initialization
logic, a struct with aggregate initialization is more appropriate
and simpler.
This removes the user-declared constructor and relies on the compiler's
default aggregate initialization from initializer lists.
Kefu Chai [Fri, 20 Jun 2025 07:31:56 +0000 (15:31 +0800)]
osd/ECExtentCache: use default-generated comparison operators
Replace manually implemented operator== and operator!= for Key with
compiler-generated defaults. In C++20, comparison operators can be
explicitly defaulted, and the generated behavior is identical to our
hand-crafted implementation.
This simplifies the code by letting the compiler handle operator
generation automatically.
projects / ceph.git / log