This came up during: https://tracker.ceph.com/issues/69406#note-25
Where an "assert_all" was called but didn't cause an abort.
Added "ignore_assert_all" to showcase this scenario along with any
other case where we are expected to abort.
The tests could be used to verify errorator's aborting behavior.
We should also invoke the errfunc (which aborts) when the
return type is no_touch_error_marker.
Added comments explaining:
* why it's forbidden to return void
* why std::is_same_v<return_t, no_touch_error_marker> is checked
Ville Ojamo [Tue, 22 Apr 2025 12:09:23 +0000 (19:09 +0700)]
doc/radosgw: Fix indentation in admin.rst
Indent the CLI command continuation lines correctly to start at the same
position as the other such commands, add one space on 2 lines.
Introduced in #62877.
Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
Alex Ainscow [Mon, 7 Apr 2025 08:20:44 +0000 (09:20 +0100)]
osd: Install stub extent cache in OSD.
The extent cache in new EC is a per OSD-shard cache will caches
reads used by read-modify-write to improve performance of sequential
IO. We want to provide a single PR with all of EC in it, so this
PR provides a non-functional stub to allow all the non-EC code to
be installed.
Alex Ainscow [Thu, 27 Mar 2025 15:37:57 +0000 (15:37 +0000)]
osd: New options for configuring new EC
Adding three new configuration options which will apply once new EC
is in place:
osd_pool_default_flag_ec_optimizations
This allows EC optimizations to be turned on by default.
ec_extent_cache_size
This allows the user to specify the size of the per-shard extent cache if
they feel that the default 10MiB is too large or too small.
The default value may well change following more extensive testing.
ec_pd_write_mode
This is a development flag for testing the parity delta write RMW mechanism
within the EC code. Setting to anything other than 0 will cause performance
problems. It is provided as a test mechanism for performance and
teuthology. Performance may wish too turn off all PDW writes for a particular
IO pattern. This will allow us to determine if the automatic mode should be
using conventional RMW writes. The force-on mode allows testing on more
unusual scenarios and on smaller configurations.
Finally, we tweak the way optimisations are enabled, so as to be common between
enabling and default-enabled.
Alex Ainscow [Thu, 27 Mar 2025 14:38:44 +0000 (14:38 +0000)]
crimson/osd: Add scrub stubs for crimson and classic, ready for new EC
The new optimised EC code is not backward compatible withold EC Code.
Before this commit there is some stub code which assumes that an hinfo
xattr will exist and can be used for scrub. This is no longer the case in new EC.
We plan to first make the scrubbing changes for new EC in classic and will
subsequently port to crimson. It will not look like the code here, so there is
little point in keeping it.
Additionally, add some stubs for scrub in classic optimized EC.
There will be a later PR specifically for dealing with scrubbing in
new EC which fix all the fix mes in class,
The crimson code will be fixed up at a later date and will only
support optimised EC.
Alex Ainscow [Fri, 28 Mar 2025 13:46:30 +0000 (13:46 +0000)]
common: Generalise to_interval_set to allow more interval_set implementations.
This generalises to_interval_set so that the interval set does not need to share a
common internal map structure with interval_map. The implementation is achieved
through iteration, so there is no requirement for the old restriction.
Alex Ainscow [Thu, 27 Mar 2025 11:44:29 +0000 (11:44 +0000)]
common: bitset_set
This bitset_set change relaxes policing of bitset_set, so that
out-of-range can be queried in the contains interface. This means
that callers cam simplifiy calls. For example:
if (key == invalid) || !set.contains(key)) {
do_stuff
}
Bill Scales [Wed, 26 Mar 2025 13:43:43 +0000 (13:43 +0000)]
osd: EC Optimizations: proc_master_log changes for partial logs
proc_master_log is part of the peering process that merges
the authorative log (in the case of EC pools the log of the
shard missing the most updates) into the primary log.
When there are partial writes it is likely that the
authorative log is behind because of partial writes that
did not update that shard. proc_master_log works out where
the logs diverge and then studies each additional log entry
to see if all the updates made in that log entry have been
applied. If any shard is missing an update then that log
entry (and all subsequent entries) need to be rolled back,
otherwise the entry can be rolled forward and included in
the authorative log.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Wed, 26 Mar 2025 13:25:07 +0000 (13:25 +0000)]
osd: EC Optimizations: Peer changes for partial logs
Changes to peering for replica/strays to handle partial
logs. For EC optimized pools shards may not have a complete
log if there have been partial writes that did not update
the shard. If the most recent entries in the log have all
skipped updating a shard then it will have a log that ends
earlier than other shards. During peering the primary which
has a full copy of the log works out whether other shards
have any missing objects and then communicates this to
the replica/stray shards during activation.
The primary uses the partial write last complete data in
pg_info_t to explain to other shards if they are missing
log entries and just need to update last_update and
last_complete.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Wed, 26 Mar 2025 13:15:45 +0000 (13:15 +0000)]
osd: EC Optimizations: Get missing changes for partial logs
Changes to the get missing step of peering to handle partial
writes. Having established the authorative log the primary
works out what shards are missing objects. With partial
writes this code needs to differentiate between a shard that
missed an update (and hence has a missing object) versus a
shard that was not updated by a partial write. The divergent
log entries are examined to see if the updates were partial
writes that did not involve the shard.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Wed, 26 Mar 2025 13:04:46 +0000 (13:04 +0000)]
osd: EC Optimizations: PG log changes for partial logs
Optimized EC pools will not add a log entry for shards that
are not modified by a partial write. This means the shard
will have a partial copy of the log.
There are several asserts in PGLog that assume that the log
is contiguous, these need to be relaxed when it is an optimized
EC pool (other pools retain the full strength asserts).
During peering the primary may provide a complete log to a
non-primary shard to merge into its log. This merge can skip
log entries for partial writes that do not update the shard.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Wed, 26 Mar 2025 12:40:13 +0000 (12:40 +0000)]
osd: EC optimizations: Get log twice when auth_log_shard is a non-primary
When an event such as splitting the PG occurs the new primary does
not have any log at the start of peering. Non-primary shards in an
EC optimized pool may not have a complete log of writes due to
partial writes. If the choosen authorative shard is a non-primary
shard then the new primary needs to first get a full copy of the
log (which extends past the authorative shard log) from another
shard and then repeat the get log step to get the authorative
shard's log so it can be merged rewinding divergent entries.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Thu, 6 Mar 2025 09:47:17 +0000 (09:47 +0000)]
osd: EC Optimizations: Partial write changes to add_next_event
add_next_event is used during peering to process log entries
that a shard is missing to build up a list of missing objects.
With EC optimized pools and partial writes not every update
modifies every shard. The log entry contains details of which
shards were modified and this can be used to work out whether
a missing entry needs to be created/updated.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Wed, 26 Mar 2025 10:46:07 +0000 (10:46 +0000)]
osd: EC Optimizations: Relax reset_complete_to for partial writes
EC Optimized pools can have shards missing log entries because
of partial writes. This means it is possible to have a missing
entry with a newer version than the log. Relax an assert in
reset_complete_to to avoid this.
reset_complete_to also resets last_complete to 0 when the
oldest missing object is before the first log entry. This
is to aggressive for partial writes and needs to be relaxed.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Wed, 26 Mar 2025 10:05:07 +0000 (10:05 +0000)]
osd: EC Optimizations: Add shard_id_sets for backfill_target and ...
acting_recovery_backfill
Optimized EC code uses shard_id_sets as a convinient and fast way of
representing sets of shards. Peering calculates a backfill_target set
and an active_recovery_backfill set as a map of pg_shard_ids during
peering and these are then used while processing I/O requests.
Modify peering so that it initializes a shard_id_set version of
these two sets and makes these available to ECBackend code.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Wed, 26 Mar 2025 08:30:32 +0000 (08:30 +0000)]
osd: EC Optimizations: Share pwlc between peers
Optimized EC pools add partial_writes_last_complete (pwlc) data to
pg_info_t to track shards that were not updated because of partial
writes. During peering the primary collects the info structure from
all the replica/strays and then having reconciled the log can send
the info back to peers. Different shards may have newer/older
versions of pwlc, the primary merges these together to create
the definitive copy and then redistributes this to the other shards.
The primary also adjusts the last_update and last_complete values
in the info structure received from peers using the pwlc data to
advance these where shards were not updated because of a partial
write.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Tue, 25 Mar 2025 17:41:57 +0000 (17:41 +0000)]
osd: EC Optimizations: Update pwlc in pg_info_t
Optimized EC pools add extra data to the log entry to track
which shards were updated by a partial write. When the log
entry is completed this needs to be summarized in the
partial_writes_last_complete map in pg_info_t.
Summarising this data in pg_info_t makes it easy to determine
whether the reason a shard is behind is because it is missing
update or has just not been involved in recent updates. This
also ensures that even if there is a long sequence of
updates that all skip updating a shard that a record of this
is retained in the info structure even after the log
has been trimmed.
Edited by aainscow as suggested in comment here:
https://github.com/ceph/ceph/pull/62522/files#r2050803678
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com> Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
cmake: Fix b2 build with postfixed compiler versions
Previously, the build process used `bootstrap.sh` to build the b2 tool,
which automatically selected the compiler based on the specified toolset.
This failed when the compiler executable had a version postfix (e.g.,
/usr/bin/clang++-19) without a symlink at the expected name, producing
errors like:
```
A C++11 capable compiler is required for building the B2 engine.
Toolset 'clang' does not appear to support C++11.
> clang++ -x c++ -std=c++11 -pthread check_clib.cpp check_cxx11.cpp
./tools/build/src/engine/build.sh: 120: clang++: not found
> clang++ -x c++ -std=c++11 check_clib.cpp check_cxx11.cpp
./tools/build/src/engine/build.sh: 120: clang++: not found
** Note, the C++11 capable compiler is _only_ required for building the B2
** engine. The B2 build system allows for using any C++ level and any other
** supported language and resource in your projects.
You can specify the toolset as the argument, i.e.:
./build.sh [options] gcc
```
The issue occurred because `bootstrap.sh` hardcodes the compiler name
based on the toolset (e.g., `clang++` for Clang) without supporting
postfixed versions.
This commit replaces the `bootstrap.sh` approach with an explicit build
command using Boost's `build.sh` script. We now:
1. Directly specify the full compiler path from CMake variables
2. Manually configure the build with `--cxx=...` and `--toolset=...`
3. Avoid reliance on symlinks or `bootstrap.sh`'s internal detection
This ensures the B2 engine is always built with the user-specified
compiler, even when installed with version postfixes.
Samuel Just [Tue, 15 Apr 2025 22:50:57 +0000 (15:50 -0700)]
crimson: use make_interruptible, coroutine, and RAII releaser for recover_object_with_throttle
791772f1c used with_throttle here in a way which caused
then_interruptible in PGRecovery::recover_object to be called outside of
an interruptible context.
Instead of using a wrapper taking a lambda, rephrase as an RAII releaser
suitable for use in a coroutine. This avoids needing to structure
with_throttle to deal correctly with both interruptible and
non-interruptible contexts.
Fixes: https://tracker.ceph.com/issues/70939 Signed-off-by: Samuel Just <sjust@redhat.com>
projects / ceph.git / log