Adam King [Thu, 11 Dec 2025 04:18:17 +0000 (23:18 -0500)]
Merge branch 'execute_command_retry_logic_if_command_fails_with_connection_error' of https://github.com/ShwetaBhosale1/ceph into wip-adk-testing-2025-12-10-2316
Adam King [Thu, 11 Dec 2025 04:16:42 +0000 (23:16 -0500)]
Merge branch 'fix_issue_73851_cephadm_crashes_when_ganesha-rados-grace_fails' of https://github.com/ShwetaBhosale1/ceph into wip-adk-testing-2025-12-10-2316
See comment:
```
//TODO: should be changed to return future<> once all calls
// to refresh are through co_await. We return LBAMapping
// for now to avoid mandating the callers to make sure
// the life of the lba mapping survives the refresh.
```
For now introduce co_refresh and mark the existing refresh as
deprecated. Following work will audit all the existing users of
refresh and move them to the new method. This change is not trivial
so I prefer to follow up on this as a separate PR.
This should help avoiding UAR in suspension points:
```
==103588==ERROR: AddressSanitizer: stack-use-after-return on address 0xffff80197e90 at pc 0xaaaacb941b24 bp 0xffff7e48dd80 sp 0xffff7e48dd78
READ of size 8 at 0xffff80197e90 thread T1
#0 0xaaaacb941b20 in boost::intrusive_ptr<crimson::os::seastore::LBACursor>::swap(boost::intrusive_ptr<crimson::os::seastore::LBACursor>&) /opt/ceph/include/boost/smart_ptr/intrusive_ptr.hpp:172:18
#1 0xaaaacb941998 in boost::intrusive_ptr<crimson::os::seastore::LBACursor>::operator=(boost::intrusive_ptr<crimson::os::seastore::LBACursor>&&) /opt/ceph/include/boost/smart_ptr/intrusive_ptr.hpp:93:61
#2 0xaaaacb933758 in crimson::os::seastore::LBAMapping::operator=(crimson::os::seastore::LBAMapping&&) /ceph/src/crimson/os/seastore/lba_mapping.h:46:48
#3 0xaaaacde2fa54 in ... crimson::os::seastore::LBAMapping&&, std::array<crimson::os::seastore::LBAManager::remap_entry_t, 1ul>) (.resume) /ceph/src/crimson/os/seastore/transaction_manager.h:1282:11
```
Deprecate is commented out since otherwise make check would fail.
Matan Breizman [Sun, 7 Dec 2025 10:36:28 +0000 (10:36 +0000)]
crimson/os/seastore/cache: Verify crc prior to complete_io
Calling check_full_extent_integrity in pin_to_extent is wrong.
By the time we partially read the extent, another transaction
might fully load the entire extent. Either by a full_extent read
or by filling the missing unloaded extent gap.
This would result in veryfing the extent crc since it's fully
loaded. However, since the extent was only partially read our
data (and crc) might be outdated.
Instead move the crc checks to read_extent prior to complete_io.
That way we would only check the crc when the extent was intended to
be fully loaded. Other users of read_extent which do not use pin_crc
(CRC_NULL), would skip this check.
Matan Breizman [Sun, 7 Dec 2025 10:31:41 +0000 (10:31 +0000)]
crimson/os/seastore/cache: get_absent_extent to accept pin_crc
Allow pin_to_extent_by_type and pin_to_extent to pass the pin CRC.
As get_absent_extent and read_extent has other users which are not
pin_to_extent - use CRC_NULL as the default value.
This change would allow us to move the integrity checks to _read_extent.
See next commit.
Kefu Chai [Mon, 8 Dec 2025 08:29:00 +0000 (16:29 +0800)]
include/cpp-btree: fix array bounds warning in child() accessors
Replace array indexing with pointer arithmetic in child() and
mutable_child() methods to avoid compiler warning when accessing
child nodes beyond the static array bounds.
The original code was functionally correct but triggered
-Warray-bounds when accessing mutable_child(32) during btree
operations. Using pointer arithmetic achieves the same result
without the bounds check warning.
This is a follow-up to commit 8458a19ab which fixed similar
warnings in other btree_node methods.
check_full_extent_integrity would allow for CRC_NULL (0) checksums
when `full_extent_integrity_check` was false. Instead, move this
check as an assertiion into TransactionManager::read_pin.
This would allow us to reuse CRC_NULL concept for more purposes e.g
skipping integrity checks when no CRC is passed (next commits).
Note: With this change check_full_extent_integrity could be called
only on non CRC_NULL, the check would be moved in next commits.
Kefu Chai [Tue, 25 Nov 2025 11:02:32 +0000 (19:02 +0800)]
script: sanitize git branch names for OCI tag compliance
Git branch names commonly use forward slashes for hierarchy
(e.g., feature/my-branch), but OCI container image tags cannot
contain slashes. This causes build-with-container.py to fail when
building images from branches with slashes in their names.
Add _sanitize_for_oci_tag() function to convert branch names into
OCI-compliant tags by:
- Replacing '/' with '-'
- Replacing other invalid characters with '_'
- Stripping leading invalid characters
- Truncating to 128 characters (OCI tag max length)
Apply sanitization consistently to both auto-detected branches
and manually specified --current-branch arguments.
Fixes branch name handling for pull request branches and other
hierarchical naming schemes.
References:
- OCI Image Spec: tags must match [a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}
https://github.com/opencontainers/image-spec/blob/main/descriptor.md
- Git allows '/' in branch names for hierarchical organization
https://git-scm.com/docs/git-check-ref-format
https://docs.github.com/en/get-started/using-git/dealing-with-special-characters-in-branch-and-tag-names
Ville Ojamo [Wed, 22 Oct 2025 07:19:31 +0000 (14:19 +0700)]
doc: Use validated links with ref instead of external links
Use :ref: for intra-docs links that are validated, instead of external
links.
Only use already existing labels.
Fixes a few anchors that pointed to now-renamed section titles.
Use automatically generated link text where appropriate.
Delete unused link definitions.
Mostly in doc/rados/ but also a few in doc/rbd/.
Try to fix all links in each of the changed documents.
Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
cmake: convert erasure_code and json_spirit to OBJECT libraries
This resolves a circular dependency issue where ceph-common was linked
against erasure_code and json_spirit static libraries, while these
libraries themselves referenced symbols from ceph-common, creating an
unresolvable circular dependency. The static libraries were incorrectly
marked PUBLIC, causing executables linking against ceph-common to also
link against them directly.
The circular dependency manifested as linker errors in tests like
ceph_test_ino_release_cb, where libjson_spirit.a and liberasure_code.a
contained undefined references to ceph-common symbols (e.g.,
ceph::__ceph_assert_fail and get_str_list) that couldn't be resolved
due to the linking order.
For instance, ceph_test_ino_release_cb failed to link:
```
/usr/bin/ld: ../../../lib/libcephfs.so.2.0.0: undefined reference to symbol '_ZN4ceph18__ceph_assert_failERKNS_11assert_dataE'
/usr/bin/ld: ../../../lib/libceph-common.so.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
```
Changes:
- Convert erasure_code and json_spirit from STATIC to OBJECT libraries
- Embed their object files directly into ceph-common during linking,
breaking the circular dependency and preventing public propagation
of these dependencies to downstream targets
- Remove direct linkage from targets that already get these symbols
through ceph-common dependencies:
* Erasure code unit tests (unittest_erasure_code_isa,
unittest_erasure_code_plugin_isa, unittest_erasure_code_example)
* Test libraries (radostest)
* Plugins (denc-mod-osd, cls_refcount, cls_rgw, cls_lua, ec_lrc)
This also prevents ODR violations that would occur if targets linked
against these libraries directly while also getting them from
ceph-common. Such violations cause undefined behavior including
segmentation faults, as static variables and vtables would exist in
duplicate, leading to crashes during destruction or when accessing
shared state.
For example, before removing direct linkage from plugins, ceph-dencoder
would segfault on certain object types:
/ceph/src/test/encoding/readable.sh: line 111: Segmentation fault
$CEPH_DENCODER type ScrubMap import ... decode encode decode dump_json
Redouane Kachach [Fri, 28 Nov 2025 08:38:45 +0000 (09:38 +0100)]
mgr/cephadm: Fix mgmt-gateway default port in get_port_start()
The mgmt-gateway port was already defaulted to 443 in most places, but
get_port_start() did not apply this default. Since the output of
get_port_start() is used both to configure the daemon ports which are
later used to to open them in firewalld, this inconsistency meant the
HTTPS port was not opened when firewalld service was active.
This change makes get_port_start() also default to port 443, ensuring
the daemon is configured correctly and the corresponding firewalld port
is opened as expected.
crimson/mclock_scheduler: Support mclock for crimson
The patch is trying to sync mclock source of crimson similar to
classic osds. Currently the feature is use by crimson only for
background recovery operations but later we will use it for other
OSD operations also.To use the same user need to configure
crimson_osd_scheduler_concurrency parameter for osd.
Replace item_t with WorkItem variant to maintain similarity
with classic OSD.
crimson/background_recovery: switch to unified SchedulerClass and introduce get_average_object_size for pg
1) Replace usage of crimson::osd::scheduler::scheduler_class_t
with unified SchedulerClass
2) Add priority to scheduler params structure
3) Introduce get_average_object_size for pg
crimson/osd,osd_operation: initialize mClock scheduler, detect rotational devices, and run OperationThrottler background task
Initialize the mClock scheduler on all shards when the device class
is non-rotational. If the device is rotational throw an exception
to prevent unsupported configurations.
In addition, introduce a background task in OperationThrottler that
continuously dequeues and schedules client requests from the mClock
scheduler based on available credits and throttling limits.
mgr/cephadm: fix nvmeof TLS handling and add coverage for ssl/mTLS
This PR fixes the value of `ssl` field on `NvmeofServiceSpec` (was
always set to enable_auth) and add some UT to make sure both specs
with ssl only and with mTLS enabled (enable_auth) generate the
expected daemon configuration.
Casey Bodley [Mon, 1 Dec 2025 15:25:16 +0000 (10:25 -0500)]
cmake: fix for -DWITH_BREAKPAD=OFF
in 1ba55a20be1023c585ba96617dc6a9d2aa79a51b, i tried to avoid the NOT
condition by swapping the option's defaults. but when the condition is
false, the option is forced to ON even if the user manually set it OFF
fix this by inverting the condition and swapping the default values
Reported-by: Joseph Mundackal <joseph.j.mundackal@gmail.com> Signed-off-by: Casey Bodley <cbodley@redhat.com>
Imran Imtiaz [Mon, 1 Dec 2025 14:25:07 +0000 (14:25 +0000)]
mgr/dashboard: add API endpoint to delete images from consistency groups
Signed-off-by: Imran Imtiaz <imran.imtiaz@uk.ibm.com> Fixes: https://tracker.ceph.com/issues/74033
Create a consistency group dashboard API endpoint that enables removal
of RBD images from the group.
Kefu Chai [Thu, 27 Nov 2025 12:28:31 +0000 (20:28 +0800)]
osd: fix ODR violation in max_prio_map
The static std::map max_prio_map was defined in the osd_types.h header
file, causing every translation unit that included this header to get
its own copy of the variable. This led to One Definition Rule (ODR)
violations where multiple instances of the same variable existed at
runtime.
During program cleanup, destructors for these multiple instances would
attempt to free the same memory regions, resulting in segmentation
faults in tcmalloc/memory allocator as seen with ceph-dencoder.
This issue surfaced after a yet-merged-change which converts erasure_code
and json_spirit to OBJECT libraries. Before that change, these were
STATIC libraries that were linked via target_link_libraries. The
incorrect linkage meant their object files (and thus their copies of
max_prio_map) were kept separate and didn't conflict at runtime.
After converting to OBJECT libraries and properly incorporating them
into libceph-common.so (commit 8b0e3fb2c23), the multiple copies of
max_prio_map from different translation units all ended up in the same
shared library, exposing the ODR violation. During program exit, the
dynamic linker attempted to run destructors for all instances, leading
to double-free crashes.
Fix by moving the map into a static helper function in PeeringState.cc
(the only file that uses it). The map is now a function-local static
const variable, ensuring a single instance that is properly initialized
and destructed.
Backtrace before fix:
```
#0 0x00007ffff7dbb1a0 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int) () from /lib/x86_64-linux-gnu/libtcmalloc.so.4
#1 0x00007ffff7dbb57f in tcmalloc::ThreadCache::Scavenge() () from /lib/x86_64-linux-gnu/libtcmalloc.so.4
#2 0x00007ffff6bc8aa2 in std::__new_allocator<std::_Rb_tree_node<std::pair<int const, int> > >::deallocate (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890, __n=1)
#3 0x00007ffff6bc89f9 in std::allocator<std::_Rb_tree_node<std::pair<int const, int> > >::deallocate (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890, __n=1)
#4 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<int const, int> > > >::deallocate (__a=..., __p=0x555555f43890, __n=1)
#5 std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_put_node (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890)
#6 0x00007ffff6bc892e in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_drop_node (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890)
#7 0x00007ffff6bc886e in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase (this=0x7ffff7d48f78 <max_prio_map>, __x=0x555555f43890)
#8 0x00007ffff6bc8854 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase (this=0x7ffff7d48f78 <max_prio_map>, __x=0x555555f43cb0)
#9 0x00007ffff6bc8854 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase (this=0x7ffff7d48f78 <max_prio_map>, __x=0x555555f43ad0)
#10 0x00007ffff6bc8805 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::~_Rb_tree (this=0x7ffff7d48f78 <max_prio_map>)
#11 0x00007ffff6bc7345 in std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > >::~map (this=0x7ffff7d48f78 <max_prio_map>)
#12 0x00007ffff484bd51 in __cxa_finalize (d=0x7ffff7d3f440) at ./stdlib/cxa_finalize.c:97
#13 0x00007ffff6af9487 in __do_global_dtors_aux () from /home/kefu/dev/ceph/build/lib/libceph-common.so.2
#14 0x00007ffff7fbfd20 in ?? ()
#15 0x00007ffff7fc8fc2 in _dl_call_fini (closure_map=0x7fffffffd0f0, closure_map@entry=0x7ffff7fbfd20) at ./elf/dl-call_fini.c:43
#16 0x00007ffff7fcbe72 in _dl_fini () at ./elf/dl-fini.c:120
#17 0x00007ffff484c291 in __run_exit_handlers (status=0, listp=0x7ffff49f1680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:118
#18 0x00007ffff484c35a in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:148
#19 0x00007ffff4833caf in __libc_start_call_main (main=main@entry=0x55555556cd90 <main(int, char const**)>, argc=argc@entry=2, argv=argv@entry=0x7fffffffd488) at ../sysdeps/nptl/libc_start_call_main.h:74
#20 0x00007ffff4833d65 in __libc_start_main_impl (main=0x55555556cd90 <main(int, char const**)>, argc=2, argv=0x7fffffffd488, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd478) at ../csu/libc-start.c:360
#21 0x00005555555695e1 in _start ()
```
Nizamudeen A [Tue, 25 Nov 2025 11:31:31 +0000 (17:01 +0530)]
mgr/dashboard: support custom prop for table item redirection
use an extra customTemplateConfig called `customRowProperty` where
you can provide the key of the property you wish to route, instead of
relying on the cell's prop itself
Fixes: https://tracker.ceph.com/issues/73989 Signed-off-by: Nizamudeen A <nia@redhat.com>
Alex Ainscow [Fri, 28 Nov 2025 14:33:13 +0000 (14:33 +0000)]
osd: Perform shard look up correctly in partial EC writes
Plugins are permitted to provide a mapping to change the order in which OSDs
are used. In practice only LRC does this and it is not currently enabled
with optimisations, so this is a theoretical bug.
The bug here was that the "first" shard was assumed to be shard_id_t(0). However,
this is not true for LRC.
Fixes: https://tracker.ceph.com/issues/74016 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Issue: Route was being force-reloaded using a two-step navigation hack causing unnecessary redirects and side effects.
Fix: Replaced the hack with Angular’s native same-URL reload using onSameUrlNavigation: 'reload' for a clean, stable route refresh.
Bill Scales [Fri, 21 Nov 2025 10:06:22 +0000 (10:06 +0000)]
qa: Reduce number of osd threads when using compression
Smithi nodes used by teuthology tests have 8 CPU cores and typically run
4 OSD processes. When bluestore software compression is enabled the size
of the OSD thread pool needs to be reduced to 2 threads per OSD because
these threads can easily use 100% of a core. This avoids excessive
amounts of context switches, which leads to OSD threads timing out,
which causes the OSD to drop heartbeat pings and for the monitor to
temporarily mark it down. In extreme cases this can lead to PGs getting
stuck in repeated loops of peering until the teuthology test times out.
Context switches happen oppurtunistically at the end of system calls
so functions with lots of logging are some of the worst affected.
Fixes: https://tracker.ceph.com/issues/72879 Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Fri, 21 Nov 2025 10:38:44 +0000 (10:38 +0000)]
osd: Restrict logging in MissingLoc::add_source_info
add_source_info can generate an excessive amount of logging
if a PG has thousands of missing objects. When a system is
under load and threads are repeatedly context switching this
can lead to timeouts (tests showed this function taking up
to 10 seconds to execute with 99% of that time being in
logging calls where the thread was being pre-empted).
Stopping logging after the function has been running for
more than 0.5 seconds strikes a balance between providing
sufficient informtion to debug problems while providing
more stability when a system is heavily loaded.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Fri, 21 Nov 2025 10:25:48 +0000 (10:25 +0000)]
osd: Increase log level for listing missing list
Logging the entire contents of a missing list can generate a
1M character log line when there are 8000 missing objects in a
PG. Other places in the code logging the missing list use debug
level 25 which is not enabled by default in teuthology tests.
Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
Bill Scales [Mon, 24 Nov 2025 09:18:21 +0000 (09:18 +0000)]
osd: reset_tp_timeout should reset timeout for all shards
ShardedThreadPools are only used by the classic OSD process
which can have more than one thread for the same shard. Each
thread has a heartbeat timeout used to detect stalled threads.
Some code that is known to take a long time makes calls to
reset_tp_timeout to reset this timeout. However for sharded
pools this can be ineffective because it is common for threads
for the same shard to use the same locks (e.g. PG Lock) and
therefore if thread A is taking a long time and resetting
its timeout while holding a lock, thread B for the same shard
is liable to be waiting for the same lock, will not be
resetting its timeout and can be timed out.
Debug for issue 72879 showed heartbeat timeouts occurring at
the same time for both shards, an attempt to fix the problem
by calling reset_tp_timeout for the slow thread still showed
the other threads for the shard timing out waiting for the PG
lock that was held bythe slow thread. Looking at the OSD code
most places where reset_tp_timeout is called the thread is
holding the PG lock.
This commit moves the concept of shard_index from OSD into
ShardedThreadPool and modifies reset_tp_timeout so that it resets
the timeout for all threads for the same shard.
Some code calls reset_tp_timeout from inside loops that can take
a long time without consideration for how long the thread has
actually been running for. There is a risk that this type of
call could repeatedly reset the timeout for another shard which
is genuinely stuck and hence defeat the heartbeat checks. To
prevent this reset_tp_timeout is modified to be a NOP unless
the thread has been processing the current workitem for more
than 0.5 seconds. Therefore threads have to be slow but making
forward progress to be abe to reset the timeout.
Fixes: https://tracker.ceph.com/issues/72879 Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
projects / ceph-ci.git / log