Ronen Friedman [Tue, 30 Jun 2026 12:55:33 +0000 (12:55 +0000)]
crimson/os/alienstore: make CnLog::_flush non-blocking
CnLog::_flush() calls alien::submit_to(...).wait(), which blocks the
log flusher thread until the seastar reactor finishes outputting the
log entries. When the reactor writes to stderr and the pipe buffer is
full (e.g. two OSDs on the same host both dumping at debug level 20),
the reactor itself blocks on the write() syscall. Meanwhile BlueStore
threads filling the log buffer past m_max_new block in submit_entry()
waiting for the flusher - which is waiting for the reactor - which is
blocked on I/O. The kv thread cannot fire commit callbacks while it is
blocked on logging, so store transactions (including osdmap persistence)
hang indefinitely.
Fix by moving the entries into the lambda capture so _flush() can
return immediately without waiting.
This is a corrected version of the approach in commit 511af83e274
("crimson/os/alienstore/alien_log: _flush concurrently", PR #55262),
which was reverted in 4d52d1d22b7 because its lambda captured the
EntryVector by reference - the caller would clear/reuse the vector
before the lambda finished, causing a "pure virtual method called"
crash (https://tracker.ceph.com/issues/64140). The fix here avoids
that by swapping the entries out of the caller's vector and moving
them into the lambda's capture, so the lambda owns them for their
full lifetime.
Fixes: https://tracker.ceph.com/issues/77826
Fixes (failure of the 1st attempt): https://tracker.ceph.com/issues/64140 Assisted-by: Claude <claude.ai> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Karthik U S [Wed, 10 Jun 2026 05:48:25 +0000 (11:18 +0530)]
mgr/mirroring,tools/cephfs_mirror: Handle checkpoint state transition
When a new checkpoint is being added or when the daemon gets restarted,
it will check whether the newly created checkpoint or any other old
checkpoints have already been mirrored onto the remote peer. If so, it
will transition to the correct state by checking for the highest snap id
present on the remote, and setting all the checkpoints which have snap id
lesser than or equal to that of the remote to COMPLETE. This is done by
sending an acquire notification from the mirroring module to the mirror
daemon, which is handled in the add_directory path, by adding the directory
to be checked for the state transition in the tick thread.
This path gets triggered:
a) when the daemon gets restarted
b) when the peer mapping changes
c) by sending the acquire notification from checkpoint add/now CLIs.
d) when mirroring module restarts
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Karthik U S [Sat, 16 May 2026 00:50:03 +0000 (06:20 +0530)]
tools/cephfs_mirror: update checkpoint status during snapshot sync
When the sync completes or fails for a checkpointed snapshot, transition
the status of that checkpoint from CREATED/FAILED to COMPLETE/FAILED in
the do_sync_snaps() along with the timestamp of the event.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Afreen Misbah [Tue, 30 Jun 2026 14:09:13 +0000 (19:39 +0530)]
mgr/prometheus: refactor hardware metrics
- export `firmware_version` labels from `node_proxy_storage_capacity_bytes` metrics to be used in device firmware panel
- improve iterations for performance - dropping redundant node_proxy_firmware() RPC; firmware data is already
present in the fullreport response, and removing unnecessary loops.
- replace health if/elif chain with HEALTH_STATUS_MAP dict lookup
- rename metrics to ceph_hardware_*, fix component labels and add tests
- added unit test cases
- add serial name, slot info to capacity metric
- fix health metrics showing ID instead of component name
- CPU/NVMe temp: change from stat to gauge panels with colored
arc thresholds (green/yellow/red) and proper °C units
- Health panels: wrap queries in max() aggregation to show
single worst-case value instead of overlapping series
- Pie charts: switch to donut style with visible legends
- Fan RPM: use locale unit for comma formatting instead of
short which auto-scaled to "K"
- Fix temperature panels units
- Add Device List and Platform Firmware table panels
Afreen Misbah [Thu, 25 Jun 2026 08:18:19 +0000 (13:48 +0530)]
monitoring: fix hardware Grafana dashboard and health metrics
- Fix fan repeating panels: set multi=true on fan_speeds template
variable so Grafana generates one panel per fan instead of one
- Remove TACH-only regex filter on fan_speeds template and AVG
Cooling query so all system fans are visible regardless of naming
- Replace duplicate Power Control panel with Network health panel
- Fix NVMe drive count to use storage_capacity_bytes{protocol=NVMe}
instead of counting temperature sensors (inaccurate proxy)
- Normalize all hostname filters to regex match (=~) for consistency
- Register hardware.libsonnet in dashboards.libsonnet so the
dashboard JSON is generated during ceph-mixin builds
- Add temperatures category to prometheus health metrics loop
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
Afreen Misbah [Thu, 25 Jun 2026 08:13:38 +0000 (13:43 +0530)]
mgr/prometheus: use orchestrator API for node-proxy hardware metrics
The hardware metrics exporter was reading cephadm's private KV store
directly via get_store_prefix('mgr/cephadm/node_proxy/data'). This
had two problems:
1. get_store_prefix() is module-scoped, so from the prometheus module
it searched under prometheus's own namespace instead of cephadm's,
resulting in zero metrics being exported despite metric definitions
appearing at /metrics.
2. The firmware key was accessed as 'firmwares' (plural) but the
stored field is 'firmware' (singular), causing all firmware version
metrics to be silently empty.
Use node_proxy_fullreport() and node_proxy_firmware() via the
OrchestratorClientMixin instead. This routes through cephadm's
NodeProxyCache which handles KV access and firmware key compat
correctly. Follows the same pattern as set_cephadm_daemon_status_metrics()
and get_smb_metadata().
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
mgr/dashboard: fix stale nvmeof spec tests from a semantic merge conflict
mgr-dashboard-frontend-unittests has been failing on main since 2026-07-01,
unrelated to whatever PR triggers it. Bisected to cfe12305fb1, the merge
commit for PR #68180.
6182e67ea4 ("mgr/dashboard: NVMe Onboarding Setup Cards") moved gateway
group selection out of NvmeofSubsystemsComponent and
NvmeofNamespacesListComponent into a new NvmeofGatewayGroupFilterComponent,
dropping gwGroups, updateGroupSelectionState, handleGatewayGroupsError,
onGroupClear, and getSubsystems from both. Clean on its own.
PR #68180 forked from main in April, before that refactor existed, and
added three new tests against the old API. Also clean on its own, and never
rebased. Both PRs only touch lines the other doesn't, so when #68180 merged
34 minutes after 6182e67ea4, git's 3-way merge combined them without a
conflict. The merge is the only place the bug exists: new tests calling
methods the sibling parent had already deleted. TypeError at runtime,
TS2339/TS2551 from tsc, every run since. CI never caught it: GitHub was
showing #68180's make check as green from a run a full day older than 6182e67ea4.
nvmeof-namespaces-list.component.spec.ts: the dedup test is real, current
logic, it just called the old listNamespaces() instead of fetchData().
Fixed the call. The other two duplicate NvmeofGatewayGroupFilterComponent's
own spec; removed.
nvmeof-subsystems.component.spec.ts: same story, all three duplicate the
filter component's spec; removed.
Both removed error tests also checked preventDefault() on the error object,
which had no coverage elsewhere. Added that to
NvmeofGatewayGroupFilterComponent's spec, and hit a second bug while writing
it: the file's throwError(() => err) is the RxJS 7 factory overload, this
project is on RxJS 6.6.3, so it was emitting the arrow function itself as
the error. The existing error test never noticed since it only checks side
effects, not the error's identity. Used throwError(err) instead.
Verified: jest passes on all three spec files (was 6 failing), tsc -p
tsconfig.spec.json --noEmit is clean (was 17 errors), full block/nvmeof
sweep is 29/29 suites, 216/216 tests.
Ilya Dryomov [Mon, 29 Jun 2026 11:39:50 +0000 (13:39 +0200)]
qa/suites/rbd: use client.0 entity in migration-external tests
Currently client.admin is passed for client_name and that doesn't
exercise client_name handling much as client.admin is the default.
When deploying multiple clusters the ceph task distributes keyrings
with only the key for the initial monitor and client.admin key. Other
keys (e.g. client.0) are present only on their respective clusters, so
client.0's key for cluster2 needs to be obtained on cluster1 explicitly
with "ceph auth get".
Leonid Chernin [Mon, 2 Feb 2026 06:01:14 +0000 (08:01 +0200)]
librbd/migration/NativeFormat: support specifying mon_host and key via spec
migration:
-take secret_key from spec
-take mon_host from the spec
-ignore source keyring, source ceph.conf - bypass them
-added validations of the new way schema
-get key from the KV DB if key in spec is actually key-ref
Fixes: https://tracker.ceph.com/issues/68177 Signed-off-by: Leonid Chernin <leonidc@il.ibm.com> Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
* refs/pull/66558/head:
qa/cephfs: minor fix in comment
qa/cephfs: give more time to tests in test_clone_stats.py
qa/cephfs: increase number of files to cloned in test_clone_stats.py
volumes/stats_util: improve log messages
Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Jos Collin <jcollin@redhat.com>
Igor Golikov [Wed, 27 May 2026 13:19:25 +0000 (16:19 +0300)]
client: fix Dentry UAF by holding DentryRef across unlink
Hold a function-scoped DentryRef in Client::unlink() to keep the dentry
alive through the entire teardown sequence.
Without the O_DIRECTORY dentry pin (PR #60909, merged after v19.2.1),
opening a directory does not call _ll_get(), so the dentry stays at
ref=1 while inode->dir is still active. When trim_cache() evicts this
dentry, Dentry::unlink() calls put() for the dir pin, ref drops to 0,
and the dentry is freed while Client::unlink() still needs it for
detach/lru_remove.
A function-scoped DentryRef guarantees the dentry survives until after
it has been properly removed from the dir and LRU, regardless of pin
state.
Signed-off-by: Igor Golikov <igolikov@ibm.com> Fixes: https://tracker.ceph.com/issues/74625
Igor Golikov [Wed, 27 May 2026 13:18:44 +0000 (16:18 +0300)]
client: add assert to catch dentry ref drop during unlink
Add ceph_assert(dn->ref > 0) after Dentry::unlink() in Client::unlink()
to catch the case where internal put() calls drop the dentry reference
to zero before we proceed to detach/lru_remove.
This turns a silent use-after-free into a clean assertion failure with
a full stack trace, making it possible to catch the exact conditions
in a test environment.
Signed-off-by: Igor Golikov <igolikov@ibm.com> Fixes: https://tracker.ceph.com/issues/74625
Anthony M [Tue, 30 Jun 2026 22:11:39 +0000 (17:11 -0500)]
doc/foundation: Removed members section
These updates will allow us to update a single page rather than two. The detailed member lists and categories have been removed. I've added a redirect for visitors to this page to https://ceph.io/en/foundation/members/
Signed-off-by: Anthony M <amiddleton@linuxfoundation.org>
Sun Yuechi [Tue, 30 Jun 2026 11:50:09 +0000 (19:50 +0800)]
librbd: fix use-after-free in trash purge on image open error
When ImageState::open() fails with an error other than -ENOENT, it
asynchronously deletes the ImageCtx before returning. The trash purge
loop only skipped the -ENOENT case and fell through on every other
error, dereferencing the already-freed ictx in diff_iterate() and
state->close(). Add the missing continue so a failed open is skipped.
The fall-through dates back to 504f4e78, which split `if (r < 0)
continue` to log non-ENOENT errors but dropped the continue.
Fixes: https://tracker.ceph.com/issues/77836 Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Mohit Agrawal [Fri, 12 Jun 2026 01:59:18 +0000 (07:29 +0530)]
core: Handle OP_TOUCH_TEMP in BlueStore without any guard(WITH_CRIMSON)
During backfill recovery on Crimson OSD replica, Bluestore
aborts with _txc_add_transaction error ENOENT on operation
due to OP_TOUCH_TEMP. OP_TOUCH_TEMP is guarded by #ifdef WITH_CRIMSON
but WITH_CRIMSON is not defined when BlueStore is compiled.
OP_TOUCH_TEMP is a crimson specific op to manage temporary
objects and the op is not used by classic osd. It is a plain
integer constant defined in the shared transaction layer so
adding it to the common code path in Transaction.h and BlueStore.cc
requires no additional includes or compile time guards.Handling
it unconditonally is therefore safe for all Objectstore backends.
Sun Yuechi [Tue, 30 Jun 2026 03:50:20 +0000 (20:50 -0700)]
blk/aio: fix mismatched parenthesis in POSIX AIO submit path
The condition `if ((cur->n_aiocb == 1) {` has an unbalanced
parenthesis and fails to compile. This code is only built on the
HAVE_POSIXAIO (BSD) path, so the breakage is normally hidden.
Xuehan Xu [Fri, 5 Jun 2026 09:19:13 +0000 (17:19 +0800)]
crimson/os/seastore/journal/circular_bounded_journal: find the real
journal tail on startup
Recover the correct journal tail during replay by scanning
JOURNAL_TAIL records instead of relying solely on the on-disk
header, which may be stale if a crash occurs between record
write and header update.
The journal header is stored *separately* from the (append-only)
journal and is not updated atomically with journal writes,
so it may lag behind after a crash.
Treat the journal as the source of truth and rebuild the tail from persisted records.
Replay is reorganized into three passes:
1) discover latest journal tail
2) rebuild allocation map
3) replay deltas
Kefu Chai [Mon, 29 Jun 2026 01:19:02 +0000 (09:19 +0800)]
crimson/test: stop the messenger-thrash config on every exit path
test_messenger_thrash put sharded_conf().stop() in the success
continuation, so an exception from the tests skipped it and left the
static sharded<ConfigProxy> holding live instances. its destructor then
asserts. a tester hit this when the run went OOM at --memory 256M:
ERROR [shard 0:main] test - Test failed: got exception
ceph::buffer::v15_2_0::bad_alloc (Bad allocation [buffer:1])
/ceph/src/seastar/include/seastar/core/sharded.hh:573:
seastar::sharded<Service>::~sharded()
[with Service = crimson::common::ConfigProxy]:
Assertion `_instances.empty()' failed.
terminate called without an active exception
run the body in seastar::async and stop the config through
seastar::deferred_stop, as crimson/osd/main.cc already does. the guard
is constructed only after start() succeeds, and its destructor stops the
config on any exit path, so a failing test exits cleanly instead of
aborting.
Ronen Friedman [Sun, 28 Jun 2026 14:42:23 +0000 (14:42 +0000)]
cmake: add WITH_MOLD option for Mold linker support
Add a WITH_MOLD cmake option that auto-configures the build for
the Mold linker. When enabled, cmake finds mold, sets CMAKE_LINKER,
injects -fuse-ld=mold into all linker flag variables, and sets
USING_MOLD_LINKER which gates the Mold-specific workarounds added
in the previous commit.
Usage: cmake -DWITH_MOLD=ON ...
Assisted-by: Claude <claude.ai> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Ronen Friedman [Sun, 28 Jun 2026 14:42:07 +0000 (14:42 +0000)]
build: add Mold linker compatibility
Mold handles several linker features differently from ld.bfd:
- --exclude-libs,ALL hides .symver-aliased symbols (e.g. rados_*)
even when the version script lists them as global. Detect Mold
(via USING_MOLD_LINKER) and disable --exclude-libs; version
scripts already control symbol visibility.
- Duplicate symbols across static archives are treated as errors.
Merge crimson-alien-common into crimson-alienstore as a single
archive. For crimson-osd, link crimson-alienstore before
crimson-common with --allow-multiple-definition so the alien
thread's non-crimson definitions win (Mold picks first definition).
- Python/Cython extension builds (via Distutils.cmake) invoked the
system default linker. Pass -fuse-ld= via MOLD_FUSE_LD_FLAG so
extensions link with the configured linker.
- Add rados_* and _rados_* to librados.map global section for Mold
compatibility with .symver aliases.
Co-authored-by: Mark Kogan <mkogan@redhat.com> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Yael Azulay [Thu, 21 May 2026 11:38:28 +0000 (14:38 +0300)]
cephadm: keep mgr ports in sync when modules are enabled/disabled
ceph orch ps - was showing stale or incorrect port information for `mgr` daemons after enabling or disabling mgr modules (e.g., dashboard, prometheus).
This fix ensures the displayed ports always reflect the actual active module endpoints.
Code Fixes:
-----------
Changes in src/pybind/mgr/cephadm/services/cephadmservice.py:
- Extract and refactor _get_mgr_service_ports:
The logic for reading active mgr module ports was inline inside prepare_create, using check_mon_command({'prefix': 'mgr services'}). It was extracted into a static helper method and refactored to read directly from mgr.get('mgr_map')['services'] instead -- which is faster and consistent with how get_dependencies reads the same data. The method is now shared by both prepare_create (existing use) and get_dependencies (new use).
- Extract `_get_mgr_service_ports`:
The logic for reading active mgr module ports from `mgr_map['services']` was inline inside `prepare_create`. It was extracted into a static helper method so it can be reused by both `prepare_create` (existing use) and `get_dependencies` (new use). Returns the port numbers currently registered by active mgr modules (e.g., `[8443, 9283]`).
- Override `get_dependencies` in `MgrService`:
The base class returns `[]` for all Ceph daemons. The serve loop compares `get_dependencies()` output against `last_deps` (saved after the previous reconfig). If they differ, a reconfig is triggered. By overriding this method in `MgrService`, we make the serve loop aware of mgr port changes. Returns a sorted list such as `['port:8443', 'port:9283', 'sd_port:8765']`.
- Override `generate_config` in `MgrService`:
After a reconfig completes, the result of `generate_config` is saved as `last_deps`. The inherited implementation returns `[]` as deps. If `[]` is saved but `get_dependencies` returns `['port:8443', 'sd_port:8765']`, they would always differ, causing an infinite reconfig loop. This override ensures that what is saved as `last_deps` matches what `get_dependencies` will return in the next iteration.
Changes in `src/pybind/mgr/cephadm/serve.py`:
- Update mgr cache on reconfig:
The existing logic skipped the cache update for all Ceph daemon reconfigs. A new `elif` branch for mgr reconfigs fetches the existing cached `DaemonDescription`, updates only its `ports` field, and saves it back. This makes `ceph orch ps` reflect the correct ports immediately after a reconfig, without showing a misleading "starting" status.
Changes in `src/cephadm/cephadm.py`:
- Update `unit.meta` on reconfig:
`unit.meta` is a per-daemon file on the host that stores metadata including ports. It is read periodically by `cephadm ls` and fed back into the orchestrator cache. Previously it was only written on initial deployment, so every periodic cache refresh would overwrite the corrected ports with stale data from `unit.meta`. This change updates the `ports` field in `unit.meta` during reconfig, ensuring the cache refresh confirms the correct state rather than reverting it.
Changes to src/pybind/mgr/cephadm/tests/services/test_mgr.py:
- Updated _prepare helper:
The original mocked check_mon_command to feed services data to prepare_create. Since our _get_mgr_service_ports now reads from mgr.get('mgr_map') instead of check_mon_command, the mock was replaced with mock_store_set('_ceph_get', 'mgr_map', ...) which sets the data in the right place.
- Added test_get_dependencies_changes_when_module_enabled:
A new test that verifies get_dependencies returns different dep strings before and after a module is enabled. This directly tests the core sync mechanism -- the serve loop compares get_dependencies output against last_deps to detect port changes and trigger a reconfig.
to run new tests:
cd src/pybind/mgr
tox -e py3 -- cephadm/tests/services/test_mgr.py -v
projects / ceph.git / log