Nizamudeen A [Wed, 8 Jul 2026 05:21:50 +0000 (10:51 +0530)]
mgr/dashboard: teardown http requests for feature-toggle
angular 19.2.latest is aggressively tearing down the test env which
forces the active rxjs timer to emit one execution as it collapses which
produces a ghost http in the case of request wrapped in the timer
service. so for now flushing all the requests manually to prevent it
Fixes: https://tracker.ceph.com/issues/77729 Signed-off-by: Nizamudeen A <nia@redhat.com>
qa/suites/rbd/valgrind: pin to centos_9.stream instead of rpm_latest
This used to be the case before commit d4b977afdc59 ("qa/distros:
rename centos_latest.yaml to rpm_latest.yaml") and subsequent changes
to enable Rocky 10. When paired with valgrind, python_api_tests* jobs
fail persistently on Ubuntu and Rocky, see [1]. Switch back until that
is resolved (or for as long as centos_9.stream remains in the mix).
Sun Yuechi [Mon, 6 Jul 2026 08:06:09 +0000 (01:06 -0700)]
test/subprocess: add missing space in SubshellKilled subcommand
9abf2388f24 rewrote "sh -c cat" as SHELL "-c cat", which concatenates
to "/bin/sh-c cat", so the subshell tries to execute a non-existent
"/bin/sh-c". The test still passed because the SIGTERM sent by kill()
usually wins the race against the shell startup; under CI load the
shell occasionally prints
/bin/sh: line 1: /bin/sh-c: No such file or directory
into the stderr pipe first, ASSERT_TRUE(buf.empty()) fails, and the
skipped join() turns this into a SIGABRT in ~SubProcess().
monitoring/ceph-mixin: scope test queries per dashboard
get_dashboards_data() keeps every query in a single map keyed by
"<panel title>-<legendFormat>", but that id is not unique: several
dashboards have a panel titled "IOPS", "OSDs" or "Throughput". The
dashboard read last overwrites the earlier entry, so one query shadows
another and never gets tested. glob() walks the files in filesystem
order, so which query survives, and whether the test passes, depends on
readdir.
run-tox-promql-query-test hit this in "Test IOPS Read"
(ceph-cluster.feature): the scenario feeds ceph_osd_op_r but the id
resolved to a CephFS pool panel.
pool_id is "UNSET VARIABLE" because the scenario does not set $mdatapool,
and no ceph_pool_rd series were fed, so the query returns nil. It only
fails when readdir returns cephfsdashboard after ceph-cluster-advanced,
so the earlier change that walked nested rows and made the winner
deterministic did not fix it; it fixed which query wins, not that the
wrong one can win.
Key the queries per dashboard (data['queries'][<dashboard>][<id>]) and
have each .feature name its dashboard in a Background step. The lookup is
confined to that dashboard, so a title/legend used elsewhere no longer
shadows it. A duplicate within one dashboard is still an error, unless it
is in a collapsed row.
ceph-cluster.feature covers ceph-cluster-advanced; the rest map to one
dashboard each.
Ronen Friedman [Tue, 30 Jun 2026 12:55:33 +0000 (12:55 +0000)]
crimson/os/alienstore: make CnLog::_flush non-blocking
CnLog::_flush() calls alien::submit_to(...).wait(), which blocks the
log flusher thread until the seastar reactor finishes outputting the
log entries. When the reactor writes to stderr and the pipe buffer is
full (e.g. two OSDs on the same host both dumping at debug level 20),
the reactor itself blocks on the write() syscall. Meanwhile BlueStore
threads filling the log buffer past m_max_new block in submit_entry()
waiting for the flusher - which is waiting for the reactor - which is
blocked on I/O. The kv thread cannot fire commit callbacks while it is
blocked on logging, so store transactions (including osdmap persistence)
hang indefinitely.
Fix by moving the entries into the lambda capture so _flush() can
return immediately without waiting.
This is a corrected version of the approach in commit 511af83e274
("crimson/os/alienstore/alien_log: _flush concurrently", PR #55262),
which was reverted in 4d52d1d22b7 because its lambda captured the
EntryVector by reference - the caller would clear/reuse the vector
before the lambda finished, causing a "pure virtual method called"
crash (https://tracker.ceph.com/issues/64140). The fix here avoids
that by swapping the entries out of the caller's vector and moving
them into the lambda's capture, so the lambda owns them for their
full lifetime.
Fixes: https://tracker.ceph.com/issues/77826
Fixes (failure of the 1st attempt): https://tracker.ceph.com/issues/64140 Assisted-by: Claude <claude.ai> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
mgr/cephadm: Add documentation for RGW keyring capability dependencies
This serves as a reminder for maintainers to verify whether capability
changes in the RGW service should also be applied to SMB and NFS
services to maintain consistency across services that use librgw.
mgr/dashboard: support clone_by_snap_id flag in clone API for group snapshots
The clone API endpoint POST /api/block/image/{image_spec}/snap/{snapshot_name}/clone
only accepted a snapshot name, which uses rbd_clone3 (clone by name). This only works
for snapshots in the user namespace.
Group snapshots (with .group prefix, in RBD_SNAP_NAMESPACE_TYPE_GROUP namespace)
require rbd_clone4 (clone by snap ID). The rbd Python binding already supports this:
passing an int for p_snapshot triggers rbd_clone4.
Add an optional clone_by_snap_id boolean flag to the clone endpoint. When set, the
path parameter is treated as a numeric snap ID and passed directly as an int to
rbd.RBD().clone(). Clone format 2 is enforced because this flag is intended for
non-user namespace snapshots which cannot be protected (a prerequisite for clone
format 1).
Usage:
POST /api/block/image/{image_spec}/snap/{snap_id}/clone
{"clone_by_snap_id": true, ...}
Fixes: https://tracker.ceph.com/issues/77875 Signed-off-by: Imran Imtiaz <imran.imtiaz@uk.ibm.com> Assisted-by: IBM Bob
mgr/smb: Allow pool creation for empty RGW buckets
When an RGW bucket is empty, SMB clients cannot create directories,
files, or upload data because the default.rgw.buckets.data pool does
not yet exist. The previous CephX capabilities ('mon allow r') did not
permit pool creation during the first write operation.
The fix changes the mon capability from 'allow r' to 'allow *' to
enable pool creation on the first write to an empty RGW bucket. Once
the pool exists, subsequent operations succeed normally.
Sun Yuechi [Thu, 2 Jul 2026 09:26:59 +0000 (02:26 -0700)]
common/crc32c: stop using gp/tp as scratch in RISC-V Zbc CRC32C
crc32c_zbc's fold-by-four loop used gp (x3) and tp (x4) as scratch for
two folding temporaries. Save/restore around the loop is not enough:
they are ABI-reserved, so a signal delivered mid-loop runs its handler
with a corrupted tp, and the first TLS access there faults.
On riscv64 this reliably crashes the crimson/seastore unittests
unittest-transaction-manager and unittest-omap-manager, where seastar's
stall-detector timer fires often: the process dies with SIGSEGV in the
linker's TLS path with garbage in gp/tp.
Fixes: https://tracker.ceph.com/issues/77904 Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
vstart.sh: only skip fs volume create when there's no mgr to run it
The mgr volumes module handles fs volume ls/create, not the monitor, so
without a mgr (CEPH_NUM_MGR=0) the "wait for volume module to load" loop
spun forever. Open-coding fs new unconditionally, my first attempt,
changed pool creation for every vstart user and dropped the wait, so
cephfs tests calling `fs volume ...` right after vstart.sh returns could
race a module still mid-load.
create_fs_volume() branches on CEPH_NUM_MGR instead. With a mgr it still
calls fs volume ls/create. Without one it runs the plain mon commands fs
volume create wraps: osd pool create cephfs.<name>.{meta,data}, then fs
new <name> <meta> <data>. The wait stays at its one call site above the
per-filesystem loop, so it runs once regardless of CEPH_NUM_FS, not once
per filesystem.
Verified against a local vstart: mgr present, fs volume ls polls once
then creates CEPH_NUM_FS=2 filesystems with no repeated poll;
CEPH_NUM_MGR=0, no poll, fs new creates 'a' directly, HEALTH_OK.
crimson/osd: skip store flush for empty peering transactions
dispatch_context_transaction submits a real empty transaction to
AlienStore even when the peering context has no store ops. This
goes through the AlienCollection lock and thread pool slot machinery.
Under load from broadcast_map_to_pgs (many PGAdvanceMap operations
running concurrently with recovery), some of these empty transactions
get stuck inside AlienStore's pipeline and never reach BlueStore,
causing PGAdvanceMap to hang at complete_rctx indefinitely.
Fix by skipping the store round-trip for empty transactions. The
on_commit callbacks are still collected and fired, preserving the
fix from b0f5e1086a6 ("crimson/osd/pg: also trigger callbacks for
empty peering transactions"). The store flush that commit added was
incidental to the callback fix and is unnecessary here: the peering
pipeline's exclusive process stage already guarantees that prior
peering transactions on the same PG have completed.
Assisted-by: Claude <claude.ai> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Karthik U S [Wed, 10 Jun 2026 05:48:25 +0000 (11:18 +0530)]
mgr/mirroring,tools/cephfs_mirror: Handle checkpoint state transition
When a new checkpoint is being added or when the daemon gets restarted,
it will check whether the newly created checkpoint or any other old
checkpoints have already been mirrored onto the remote peer. If so, it
will transition to the correct state by checking for the highest snap id
present on the remote, and setting all the checkpoints which have snap id
lesser than or equal to that of the remote to COMPLETE. This is done by
sending an acquire notification from the mirroring module to the mirror
daemon, which is handled in the add_directory path, by adding the directory
to be checked for the state transition in the tick thread.
This path gets triggered:
a) when the daemon gets restarted
b) when the peer mapping changes
c) by sending the acquire notification from checkpoint add/now CLIs.
d) when mirroring module restarts
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Karthik U S [Sat, 16 May 2026 00:50:03 +0000 (06:20 +0530)]
tools/cephfs_mirror: update checkpoint status during snapshot sync
When the sync completes or fails for a checkpointed snapshot, transition
the status of that checkpoint from CREATED/FAILED to COMPLETE/FAILED in
the do_sync_snaps() along with the timestamp of the event.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Afreen Misbah [Tue, 30 Jun 2026 14:09:13 +0000 (19:39 +0530)]
mgr/prometheus: refactor hardware metrics
- export `firmware_version` labels from `node_proxy_storage_capacity_bytes` metrics to be used in device firmware panel
- improve iterations for performance - dropping redundant node_proxy_firmware() RPC; firmware data is already
present in the fullreport response, and removing unnecessary loops.
- replace health if/elif chain with HEALTH_STATUS_MAP dict lookup
- rename metrics to ceph_hardware_*, fix component labels and add tests
- added unit test cases
- add serial name, slot info to capacity metric
- fix health metrics showing ID instead of component name
- CPU/NVMe temp: change from stat to gauge panels with colored
arc thresholds (green/yellow/red) and proper °C units
- Health panels: wrap queries in max() aggregation to show
single worst-case value instead of overlapping series
- Pie charts: switch to donut style with visible legends
- Fan RPM: use locale unit for comma formatting instead of
short which auto-scaled to "K"
- Fix temperature panels units
- Add Device List and Platform Firmware table panels
Afreen Misbah [Thu, 25 Jun 2026 08:18:19 +0000 (13:48 +0530)]
monitoring: fix hardware Grafana dashboard and health metrics
- Fix fan repeating panels: set multi=true on fan_speeds template
variable so Grafana generates one panel per fan instead of one
- Remove TACH-only regex filter on fan_speeds template and AVG
Cooling query so all system fans are visible regardless of naming
- Replace duplicate Power Control panel with Network health panel
- Fix NVMe drive count to use storage_capacity_bytes{protocol=NVMe}
instead of counting temperature sensors (inaccurate proxy)
- Normalize all hostname filters to regex match (=~) for consistency
- Register hardware.libsonnet in dashboards.libsonnet so the
dashboard JSON is generated during ceph-mixin builds
- Add temperatures category to prometheus health metrics loop
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
Afreen Misbah [Thu, 25 Jun 2026 08:13:38 +0000 (13:43 +0530)]
mgr/prometheus: use orchestrator API for node-proxy hardware metrics
The hardware metrics exporter was reading cephadm's private KV store
directly via get_store_prefix('mgr/cephadm/node_proxy/data'). This
had two problems:
1. get_store_prefix() is module-scoped, so from the prometheus module
it searched under prometheus's own namespace instead of cephadm's,
resulting in zero metrics being exported despite metric definitions
appearing at /metrics.
2. The firmware key was accessed as 'firmwares' (plural) but the
stored field is 'firmware' (singular), causing all firmware version
metrics to be silently empty.
Use node_proxy_fullreport() and node_proxy_firmware() via the
OrchestratorClientMixin instead. This routes through cephadm's
NodeProxyCache which handles KV access and firmware key compat
correctly. Follows the same pattern as set_cephadm_daemon_status_metrics()
and get_smb_metadata().
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
projects / ceph.git / log