Matan Breizman [Mon, 8 Jun 2026 12:45:20 +0000 (15:45 +0300)]
crimson/osd: fix CyanStore boot with crimson OSD
- Remove dead cyan_store.h includes from pg.cc and shard_services.cc.
- Fix CyanStore::mkfs to write type="cyanstore" instead of "memstore"
- Handle missing device class in OSD::start: stores that return an empty
device class (e.g CyanStore) now fall back to is_rotational=false with
a warning instead of throwing
monitoring/ceph-mixin: scope test queries per dashboard
get_dashboards_data() keeps every query in a single map keyed by
"<panel title>-<legendFormat>", but that id is not unique: several
dashboards have a panel titled "IOPS", "OSDs" or "Throughput". The
dashboard read last overwrites the earlier entry, so one query shadows
another and never gets tested. glob() walks the files in filesystem
order, so which query survives, and whether the test passes, depends on
readdir.
run-tox-promql-query-test hit this in "Test IOPS Read"
(ceph-cluster.feature): the scenario feeds ceph_osd_op_r but the id
resolved to a CephFS pool panel.
pool_id is "UNSET VARIABLE" because the scenario does not set $mdatapool,
and no ceph_pool_rd series were fed, so the query returns nil. It only
fails when readdir returns cephfsdashboard after ceph-cluster-advanced,
so the earlier change that walked nested rows and made the winner
deterministic did not fix it; it fixed which query wins, not that the
wrong one can win.
Key the queries per dashboard (data['queries'][<dashboard>][<id>]) and
have each .feature name its dashboard in a Background step. The lookup is
confined to that dashboard, so a title/legend used elsewhere no longer
shadows it. A duplicate within one dashboard is still an error, unless it
is in a collapsed row.
ceph-cluster.feature covers ceph-cluster-advanced; the rest map to one
dashboard each.
Ronen Friedman [Tue, 30 Jun 2026 12:55:33 +0000 (12:55 +0000)]
crimson/os/alienstore: make CnLog::_flush non-blocking
CnLog::_flush() calls alien::submit_to(...).wait(), which blocks the
log flusher thread until the seastar reactor finishes outputting the
log entries. When the reactor writes to stderr and the pipe buffer is
full (e.g. two OSDs on the same host both dumping at debug level 20),
the reactor itself blocks on the write() syscall. Meanwhile BlueStore
threads filling the log buffer past m_max_new block in submit_entry()
waiting for the flusher - which is waiting for the reactor - which is
blocked on I/O. The kv thread cannot fire commit callbacks while it is
blocked on logging, so store transactions (including osdmap persistence)
hang indefinitely.
Fix by moving the entries into the lambda capture so _flush() can
return immediately without waiting.
This is a corrected version of the approach in commit 511af83e274
("crimson/os/alienstore/alien_log: _flush concurrently", PR #55262),
which was reverted in 4d52d1d22b7 because its lambda captured the
EntryVector by reference - the caller would clear/reuse the vector
before the lambda finished, causing a "pure virtual method called"
crash (https://tracker.ceph.com/issues/64140). The fix here avoids
that by swapping the entries out of the caller's vector and moving
them into the lambda's capture, so the lambda owns them for their
full lifetime.
Fixes: https://tracker.ceph.com/issues/77826
Fixes (failure of the 1st attempt): https://tracker.ceph.com/issues/64140 Assisted-by: Claude <claude.ai> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
mgr/cephadm: Add documentation for RGW keyring capability dependencies
This serves as a reminder for maintainers to verify whether capability
changes in the RGW service should also be applied to SMB and NFS
services to maintain consistency across services that use librgw.
mgr/dashboard: support clone_by_snap_id flag in clone API for group snapshots
The clone API endpoint POST /api/block/image/{image_spec}/snap/{snapshot_name}/clone
only accepted a snapshot name, which uses rbd_clone3 (clone by name). This only works
for snapshots in the user namespace.
Group snapshots (with .group prefix, in RBD_SNAP_NAMESPACE_TYPE_GROUP namespace)
require rbd_clone4 (clone by snap ID). The rbd Python binding already supports this:
passing an int for p_snapshot triggers rbd_clone4.
Add an optional clone_by_snap_id boolean flag to the clone endpoint. When set, the
path parameter is treated as a numeric snap ID and passed directly as an int to
rbd.RBD().clone(). Clone format 2 is enforced because this flag is intended for
non-user namespace snapshots which cannot be protected (a prerequisite for clone
format 1).
Usage:
POST /api/block/image/{image_spec}/snap/{snap_id}/clone
{"clone_by_snap_id": true, ...}
Fixes: https://tracker.ceph.com/issues/77875 Signed-off-by: Imran Imtiaz <imran.imtiaz@uk.ibm.com> Assisted-by: IBM Bob
mgr/smb: Allow pool creation for empty RGW buckets
When an RGW bucket is empty, SMB clients cannot create directories,
files, or upload data because the default.rgw.buckets.data pool does
not yet exist. The previous CephX capabilities ('mon allow r') did not
permit pool creation during the first write operation.
The fix changes the mon capability from 'allow r' to 'allow *' to
enable pool creation on the first write to an empty RGW bucket. Once
the pool exists, subsequent operations succeed normally.
Sun Yuechi [Thu, 2 Jul 2026 09:26:59 +0000 (02:26 -0700)]
common/crc32c: stop using gp/tp as scratch in RISC-V Zbc CRC32C
crc32c_zbc's fold-by-four loop used gp (x3) and tp (x4) as scratch for
two folding temporaries. Save/restore around the loop is not enough:
they are ABI-reserved, so a signal delivered mid-loop runs its handler
with a corrupted tp, and the first TLS access there faults.
On riscv64 this reliably crashes the crimson/seastore unittests
unittest-transaction-manager and unittest-omap-manager, where seastar's
stall-detector timer fires often: the process dies with SIGSEGV in the
linker's TLS path with garbage in gp/tp.
Fixes: https://tracker.ceph.com/issues/77904 Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
vstart.sh: only skip fs volume create when there's no mgr to run it
The mgr volumes module handles fs volume ls/create, not the monitor, so
without a mgr (CEPH_NUM_MGR=0) the "wait for volume module to load" loop
spun forever. Open-coding fs new unconditionally, my first attempt,
changed pool creation for every vstart user and dropped the wait, so
cephfs tests calling `fs volume ...` right after vstart.sh returns could
race a module still mid-load.
create_fs_volume() branches on CEPH_NUM_MGR instead. With a mgr it still
calls fs volume ls/create. Without one it runs the plain mon commands fs
volume create wraps: osd pool create cephfs.<name>.{meta,data}, then fs
new <name> <meta> <data>. The wait stays at its one call site above the
per-filesystem loop, so it runs once regardless of CEPH_NUM_FS, not once
per filesystem.
Verified against a local vstart: mgr present, fs volume ls polls once
then creates CEPH_NUM_FS=2 filesystems with no repeated poll;
CEPH_NUM_MGR=0, no poll, fs new creates 'a' directly, HEALTH_OK.
crimson/osd: skip store flush for empty peering transactions
dispatch_context_transaction submits a real empty transaction to
AlienStore even when the peering context has no store ops. This
goes through the AlienCollection lock and thread pool slot machinery.
Under load from broadcast_map_to_pgs (many PGAdvanceMap operations
running concurrently with recovery), some of these empty transactions
get stuck inside AlienStore's pipeline and never reach BlueStore,
causing PGAdvanceMap to hang at complete_rctx indefinitely.
Fix by skipping the store round-trip for empty transactions. The
on_commit callbacks are still collected and fired, preserving the
fix from b0f5e1086a6 ("crimson/osd/pg: also trigger callbacks for
empty peering transactions"). The store flush that commit added was
incidental to the callback fix and is unnecessary here: the peering
pipeline's exclusive process stage already guarantees that prior
peering transactions on the same PG have completed.
Assisted-by: Claude <claude.ai> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Karthik U S [Wed, 10 Jun 2026 05:48:25 +0000 (11:18 +0530)]
mgr/mirroring,tools/cephfs_mirror: Handle checkpoint state transition
When a new checkpoint is being added or when the daemon gets restarted,
it will check whether the newly created checkpoint or any other old
checkpoints have already been mirrored onto the remote peer. If so, it
will transition to the correct state by checking for the highest snap id
present on the remote, and setting all the checkpoints which have snap id
lesser than or equal to that of the remote to COMPLETE. This is done by
sending an acquire notification from the mirroring module to the mirror
daemon, which is handled in the add_directory path, by adding the directory
to be checked for the state transition in the tick thread.
This path gets triggered:
a) when the daemon gets restarted
b) when the peer mapping changes
c) by sending the acquire notification from checkpoint add/now CLIs.
d) when mirroring module restarts
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Karthik U S [Sat, 16 May 2026 00:50:03 +0000 (06:20 +0530)]
tools/cephfs_mirror: update checkpoint status during snapshot sync
When the sync completes or fails for a checkpointed snapshot, transition
the status of that checkpoint from CREATED/FAILED to COMPLETE/FAILED in
the do_sync_snaps() along with the timestamp of the event.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
Afreen Misbah [Tue, 30 Jun 2026 14:09:13 +0000 (19:39 +0530)]
mgr/prometheus: refactor hardware metrics
- export `firmware_version` labels from `node_proxy_storage_capacity_bytes` metrics to be used in device firmware panel
- improve iterations for performance - dropping redundant node_proxy_firmware() RPC; firmware data is already
present in the fullreport response, and removing unnecessary loops.
- replace health if/elif chain with HEALTH_STATUS_MAP dict lookup
- rename metrics to ceph_hardware_*, fix component labels and add tests
- added unit test cases
- add serial name, slot info to capacity metric
- fix health metrics showing ID instead of component name
- CPU/NVMe temp: change from stat to gauge panels with colored
arc thresholds (green/yellow/red) and proper °C units
- Health panels: wrap queries in max() aggregation to show
single worst-case value instead of overlapping series
- Pie charts: switch to donut style with visible legends
- Fan RPM: use locale unit for comma formatting instead of
short which auto-scaled to "K"
- Fix temperature panels units
- Add Device List and Platform Firmware table panels
Afreen Misbah [Thu, 25 Jun 2026 08:18:19 +0000 (13:48 +0530)]
monitoring: fix hardware Grafana dashboard and health metrics
- Fix fan repeating panels: set multi=true on fan_speeds template
variable so Grafana generates one panel per fan instead of one
- Remove TACH-only regex filter on fan_speeds template and AVG
Cooling query so all system fans are visible regardless of naming
- Replace duplicate Power Control panel with Network health panel
- Fix NVMe drive count to use storage_capacity_bytes{protocol=NVMe}
instead of counting temperature sensors (inaccurate proxy)
- Normalize all hostname filters to regex match (=~) for consistency
- Register hardware.libsonnet in dashboards.libsonnet so the
dashboard JSON is generated during ceph-mixin builds
- Add temperatures category to prometheus health metrics loop
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
Afreen Misbah [Thu, 25 Jun 2026 08:13:38 +0000 (13:43 +0530)]
mgr/prometheus: use orchestrator API for node-proxy hardware metrics
The hardware metrics exporter was reading cephadm's private KV store
directly via get_store_prefix('mgr/cephadm/node_proxy/data'). This
had two problems:
1. get_store_prefix() is module-scoped, so from the prometheus module
it searched under prometheus's own namespace instead of cephadm's,
resulting in zero metrics being exported despite metric definitions
appearing at /metrics.
2. The firmware key was accessed as 'firmwares' (plural) but the
stored field is 'firmware' (singular), causing all firmware version
metrics to be silently empty.
Use node_proxy_fullreport() and node_proxy_firmware() via the
OrchestratorClientMixin instead. This routes through cephadm's
NodeProxyCache which handles KV access and firmware key compat
correctly. Follows the same pattern as set_cephadm_daemon_status_metrics()
and get_smb_metadata().
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
mgr/cephadm: remove spec.config options when a service is removed
For each key in spec.config, validate the option name and run
config rm on the service's config section. Call this from
remove_service() before spec_store.rm() schedules daemon removal.
mgr/dashboard: fix stale nvmeof spec tests from a semantic merge conflict
mgr-dashboard-frontend-unittests has been failing on main since 2026-07-01,
unrelated to whatever PR triggers it. Bisected to cfe12305fb1, the merge
commit for PR #68180.
6182e67ea4 ("mgr/dashboard: NVMe Onboarding Setup Cards") moved gateway
group selection out of NvmeofSubsystemsComponent and
NvmeofNamespacesListComponent into a new NvmeofGatewayGroupFilterComponent,
dropping gwGroups, updateGroupSelectionState, handleGatewayGroupsError,
onGroupClear, and getSubsystems from both. Clean on its own.
PR #68180 forked from main in April, before that refactor existed, and
added three new tests against the old API. Also clean on its own, and never
rebased. Both PRs only touch lines the other doesn't, so when #68180 merged
34 minutes after 6182e67ea4, git's 3-way merge combined them without a
conflict. The merge is the only place the bug exists: new tests calling
methods the sibling parent had already deleted. TypeError at runtime,
TS2339/TS2551 from tsc, every run since. CI never caught it: GitHub was
showing #68180's make check as green from a run a full day older than 6182e67ea4.
nvmeof-namespaces-list.component.spec.ts: the dedup test is real, current
logic, it just called the old listNamespaces() instead of fetchData().
Fixed the call. The other two duplicate NvmeofGatewayGroupFilterComponent's
own spec; removed.
nvmeof-subsystems.component.spec.ts: same story, all three duplicate the
filter component's spec; removed.
Both removed error tests also checked preventDefault() on the error object,
which had no coverage elsewhere. Added that to
NvmeofGatewayGroupFilterComponent's spec, and hit a second bug while writing
it: the file's throwError(() => err) is the RxJS 7 factory overload, this
project is on RxJS 6.6.3, so it was emitting the arrow function itself as
the error. The existing error test never noticed since it only checks side
effects, not the error's identity. Used throwError(err) instead.
Verified: jest passes on all three spec files (was 6 failing), tsc -p
tsconfig.spec.json --noEmit is clean (was 17 errors), full block/nvmeof
sweep is 29/29 suites, 216/216 tests.
python3_pkgversion picked the native python3 package name on RHEL7, where
python3 was python34/python36 before 7.6 (cea9d18c). RHEL7 and RHEL8 are
gone, so on every supported distro (el9, el10, fedora, suse, openEuler) it
is always 3.
Replace python%{python3_pkgversion} with python3 and drop the fallback
default. No supported distro overrides it to build against a non-default
python flavor, so that capability goes too.
ceph.spec.in: use python3dist() for Python BuildRequires
Many Python deps were split across %if suse / %if rhel branches only
because openSUSE spells them CamelCase (PrettyTable, PyYAML, Sphinx,
CherryPy, Routes, Jinja2) and fedora/rhel/openEuler lowercase. The
python3dist() provides resolve the same on el9, el10, openEuler and
openSUSE, so those splits collapse to single lines.
Spell the deps python%{python3_pkgversion}dist(), matching the spec's
existing python%{python3_pkgversion}-Foo convention. Version constraints
and feature guards are preserved.
The dist name is the PyPI project name, not always the rpm name minus
python3-: python3-dateutil provides python3dist(python-dateutil), and
python3-saml provides python3dist(python3-saml).
Left as package names: the bare interpreter; python3-devel and numpy-devel,
which have no python3dist() provide; the build toolchain (setuptools,
Cython, pip, wheel); and the ceph-internal python3-* subpackages, which are
versioned = release deps.
ceph.spec.in: use pkgconfig() for library BuildRequires
Library BuildRequires used per-distro -devel package names, some split
across %if suse / %if rhel branches only because the names differ. The
pkgconfig() virtual provides resolve identically on every rpm distro; the
spec already relies on them for fuse3, systemd and udev.
Switch every library dependency that ships a .pc to pkgconfig(), preserving
version constraints and feature guards. This collapses the suse-vs-rhel
naming splits (nss, keyutils, openssl, ldap, ibverbs, rdmacm, lz4, thrift,
libcryptopp) and the dedicated split blocks (nlohmann_json, lttng-ust,
babeltrace, expat) into single lines, and folds the duplicated xmlsec1 devel
packages together.
Left as package names: libaio and xfsprogs ship no reliable .pc; gperftools
carries version and tcmalloc-variant logic; numa is unconditional on rhel
but crimson-only on suse; and the xmlsec1 runtime backend plugins are
runtime test deps, not build-time pkgconfig() deps.
RHEL 8 and CentOS 8 left the supported test matrix in 2023 (449db416, 7a1dce1e). RHEL 8 has also been unbuildable since CMakeLists.txt began
requiring Python >= 3.9: it ships python 3.6 by default, so cmake refuses
to configure. The spec still carried the el8 conditionals, so assume
rhel >= 9 and drop them:
- remove branches reachable only on rhel <= 8: the gperftools 2.6.1
BuildRequires, the tracker-36508 tcmalloc-libs requirement, and the
dataclasses backport, which also targeted that dead python 3.6
- collapse "rhel >= 8" and "rhel >= 9" to plain "rhel"; with the floor at
9 this is a no-op for rhel 9/10, fedora, suse and openEuler
- keep the rhel 9-vs-10 checks (== 9, < 10, >= 10)
Ilya Dryomov [Mon, 29 Jun 2026 11:39:50 +0000 (13:39 +0200)]
qa/suites/rbd: use client.0 entity in migration-external tests
Currently client.admin is passed for client_name and that doesn't
exercise client_name handling much as client.admin is the default.
When deploying multiple clusters the ceph task distributes keyrings
with only the key for the initial monitor and client.admin key. Other
keys (e.g. client.0) are present only on their respective clusters, so
client.0's key for cluster2 needs to be obtained on cluster1 explicitly
with "ceph auth get".
Leonid Chernin [Mon, 2 Feb 2026 06:01:14 +0000 (08:01 +0200)]
librbd/migration/NativeFormat: support specifying mon_host and key via spec
migration:
-take secret_key from spec
-take mon_host from the spec
-ignore source keyring, source ceph.conf - bypass them
-added validations of the new way schema
-get key from the KV DB if key in spec is actually key-ref
Fixes: https://tracker.ceph.com/issues/68177 Signed-off-by: Leonid Chernin <leonidc@il.ibm.com> Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
projects / ceph.git / log