Afreen Misbah [Tue, 30 Jun 2026 14:09:13 +0000 (19:39 +0530)]
mgr/prometheus: refactor hardware metrics
- export `firmware_version` labels from `node_proxy_storage_capacity_bytes` metrics to be used in device firmware panel
- improve iterations for performance - dropping redundant node_proxy_firmware() RPC; firmware data is already
present in the fullreport response, and removing unnecessary loops.
- replace health if/elif chain with HEALTH_STATUS_MAP dict lookup
- rename metrics to ceph_hardware_*, fix component labels and add tests
- added unit test cases
- add serial name, slot info to capacity metric
- fix health metrics showing ID instead of component name
- CPU/NVMe temp: change from stat to gauge panels with colored
arc thresholds (green/yellow/red) and proper °C units
- Health panels: wrap queries in max() aggregation to show
single worst-case value instead of overlapping series
- Pie charts: switch to donut style with visible legends
- Fan RPM: use locale unit for comma formatting instead of
short which auto-scaled to "K"
- Fix temperature panels units
- Add Device List and Platform Firmware table panels
Afreen Misbah [Thu, 25 Jun 2026 08:18:19 +0000 (13:48 +0530)]
monitoring: fix hardware Grafana dashboard and health metrics
- Fix fan repeating panels: set multi=true on fan_speeds template
variable so Grafana generates one panel per fan instead of one
- Remove TACH-only regex filter on fan_speeds template and AVG
Cooling query so all system fans are visible regardless of naming
- Replace duplicate Power Control panel with Network health panel
- Fix NVMe drive count to use storage_capacity_bytes{protocol=NVMe}
instead of counting temperature sensors (inaccurate proxy)
- Normalize all hostname filters to regex match (=~) for consistency
- Register hardware.libsonnet in dashboards.libsonnet so the
dashboard JSON is generated during ceph-mixin builds
- Add temperatures category to prometheus health metrics loop
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
(cherry picked from commit 04207ca6dcfc86c51845912848d9cbf1a7cb87d0)
Afreen Misbah [Thu, 25 Jun 2026 08:13:38 +0000 (13:43 +0530)]
mgr/prometheus: use orchestrator API for node-proxy hardware metrics
The hardware metrics exporter was reading cephadm's private KV store
directly via get_store_prefix('mgr/cephadm/node_proxy/data'). This
had two problems:
1. get_store_prefix() is module-scoped, so from the prometheus module
it searched under prometheus's own namespace instead of cephadm's,
resulting in zero metrics being exported despite metric definitions
appearing at /metrics.
2. The firmware key was accessed as 'firmwares' (plural) but the
stored field is 'firmware' (singular), causing all firmware version
metrics to be silently empty.
Use node_proxy_fullreport() and node_proxy_firmware() via the
OrchestratorClientMixin instead. This routes through cephadm's
NodeProxyCache which handles KV access and firmware key compat
correctly. Follows the same pattern as set_cephadm_daemon_status_metrics()
and get_smb_metadata().
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
(cherry picked from commit 6bb07c6e5393daeb72fe0cf3832955721e7b4b18)
* refs/pull/69773/head:
mClockScheduler: adjust mClock profile parameters to prevent backfill starvation
src/messages, osd: Calculate and set cost for subOpReads for mClock scheduler
osd/scheduler: Classify EC subOp reads according to op priority for mClock
osd/scheduler/mClockScheduler: Fix line alignments
osd/scheduler/mClockScheduler: Log the size of high priority queues.
* refs/pull/69113/head:
rgw/datalog: DataLogBackends::trim_entries: fix crash when target_gen > head_gen
test/rgw/datalog: test for trim_entries with max_marker
rgw/datalog: also handle errors properly for fifo trim
neocls log trimming (time based): fix infinite loop on ENODATA
neocls log trimming (marker based): fix infinite loop on ENODATA
test/neocls/log trimming: reproduce log trimming can go into an infinite loop
Bootstrap fails on v20.2.0 upgrade because of
cephadm binary and ceph image version mismatch.
This fixes following problem in bootstrap by using
tentacle cephadm binary:
```
Error: Container release tentacle != cephadm release umbrella; please use matching version of cephadm (pass --allow-mismatched-release to continue anyway)
```
Gil Bregman [Mon, 23 Feb 2026 10:56:54 +0000 (12:56 +0200)]
nvmeof: Change the NVMEOF image version to 1.7 Fixes: https://tracker.ceph.com/issues/75097 Signed-off-by: Gil Bregman <gbregman@il.ibm.com>
(cherry picked from commit 02587347b0a4e7ae1d7f5d738bd33808e2d56bc9)
Conflicts:
src/python-common/ceph/cephadm/images.py
The indentation was changed between the main branch and tentacle which
caused a conflict
Afreen Misbah [Wed, 27 May 2026 00:07:38 +0000 (05:37 +0530)]
mgr/dashboard: fix nested shell quoting in cephadm e2e start-cluster
with_libvirt wraps commands in sg libvirt -c "$1", adding an extra
shell layer. Nested double quotes inside the outer double-quoted
string caused the argument to be split — with_libvirt received a
truncated $1, producing "Unterminated quoted string" on the remote
shell.
Drop the unnecessary inner double quotes around cephadm shell
arguments since cephadm shell accepts the command as separate args.
Use single quotes for the grep pattern inside the double-quoted
string so it survives the sg subshell.
Nizamudeen A [Thu, 2 Apr 2026 10:55:19 +0000 (16:25 +0530)]
mgr/dashboard: run kcli commands in libvritd group
Also https://github.com/ceph/ceph-build/pull/2562/ Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 8293aa38bd9c9abca6353ebbdb64401bf3263017)
ceph-volume: skip internal raid mirror LVs in inventory
ceph-volume inventory started including all LVM mapper devices after c06bee965f1. On hosts with raid mirrored system volumes, that pulls in
hidden legs like var_rmeta_0 which have no /dev/vg/lv node and makes
cephadm's ceph-volume inventory call fail.
Skip those internal LVs in get_devices() and avoid rewriting the device
path to a missing lv_path in Device._parse().
ceph-volume: fix inventory without /dev/vg/lv (slashed paths)
Ths makes ceph-volume use UdevData.preferred_block_path() in
get_devices() so it keeps /dev/vg/lv (slashed path form) when
it exists, else /dev/mapper/<name> (dashed path form).
This is needed for thin-pool LVs and environments where udev
does not create slashed paths.
mClockScheduler: adjust mClock profile parameters to prevent backfill starvation
Adjust the 'background_best_effort' queue parameters across the
three standard mClock profiles (high_client_ops, balanced, and
high_recovery_ops) to ensure best effort ops are not starved.
Previously, the 'background_best_effort' queue carried a default allocation
of 0% (MIN) reservation and a weight of 1 under these profiles. When
concurrent client traffic is dense, the zero-reservation for example completely
starves backfill sub-ops (MSG_OSD_EC_READ) on pools with
'allow_ec_optimizations' set to false. This starvation forces the Primary OSD
to hold internal BlueStore transactions and PG object locks for extended
windows, causing severe client median (50th) latency inflation.
To prevent background starvation and resolve the effects of the primary lock
retention, the profile configurations are tuned as follows:
The following profile changes forces low-cost sub-ops to clear out of peer
queues rapidly to drop primary locks, which helps improve the client
completion latency and tail latency (95th, 99th and 99.5th) percentile.
1. high_client_ops profile:
- Grant 'background_best_effort' a safe 5% minimum reservation.
- Scale the queue weight to 4.
2. balanced profile:
- Grant 'background_best_effort' a 5% minimum reservation.
- Set the queue weight to 2.
3. high_recovery_ops profile:
- Grant 'background_best_effort' a 5% minimum reservation.
- Set the queue weight to 2.
4. Modify the mClock config reference documentation to reflect the tuning
changes to the best-effort QoS parameters across the profiles.
Note on Proportional Scaling Compatibility:
Configuring these changes shifts total reservations to 105% (e.g., 50%
client + 50% recovery + 5% best-effort under the Balanced profile). Under
heavy concurrent saturation, mClock's internal controls resolves this
gracefully via proportional down-scaling, preserving the underlying
device bandwidth limits for different classes of clients. For example instead
of the client being allocated 50% bandwidth, a slightly lower reservation is
allocated while shifting the remaining bandwidth to the best-effort queue.
This minor scaling shift is virtually unnoticeable to the client application,
but it prevents the internal queue deadlocks.
Conflicts:
src/common/mclock_common.h
- src/common/mclock_common.h doesn't exist in tentacle.
mclock_common.h/cc was created as a refactor in later releases to share
common bits between classic and crimson OSDs. Introduced in PR:
https://github.com/ceph/ceph/pull/63516.
Therefore, the profile parameters change is added to the original
location in src/osd/mClockScheduler.cc.
src/messages, osd: Calculate and set cost for subOpReads for mClock scheduler
Previously, sub-op reads returned a hardcoded cost of 0, bypassing
mClock's background bandwidth and tag calculation mechanisms. This
allowed backfill operations to proceed un-metered, occasionally causing
backend resource contention and driving up client tail latencies.
Cost is calculated based on whether the complete chunk/shard or a subchunk
needs to be read. The possible cases are:
1. Read the complete chunk aligned length:
- Cost is set to the length of the chunk aligned extent size.
2. Fragmented reads:
- Consider the subchunk length and count to calculate the cost.
- compute_cost evaluates the exact layout of fragmented shard bytes on
disk by summing up the active subchunk allocations exactly once
(`fragmented_shard_bytes += k.second * subchunk_size`).
- Linear Extent Scaling: Scale the baseline footprint cleanly by
multiplying it against the true count of read extents (`tl.size()`),
achieving a highly efficient O(N) time complexity.
This linear cost model is compatible with pools running with
'allow_ec_optimizations' set to true. Under the FastEC optimized
pipeline, most operations are unified and bypass fragment slicing,
meaning requests will primarily match the Case 1 chunk-aligned path.
In Case 2 where applicable, the O(N) loop ensures that cost will
scale proportionally according to the layout.
It is important to note that the amount of data to read was set to an upper
bound defined by osd_recovery_max_chunk (8 MiB) and was rounded up to the
stripe width. The reason for setting a higher than actual upper bound is that
there may be cases where the object doesn't have the xattrs yet to determine
its size. Therefore, the amount to read was ultimatly set to ~(8 MiB / k)
where k is the number of data shards. This can cause mClock to prolong
the recovery times as items stay longer in the queue. To address this, the
amount to read is set to the remaining length of the object to recover
if the object size is known. Otherwise, the amount to read is set to the
recovery chunk size as before. Therefore, in some cases, only the first
recovery read could be costly if the object context is not known.
The MOSDECSubOpRead class introduces the following:
- cost member. This necessitates an increment to the HEAD_VERSION and
appropriate handling within the encode and decode methods.
- compute_cost() that is called when creating the message by
ECCommonL::ReadPipeline::do_read_op(). This calls into ECSubRead::cost()
that performs the actual calculations to set the cost based on the cases
mentioned above.
- The same sequence applies to the EC optimized path in
ECCommon::ReadPipeline::do_read_op().
Conflicts:
src/osd/ECMsgTypes.h
- Removed a couple of variable declarations related to recovery of OMAP
header and entries in EC pools which is yet to be backported to
tentacle. See commit: a5f4a4902075e343df154da61e3d205d2bd2a5d5.
osd/scheduler: Classify EC subOp reads according to op priority for mClock
The change brings MSG_OSD_EC_READ into the fold of mClock scheduler. This
improves the scheduling of client and other classes of operation as they
are no longer unnecessarily preempted by the 'immediate' queue.
EC SubOps are now handled as follows:
- EC SubOp reads generated during recovery will either go into the
'background_recovery' or 'background_best_effort' class based on
the recovery priority set for the op. EC SubOp reads generated due
to client will continue to be classified as 'immediate'.
- EC SubOp writes generated as a result of client operations will
continue to be classified as 'immediate'.
- EC SubOp replies are considered high priority and therefore
continue to be classed as 'immediate'.
Conflicts:
src/osd/scheduler/OpSchedulerItem.h
- enum SchedulerClass doesn't exist in tentacle.Therefore, the original
enum op_scheduler_class is used.
enum SchedulerClass was introduced when mclock_common.h/cc was created
as a refactor in later releases to share common bits between classic and
crimson OSDs. Introduced in PR: https://github.com/ceph/ceph/pull/63516.
Kefu Chai [Sat, 13 Jun 2026 01:50:09 +0000 (09:50 +0800)]
python-common/cryptotools: stop using the removed X509Req API
pyOpenSSL deprecated OpenSSL.crypto.X509Req in 24.2.0 (2024-07-20) and
removed it in 26.3.0 (2026-06-12). as we don't pin pyopenssl, CI picked
up the new release, and create_self_signed_cert() started failing with:
AttributeError: module 'OpenSSL.crypto' has no attribute 'X509Req'
this took down run-tox-mgr, run-tox-mgr-dashboard-py3 and the mypy check.
we only used X509Req to build a subject name and then copied it into the
X509 cert. so drop it, and set the subject on the cert directly. the
resulting cert stays the same: subject from dname, issuer set to the same
subject, self-signed.
Kefu Chai [Tue, 23 Jun 2026 07:43:28 +0000 (15:43 +0800)]
mgr/dashboard: skip the table when an nvmeof cli result has no columns
The dashboard leaves prettytable unpinned. prettytable commit 2574492 ("Apply
some Pylint rules (PLR)", #436) rewrote _stringify_row()'s row_height as
`max(_get_size(c)[1] for c in row)`, which raises ValueError("max() iterable
argument is empty") on a row with no cells. The change is undocumented and
shipped in 3.18.0; get_string() trips on it when a table has a row but no
columns.
AnnotatedDataTextOutputFormatter builds such a table for an empty result, or
one whose only field is status or error_message, so NvmeofCLICommand.call()
returns -EINVAL and the command fails. This broke run-tox-mgr-dashboard-py3
once the tox virtualenv picked up prettytable 3.18.0.
Return an empty string when there are no columns instead of formatting a
degenerate table.
Conflicts:
src/pybind/mgr/dashboard/tests/test_nvmeof_cli.py
Resolution: tentacle has only the empty-result test case; updated its
expected stdout to ''
mgr/DaemonServer: Aggregate and globally sort OSDs for ok-to-upgrade
The 'ok-to-upgrade' command output sorting did not scale accurately
when target CRUSH buckets contained multiple child buckets (e.g., a
chassis containing multiple hosts). OSDs were previously sorted
individually per child bucket and appended sequentially. This created
fragmented, per-host sort segments rather than a globally sorted list
for the parent bucket.
Changes:
1. Fix the issue above by aggregating all child OSDs into a single vector prior
to executing a single, global sort operation based on PG counts. Additionally,
optimize memory efficiency and future-proof the logic by reserving continuous
vector blocks to avoid dynamic heap reallocations.
2. Add integration tests with chassis and rack based CRUSH hierarchies which
verifies the ok-to-upgrade functionality. In addition, the tests crucially
verify the order of OSDs returned is according to the ascending order of
acting PG count. Additionally, make minor fix-ups to lines that determine the
length of a list in JSON response by removing the redundant "| bc".
Alex Ainscow [Wed, 8 Apr 2026 10:49:58 +0000 (11:49 +0100)]
osd: Allow multiple objects with same version in missing list.
Most of the time, a single version in a PG can only correspond to a single object.
However, following a PG merge it is possible, even likely, that two objects will
have the same version. The PG Log works around this by discarding the log.
However, during backfill, it is possible for the missing list to be build with
these duplicate versions.
A recently added assert detected that this scenario was corrupting the reverse
missing list (rmissing). This behaviour has always existed, but was previously
unnoticed. It could cause some bugs and potentially loop-asserts on OSDs,
although mostly would not be noticed.
Here we fix this properly, by converting rmissing to a multimap. This is wrapped
in some insert functions, which assert that the rmissing list does not end up
with duplicate entries. The code is optimised for the case where there are no
duplicate versions.
Additionally, some of the old asserts have been rolled into the insert functions.
Fixes: https://tracker.ceph.com/issues/75778 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
(cherry picked from commit f3940400952b444a31f59b633fa3fa35437c87a9)
mgr: guard close_section calls in get_perf_schema_python
When a daemon exists in the daemon state map but has no perf counters
(e.g. an OSD that was running but is now down or destroyed), the loop
in get_perf_schema_python never executes, leaving prev_key_name empty
and no formatter sections opened. The unconditional close_section()
calls after the loop then trigger an assertion failure in PyFormatter
(cursor != root).
Add the same `if (!prev_key_name.empty())` guard that already protects
the close_section() calls inside the loop, so we only close sections
that were actually opened.
Nitzan Mordechai [Sun, 10 May 2026 06:51:08 +0000 (06:51 +0000)]
qa: ignore evicted client warnings for singletone bluestore
After adding mds client into singletone bluestore and increas debug
levels and bluestore_allocator=stupid client is slow enough that MDS
evicts it after ~302 seconds
mgr/DaemonServer: clarify ok-to-upgrade error message for CRUSH buckets
Refine the error string in DaemonServer.cc returned by the
ok-to-upgrade command when OSDs in a CRUSH bucket cannot be upgraded.
The original message is ambiguous. It fails to clearly convey that
stopping *any* individual OSD in that specific bucket will drop PGs
offline, meaning no OSDs within that bucket can be safely upgraded at
this time.
Update the phrasing to explicitly state that at least X PGs will go offline
if any OSD out of the total count in that CRUSH bucket is stopped. Also
standardize on capitalized acronyms (PG, OSD, CRUSH) and wrap the bucket
name in single quotes for better log readability.
Vallari Agrawal [Wed, 27 May 2026 12:17:55 +0000 (17:47 +0530)]
qa/suites/nvmeof: ignore "have only 1 nvmeof gateway"
Add "have only 1 nvmeof gateway" to ignorelist.
NVMEOF_SINGLE_GATEWAY is already part of ignorelist
but tests sometimes fail on "have only 1 nvmeof gateway".
Thrasher or scalability tests can trigger this but there
are enough asserts to ensure all expected gateways are
up, we can safely ignore this healthcheck warning.
ceph-volume: has_bluestore_label checks all bluestore label replica offsets
BlueStore replicates the block device label at fixed offsets (0 and
multiples of 1Gb up to 1000gb). has_bluestore_label() only read the
first 22 bytes, so disks with a wiped primary label but intact
replicas are missed.
with this commit, has_bluestore_label() scans each known offset with
seek/read and compares the ASCII prefix as bytes.
The thrashosds task is occasionally restarting OSDs and mon/mgr log
warnings are wrongly flagging this as a problem.
Fixes: https://tracker.ceph.com/issues/76747 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit a1430973f2c3e8758890447a1de3b906da77b60e)
Conflicts:
qa/suites/upgrade/reef-x/stress-split/1-start.yaml
qa/suites/upgrade/squid-x/stress-split/1-start.yaml
Resolution: different upgrade suites, just add the change manually
libcephsqlite: ensure atexit handlers are registered after openssl
When the sqlite3 executable encounters an error with .bail=on, it will
make a call to exit(). The atexit() handlers will execute in LIFO order.
We need to ensure that openssl (before OpenSSL 4.0 [1]) atexit handlers are
registered before libcephsqlite.
[1] http://github.com/openssl/openssl/commit/31659fe32673a6bd66abf3f8a7d803e81c6ffeed (OpenSSL 4.0 no longer arms `OPENSSL_cleanup()` function as an `atexit(3)`)
Fixes: https://tracker.ceph.com/issues/59335 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 7949cd5f12eb7cc0dc85fd1b5c1d795fad1df922)
Ronen Friedman [Tue, 26 May 2026 15:29:32 +0000 (15:29 +0000)]
osd/scrub: 'repairing' scrubs allowed at all times
Fix ScrubJob::observes_allowed_hours() to not block 'repairing'
scrubs outside of the allowed hours. This allows repair scrubs
to run at any time or day-of-week.
The fixed behaviour matches the documented requirements.
Redouane Kachach [Fri, 29 May 2026 09:09:44 +0000 (11:09 +0200)]
qa/tasks: capture CommandCrashedError when running nvme list cmd
The safe_while retry loop does not catch exceptions, so a
CommandCrashedError from `nvme list` bypasses it entirely. Catch
CommandCrashedError and continue the retry loop instead.
ceph-volume: retry lvs after empty result and "devices file is missing" stderr
When LVM's devices file is out of sync with the runtime device view (common
in teuthology/container namespaces with multipath), `lvs` can exit 0 with
empty stdout and only stderr warnings about missing mapper entries.
It can leave get_lvs() empty and cause Device() to fall through to lsblk on a
vg/lv path which can produce a misleading "not a block device" error.
With this fix, ceph-volume retries once with 'use_devicesfile=0' when it
detects this specific pattern.
Bill Scales [Fri, 15 May 2026 14:39:25 +0000 (15:39 +0100)]
osd: Fix bug when calculating min_peer_features
PeeringState calculates the minimum set of features for the set
of OSDs within a PG. There is a bug when the peer info has
already been cached where these peers features are not included
in the calculation. This can lead to the min feature set
including features that not all OSDs have.
Previously this just made some asserts less aggressive than they
should have been. Pull request https://github.com/ceph/ceph/pull/57740
uses min_peer_features to decide how to encode messages to other OSDs.
Midway through an upgrade this bug can cause an OSD to send
the wrong version of a message to a downlevel OSD causing
it to abort.
Fixes: https://tracker.ceph.com/issues/76600 Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
(cherry picked from commit ce5882778db9b92b14a6bb5eca22e3cecf2be9dc)
Casey Bodley [Wed, 25 Mar 2026 16:33:40 +0000 (12:33 -0400)]
qa/rgw: remove ragweed from verify subsuite
it's currently broken with newer python on rocky 10 and ubuntu 24
(tracked in https://tracker.ceph.com/issues/72500) and doesn't provide
interesting test coverage outside of rgw/upgrade
projects / ceph.git / log