Patrick Donnelly [Sat, 11 Jul 2026 01:18:14 +0000 (21:18 -0400)]
Merge PR #69708 into tentacle
* refs/pull/69708/head:
node-proxy: expose FCM stats
node-proxy: rename firmwares to firmware with legacy aliases
node-proxy: override with atollon specific
node-proxy: add temperatures and fan speed
Reviewed-by: Adam King <adking@redhat.com> Reviewed-by: Afreen Misbah <afreen@ibm.com>
Patrick Donnelly [Fri, 10 Jul 2026 20:26:13 +0000 (16:26 -0400)]
Merge PR #69775 into tentacle
* refs/pull/69775/head:
tentacle: qa/rgw: Remove 'force-branch' from s3tests configs
tentacle: qa/rgw: Run s3-tests from within the Ceph repo
tentacle: test/rgw: Include s3-tests in Ceph repo
Patrick Donnelly [Fri, 10 Jul 2026 20:17:51 +0000 (16:17 -0400)]
Merge PR #69742 into tentacle
* refs/pull/69742/head:
tests/neorados: fix ceph_test_neorados_completions being not installed
common/async: async_cond::notify/cancel must post to handler's associated executor
test/neorados: test cross-executor completions
osdc: deliver neorados completions to associated executor
Afreen Misbah [Tue, 23 Jun 2026 13:30:22 +0000 (19:00 +0530)]
mgr/dashboard: fix bind address regression from CherryPy isolation
The CherryPy isolation refactor (PR #67227) accidentally changed the
dashboard bind address from wildcard (*:8443) to mon_ip:8443. The
get_mgr_ip() replacement was originally only for URI generation, but
the refactor passed the mutated address to CherryPyMgr.mount() as the
actual socket bind address.
This breaks the management gateway when its VIP is not on the same
interface as mon_ip, as the dashboard becomes unreachable on other
interfaces.
Preserve the original wildcard address for binding and only use
get_mgr_ip() for the advertised URI. Add regression test to prevent
future confusion between bind_addr and server_addr.
Afreen Misbah [Tue, 30 Jun 2026 14:09:13 +0000 (19:39 +0530)]
mgr/prometheus: refactor hardware metrics
- export `firmware_version` labels from `node_proxy_storage_capacity_bytes` metrics to be used in device firmware panel
- improve iterations for performance - dropping redundant node_proxy_firmware() RPC; firmware data is already
present in the fullreport response, and removing unnecessary loops.
- replace health if/elif chain with HEALTH_STATUS_MAP dict lookup
- rename metrics to ceph_hardware_*, fix component labels and add tests
- added unit test cases
- add serial name, slot info to capacity metric
- fix health metrics showing ID instead of component name
- CPU/NVMe temp: change from stat to gauge panels with colored
arc thresholds (green/yellow/red) and proper °C units
- Health panels: wrap queries in max() aggregation to show
single worst-case value instead of overlapping series
- Pie charts: switch to donut style with visible legends
- Fan RPM: use locale unit for comma formatting instead of
short which auto-scaled to "K"
- Fix temperature panels units
- Add Device List and Platform Firmware table panels
Afreen Misbah [Thu, 25 Jun 2026 08:18:19 +0000 (13:48 +0530)]
monitoring: fix hardware Grafana dashboard and health metrics
- Fix fan repeating panels: set multi=true on fan_speeds template
variable so Grafana generates one panel per fan instead of one
- Remove TACH-only regex filter on fan_speeds template and AVG
Cooling query so all system fans are visible regardless of naming
- Replace duplicate Power Control panel with Network health panel
- Fix NVMe drive count to use storage_capacity_bytes{protocol=NVMe}
instead of counting temperature sensors (inaccurate proxy)
- Normalize all hostname filters to regex match (=~) for consistency
- Register hardware.libsonnet in dashboards.libsonnet so the
dashboard JSON is generated during ceph-mixin builds
- Add temperatures category to prometheus health metrics loop
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
(cherry picked from commit 04207ca6dcfc86c51845912848d9cbf1a7cb87d0)
Afreen Misbah [Thu, 25 Jun 2026 08:13:38 +0000 (13:43 +0530)]
mgr/prometheus: use orchestrator API for node-proxy hardware metrics
The hardware metrics exporter was reading cephadm's private KV store
directly via get_store_prefix('mgr/cephadm/node_proxy/data'). This
had two problems:
1. get_store_prefix() is module-scoped, so from the prometheus module
it searched under prometheus's own namespace instead of cephadm's,
resulting in zero metrics being exported despite metric definitions
appearing at /metrics.
2. The firmware key was accessed as 'firmwares' (plural) but the
stored field is 'firmware' (singular), causing all firmware version
metrics to be silently empty.
Use node_proxy_fullreport() and node_proxy_firmware() via the
OrchestratorClientMixin instead. This routes through cephadm's
NodeProxyCache which handles KV access and firmware key compat
correctly. Follows the same pattern as set_cephadm_daemon_status_metrics()
and get_smb_metadata().
Signed-off-by: Afreen Misbah <afreen@ibm.com> Assisted-by: Claude Signed-off-by: Afreen Misbah <afreen@ibm.com>
(cherry picked from commit 6bb07c6e5393daeb72fe0cf3832955721e7b4b18)
* refs/pull/69773/head:
mClockScheduler: adjust mClock profile parameters to prevent backfill starvation
src/messages, osd: Calculate and set cost for subOpReads for mClock scheduler
osd/scheduler: Classify EC subOp reads according to op priority for mClock
osd/scheduler/mClockScheduler: Fix line alignments
osd/scheduler/mClockScheduler: Log the size of high priority queues.
* refs/pull/69113/head:
rgw/datalog: DataLogBackends::trim_entries: fix crash when target_gen > head_gen
test/rgw/datalog: test for trim_entries with max_marker
rgw/datalog: also handle errors properly for fifo trim
neocls log trimming (time based): fix infinite loop on ENODATA
neocls log trimming (marker based): fix infinite loop on ENODATA
test/neocls/log trimming: reproduce log trimming can go into an infinite loop
Bootstrap fails on v20.2.0 upgrade because of
cephadm binary and ceph image version mismatch.
This fixes following problem in bootstrap by using
tentacle cephadm binary:
```
Error: Container release tentacle != cephadm release umbrella; please use matching version of cephadm (pass --allow-mismatched-release to continue anyway)
```
Gil Bregman [Mon, 23 Feb 2026 10:56:54 +0000 (12:56 +0200)]
nvmeof: Change the NVMEOF image version to 1.7 Fixes: https://tracker.ceph.com/issues/75097 Signed-off-by: Gil Bregman <gbregman@il.ibm.com>
(cherry picked from commit 02587347b0a4e7ae1d7f5d738bd33808e2d56bc9)
Conflicts:
src/python-common/ceph/cephadm/images.py
The indentation was changed between the main branch and tentacle which
caused a conflict
Aashish Sharma [Fri, 5 Jun 2026 07:07:38 +0000 (12:37 +0530)]
mgr/dashboard: Fix username validation for special characters by URL-encoding user lookup requests
When validating usernames, special characters such as '#' were not being URL-encoded before being appended to the user lookup endpoint.
Because the browser treats '#' as a URL fragment, the request path was truncated, causing the backend to return an incorrect response and the UI to display a misleading "User already exists" validation error.
ceph-volume: detect rotational media under dm-crypt for workqueue bypass
bypass_workqueue() was inspecting the top level block device
(e.g: /dev/mapper/*) when deciding whether to disable read/write
workqueues for nvme devices, it must look at the real disk under
dmcrypt/lvm, not the mapper. On osd block paths the top device
often lies about rotational, so --perf-no_workqueue was wrong.
The idea of this fix is to walk sysfs 'slaves/' to the leaf, then
check rotational there (udev + rota).
Collect FCM stats locally from NVMe drives (vendor log page 0xCA)
and expose them via node-proxy, the cephadm agent, and
`ceph orch hardware status --category fcm`.
Introduce a node backend that aggregates Redfish data with
node-local collectors, since FCM metrics are not available
from the BMC.
Afreen Misbah [Wed, 27 May 2026 00:07:38 +0000 (05:37 +0530)]
mgr/dashboard: fix nested shell quoting in cephadm e2e start-cluster
with_libvirt wraps commands in sg libvirt -c "$1", adding an extra
shell layer. Nested double quotes inside the outer double-quoted
string caused the argument to be split — with_libvirt received a
truncated $1, producing "Unterminated quoted string" on the remote
shell.
Drop the unnecessary inner double quotes around cephadm shell
arguments since cephadm shell accepts the command as separate args.
Use single quotes for the grep pattern inside the double-quoted
string so it survives the sg subshell.
Nizamudeen A [Thu, 2 Apr 2026 10:55:19 +0000 (16:25 +0530)]
mgr/dashboard: run kcli commands in libvritd group
Also https://github.com/ceph/ceph-build/pull/2562/ Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 8293aa38bd9c9abca6353ebbdb64401bf3263017)
ceph-volume: skip internal raid mirror LVs in inventory
ceph-volume inventory started including all LVM mapper devices after c06bee965f1. On hosts with raid mirrored system volumes, that pulls in
hidden legs like var_rmeta_0 which have no /dev/vg/lv node and makes
cephadm's ceph-volume inventory call fail.
Skip those internal LVs in get_devices() and avoid rewriting the device
path to a missing lv_path in Device._parse().
ceph-volume: fix inventory without /dev/vg/lv (slashed paths)
Ths makes ceph-volume use UdevData.preferred_block_path() in
get_devices() so it keeps /dev/vg/lv (slashed path form) when
it exists, else /dev/mapper/<name> (dashed path form).
This is needed for thin-pool LVs and environments where udev
does not create slashed paths.
mClockScheduler: adjust mClock profile parameters to prevent backfill starvation
Adjust the 'background_best_effort' queue parameters across the
three standard mClock profiles (high_client_ops, balanced, and
high_recovery_ops) to ensure best effort ops are not starved.
Previously, the 'background_best_effort' queue carried a default allocation
of 0% (MIN) reservation and a weight of 1 under these profiles. When
concurrent client traffic is dense, the zero-reservation for example completely
starves backfill sub-ops (MSG_OSD_EC_READ) on pools with
'allow_ec_optimizations' set to false. This starvation forces the Primary OSD
to hold internal BlueStore transactions and PG object locks for extended
windows, causing severe client median (50th) latency inflation.
To prevent background starvation and resolve the effects of the primary lock
retention, the profile configurations are tuned as follows:
The following profile changes forces low-cost sub-ops to clear out of peer
queues rapidly to drop primary locks, which helps improve the client
completion latency and tail latency (95th, 99th and 99.5th) percentile.
1. high_client_ops profile:
- Grant 'background_best_effort' a safe 5% minimum reservation.
- Scale the queue weight to 4.
2. balanced profile:
- Grant 'background_best_effort' a 5% minimum reservation.
- Set the queue weight to 2.
3. high_recovery_ops profile:
- Grant 'background_best_effort' a 5% minimum reservation.
- Set the queue weight to 2.
4. Modify the mClock config reference documentation to reflect the tuning
changes to the best-effort QoS parameters across the profiles.
Note on Proportional Scaling Compatibility:
Configuring these changes shifts total reservations to 105% (e.g., 50%
client + 50% recovery + 5% best-effort under the Balanced profile). Under
heavy concurrent saturation, mClock's internal controls resolves this
gracefully via proportional down-scaling, preserving the underlying
device bandwidth limits for different classes of clients. For example instead
of the client being allocated 50% bandwidth, a slightly lower reservation is
allocated while shifting the remaining bandwidth to the best-effort queue.
This minor scaling shift is virtually unnoticeable to the client application,
but it prevents the internal queue deadlocks.
Conflicts:
src/common/mclock_common.h
- src/common/mclock_common.h doesn't exist in tentacle.
mclock_common.h/cc was created as a refactor in later releases to share
common bits between classic and crimson OSDs. Introduced in PR:
https://github.com/ceph/ceph/pull/63516.
Therefore, the profile parameters change is added to the original
location in src/osd/mClockScheduler.cc.
src/messages, osd: Calculate and set cost for subOpReads for mClock scheduler
Previously, sub-op reads returned a hardcoded cost of 0, bypassing
mClock's background bandwidth and tag calculation mechanisms. This
allowed backfill operations to proceed un-metered, occasionally causing
backend resource contention and driving up client tail latencies.
Cost is calculated based on whether the complete chunk/shard or a subchunk
needs to be read. The possible cases are:
1. Read the complete chunk aligned length:
- Cost is set to the length of the chunk aligned extent size.
2. Fragmented reads:
- Consider the subchunk length and count to calculate the cost.
- compute_cost evaluates the exact layout of fragmented shard bytes on
disk by summing up the active subchunk allocations exactly once
(`fragmented_shard_bytes += k.second * subchunk_size`).
- Linear Extent Scaling: Scale the baseline footprint cleanly by
multiplying it against the true count of read extents (`tl.size()`),
achieving a highly efficient O(N) time complexity.
This linear cost model is compatible with pools running with
'allow_ec_optimizations' set to true. Under the FastEC optimized
pipeline, most operations are unified and bypass fragment slicing,
meaning requests will primarily match the Case 1 chunk-aligned path.
In Case 2 where applicable, the O(N) loop ensures that cost will
scale proportionally according to the layout.
It is important to note that the amount of data to read was set to an upper
bound defined by osd_recovery_max_chunk (8 MiB) and was rounded up to the
stripe width. The reason for setting a higher than actual upper bound is that
there may be cases where the object doesn't have the xattrs yet to determine
its size. Therefore, the amount to read was ultimatly set to ~(8 MiB / k)
where k is the number of data shards. This can cause mClock to prolong
the recovery times as items stay longer in the queue. To address this, the
amount to read is set to the remaining length of the object to recover
if the object size is known. Otherwise, the amount to read is set to the
recovery chunk size as before. Therefore, in some cases, only the first
recovery read could be costly if the object context is not known.
The MOSDECSubOpRead class introduces the following:
- cost member. This necessitates an increment to the HEAD_VERSION and
appropriate handling within the encode and decode methods.
- compute_cost() that is called when creating the message by
ECCommonL::ReadPipeline::do_read_op(). This calls into ECSubRead::cost()
that performs the actual calculations to set the cost based on the cases
mentioned above.
- The same sequence applies to the EC optimized path in
ECCommon::ReadPipeline::do_read_op().
Conflicts:
src/osd/ECMsgTypes.h
- Removed a couple of variable declarations related to recovery of OMAP
header and entries in EC pools which is yet to be backported to
tentacle. See commit: a5f4a4902075e343df154da61e3d205d2bd2a5d5.
osd/scheduler: Classify EC subOp reads according to op priority for mClock
The change brings MSG_OSD_EC_READ into the fold of mClock scheduler. This
improves the scheduling of client and other classes of operation as they
are no longer unnecessarily preempted by the 'immediate' queue.
EC SubOps are now handled as follows:
- EC SubOp reads generated during recovery will either go into the
'background_recovery' or 'background_best_effort' class based on
the recovery priority set for the op. EC SubOp reads generated due
to client will continue to be classified as 'immediate'.
- EC SubOp writes generated as a result of client operations will
continue to be classified as 'immediate'.
- EC SubOp replies are considered high priority and therefore
continue to be classed as 'immediate'.
Conflicts:
src/osd/scheduler/OpSchedulerItem.h
- enum SchedulerClass doesn't exist in tentacle.Therefore, the original
enum op_scheduler_class is used.
enum SchedulerClass was introduced when mclock_common.h/cc was created
as a refactor in later releases to share common bits between classic and
crimson OSDs. Introduced in PR: https://github.com/ceph/ceph/pull/63516.
common/async: async_cond::notify/cancel must post to handler's associated executor
RGWDeleteMultiObj spawns child coroutines via spawn_throttle to delete
objects in parallel. each child coroutine carries the connection strand
as its associated executor, serializing concurrent operations on shared
state like the response formatter and ops_log_entries.
but in multisite env, concurrent deletions in the same bucket shard
contend on async_cond in RGWDataChangesLog::add_entry(). when notify()
fires, waiting coroutines resume on the raw io_context executor
instead of their connection strand, breaking the serialization that prevents data
races in send_partial_response()
any_completion_handler doesn't support post(). so post() to the
default executor and dispatch to the associated executor from there
Casey Bodley [Tue, 2 Jun 2026 20:17:59 +0000 (16:17 -0400)]
osdc: deliver neorados completions to associated executor
while Objecter delivers librados completions directly by calling
Context::complete(), neorados completions are passed in as
boost::asio::any_completion_handler and delivered to an asio executor
via boost::asio::defer() on completion
asio handlers may have an "associated executor" so callers can customize
where these completions are delivered. for example, multithreaded
applications often use strand executors to synchronize completion
handlers and prevent data races between concurrent operations
however, applications like radosgw that depend on strands for
thread-safety did not get it due to the fact that Objecter's
Op::complete() delivered all neorados completions to the default
io_context executor
use boost::asio::get_associated_executor() to respect the handler's
executor affinity, if any. but because the Op's handler is the
type-erased any_completion_handler, its associated executor is also
type-erased as any_completion_executor. that any_completion_executor
doesn't support the blocking::never_t property required by defer/post,
so defer() was changed to dispatch() which may call the handler directly
if Objecter is already running on the requested executor. i assume this
is safe, given that librados' Context-based completions already do this
Kefu Chai [Sat, 13 Jun 2026 01:50:09 +0000 (09:50 +0800)]
python-common/cryptotools: stop using the removed X509Req API
pyOpenSSL deprecated OpenSSL.crypto.X509Req in 24.2.0 (2024-07-20) and
removed it in 26.3.0 (2026-06-12). as we don't pin pyopenssl, CI picked
up the new release, and create_self_signed_cert() started failing with:
AttributeError: module 'OpenSSL.crypto' has no attribute 'X509Req'
this took down run-tox-mgr, run-tox-mgr-dashboard-py3 and the mypy check.
we only used X509Req to build a subject name and then copied it into the
X509 cert. so drop it, and set the subject on the cert directly. the
resulting cert stays the same: subject from dname, issuer set to the same
subject, self-signed.
Kefu Chai [Tue, 23 Jun 2026 07:43:28 +0000 (15:43 +0800)]
mgr/dashboard: skip the table when an nvmeof cli result has no columns
The dashboard leaves prettytable unpinned. prettytable commit 2574492 ("Apply
some Pylint rules (PLR)", #436) rewrote _stringify_row()'s row_height as
`max(_get_size(c)[1] for c in row)`, which raises ValueError("max() iterable
argument is empty") on a row with no cells. The change is undocumented and
shipped in 3.18.0; get_string() trips on it when a table has a row but no
columns.
AnnotatedDataTextOutputFormatter builds such a table for an empty result, or
one whose only field is status or error_message, so NvmeofCLICommand.call()
returns -EINVAL and the command fails. This broke run-tox-mgr-dashboard-py3
once the tox virtualenv picked up prettytable 3.18.0.
Return an empty string when there are no columns instead of formatting a
degenerate table.
Conflicts:
src/pybind/mgr/dashboard/tests/test_nvmeof_cli.py
Resolution: tentacle has only the empty-result test case; updated its
expected stdout to ''
mgr/DaemonServer: Aggregate and globally sort OSDs for ok-to-upgrade
The 'ok-to-upgrade' command output sorting did not scale accurately
when target CRUSH buckets contained multiple child buckets (e.g., a
chassis containing multiple hosts). OSDs were previously sorted
individually per child bucket and appended sequentially. This created
fragmented, per-host sort segments rather than a globally sorted list
for the parent bucket.
Changes:
1. Fix the issue above by aggregating all child OSDs into a single vector prior
to executing a single, global sort operation based on PG counts. Additionally,
optimize memory efficiency and future-proof the logic by reserving continuous
vector blocks to avoid dynamic heap reallocations.
2. Add integration tests with chassis and rack based CRUSH hierarchies which
verifies the ok-to-upgrade functionality. In addition, the tests crucially
verify the order of OSDs returned is according to the ascending order of
acting PG count. Additionally, make minor fix-ups to lines that determine the
length of a list in JSON response by removing the redundant "| bc".
Alex Ainscow [Wed, 8 Apr 2026 10:49:58 +0000 (11:49 +0100)]
osd: Allow multiple objects with same version in missing list.
Most of the time, a single version in a PG can only correspond to a single object.
However, following a PG merge it is possible, even likely, that two objects will
have the same version. The PG Log works around this by discarding the log.
However, during backfill, it is possible for the missing list to be build with
these duplicate versions.
A recently added assert detected that this scenario was corrupting the reverse
missing list (rmissing). This behaviour has always existed, but was previously
unnoticed. It could cause some bugs and potentially loop-asserts on OSDs,
although mostly would not be noticed.
Here we fix this properly, by converting rmissing to a multimap. This is wrapped
in some insert functions, which assert that the rmissing list does not end up
with duplicate entries. The code is optimised for the case where there are no
duplicate versions.
Additionally, some of the old asserts have been rolled into the insert functions.
Fixes: https://tracker.ceph.com/issues/75778 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
(cherry picked from commit f3940400952b444a31f59b633fa3fa35437c87a9)
projects / ceph.git / log