* refs/pull/67321/head:
qa: set column for insertion
qa: bail sqlite3 on any error
qa: use actual sqlite3 blob instead of string
test: use json_extract instead of awkward json_tree
Patrick Donnelly [Thu, 30 Apr 2026 21:01:45 +0000 (14:01 -0700)]
Merge PR #68371 into tentacle
* refs/pull/68371/head:
qa/tasks/pykmip: archive pykmip log after server down
qa/tasks/pykmip: use OpenSSL names instead IANA
qa/tasks/pykmip: drop py2 deps
Revert "qa/rgw/crypt: disable failing kmip testing"
Patrick Donnelly [Thu, 30 Apr 2026 21:00:45 +0000 (14:00 -0700)]
Merge PR #66358 into tentacle
* refs/pull/66358/head:
rgw/auth: a forwarded CreateBucket request in case of multisite has an empty
rgw/s3: Always include x-amz-content-sha256 header in AWS v4 signatures for S3 compatibility
Patrick Donnelly [Thu, 30 Apr 2026 15:52:36 +0000 (08:52 -0700)]
Merge PR #68512 into tentacle
* refs/pull/68512/head:
mgr/dashboard: sync policy created for a bucket in Object >> Multi-site >> Sync-policy, is not reflecting under bucket's replication
Patrick Donnelly [Thu, 30 Apr 2026 15:51:59 +0000 (08:51 -0700)]
Merge PR #67840 into tentacle
* refs/pull/67840/head:
mgr/dashboard: Fix make check failures
mgr/dashboard: Round off y-axis value of area chart
mgr/dashboard: Fix padding of overview page
mgr/dashboard: Add capacity thresholds
mgr/dashboard: Fix loading states in storage overview card
mgr/dashboard: Add tootltip to storage overview
mgr/dashboard: Fixing message when prometheus is disabled in performance charts
mgr/dashboard: show miscellaneous data used
mgr/dashboard: fix consumption chart units
mgr/dashboard: rename expand-cluster to add-storage
mgr/dashboard: update onboarding screen as per design
mgr/dashboard: Fix scrubbing state
mgr/dashboard: Fix snapshot Api firing twice
mgr/dashboard: Add data resileincy panel
mgr/dashboard: Add data resileincy card
mgr/dashboard: add storage consumption card
mgr/dashboard : update telemetry notification for simple mode
mgr/dashboard:revamp on-borading screen
mgr/dashboard: Generic Performace Chart - Carbon
mgr/dashboard: Add filtering of alerts via route
mgr/dashboard: Add skeleton states for alerts card
mgr/dashboard: Fix css in alerts card
mgr/dashboard: Fix breaking layout in overview page
mgr/dashboard: Add hardware tab to health card
mgr/dashboard: Added variations of alerts card sub total layout
mgr/dashboard: Css fixes for health card and alerts card
fix for quorum in API
mgr/dashboard: Add systems tab to health card
mgr/dashboard: Add alerts card
mgr/dashboard: Add health check panel
mgr/dashboard: Add health card
mgr/dashboard: side-panel enhancements
mgr/dashboard: introduce side panel as a reusable component
mgr/dashboard: Removed Raw capacity toggle
mgr/dashboard: Added unit tests
Added qurey data
mgr/dashboard: Added tool definition tip
Added query tital and used capacity data
mgr/dashboard: Add storage card to overview page
Patrick Donnelly [Tue, 28 Apr 2026 19:26:46 +0000 (12:26 -0700)]
Merge PR #68357 into tentacle
* refs/pull/68357/head:
qa/suites/upgrade/telemetry-upgrade: ignore expected health warning
qa/suites/orch/cephadm: replace "reef" with "v18.2.8"
qa/suites/fs/upgrade/mds_upgrade_sequence: replace "reef" with "v18.2.8"
qa/suites/upgrade: use tagged versions of reef
rgw: read_obj_policy() consults s3:prefix when deciding between 403/404
when read_obj_policy() gets ENOENT, it only returns 404 NoSuchKey if the
requester has s3:ListBucket permission. however, policy that allows
s3:ListBucket may be conditional on the s3:prefix to restrict listings
to certain paths/object names. add the requested object name to the iam
environment as s3:prefix to match aws behavior here
Patrick Donnelly [Mon, 27 Apr 2026 20:46:49 +0000 (16:46 -0400)]
Merge PR #68148 into tentacle
* refs/pull/68148/head:
qa: ignore NVMEOF_GATEWAY_DOWN in nvmeof_scalability.yaml
qa/tasks/nvmeof.py: retry do_check if gw in CREATED
qa/tasks/nvmeof.py: Fix tharsher daemon_rm revival
Patrick Donnelly [Mon, 27 Apr 2026 19:28:45 +0000 (15:28 -0400)]
Merge PR #68038 into tentacle
* refs/pull/68038/head:
librbd: store CRC32C with initial value -1 to match msgr2 validation
librbd: add rbd_aio_write_with_crc32c API for precomputed checksums
Patrick Donnelly [Wed, 22 Apr 2026 17:22:51 +0000 (13:22 -0400)]
Merge PR #67418 into tentacle
* refs/pull/67418/head:
mgr/cephadm: validate hostname in NodeProxyCache
node-proxy: improve HTTP error logging in client
node-proxy: get serial number instead of SKU
node-proxy: allow multiple sources per component
node-proxy: re-auth and retry once on 401
node-proxy: fix flake8 E721 in _dict_diff
node-proxy: make the update loop interval configurable
mgr/node-proxy: fix "ceph orch hardware status --category criticals"
node-proxy: normalize storage data per member
node-proxy: encapsulate send logic in dedicated method
node-proxy: log actual data delta in reporter
node-proxy: add periodic heartbeats in main and reporter loops
node-proxy: adjust log levels
node-proxy: add unit tests
node-proxy: add tox config for mypy, flake8, isort, black
node-proxy: black and isort formatting pass
node-proxy: fix mypy errors
node-proxy: handle nested Redfish paths for components
node-proxy: split out config, bootstrap and redfish logic
node-proxy: refactor config loading
node-proxy: add 'vendor based' redfish system selection
node-proxy: introduce component spec registry and overrides for updates
mgr/cephadm: safe status/health access in node-proxy agent and inventory
node-proxy: narrow build_data exception handling and re-raise
node-proxy: refactor Endpoint/EndpointMgr and fix chassis paths
node-proxy: use safe field access in storage update
node-proxy: reduce log verbosity for missing optional fields
Patrick Donnelly [Wed, 22 Apr 2026 17:21:13 +0000 (13:21 -0400)]
Merge PR #67343 into tentacle
* refs/pull/67343/head:
ceph-volume: fix test_reject_readonly_device unit test
ceph-volume: single lvs call to speed up exclude_lvm_osd_devices
ceph-volume: avoid Device() instantiation in lvm OSD filtering
ceph-volume: avoid RuntimeError on ceph-volume raw list with non-existent loop devices
Patrick Donnelly [Wed, 22 Apr 2026 14:38:38 +0000 (10:38 -0400)]
Merge PR #67850 into tentacle
* refs/pull/67850/head:
mgr, qa: clarify module checks in DaemonServer
mgr, qa: add `pending_modules` to asock command
mgr, common, qa, doc: issue health error after max expiration is exceeded
mgr: ensure that all modules have started before advertising active mgr
mgr/dashboard: Round off y-axis value of area chart
- by default y-axos set to 1 for all
- the value round off for area chart is seperated from y-axis ticks
- also fixes a bug where all IOPS y-ticks being repeated 1,1,0,0
Laura Flores [Fri, 12 Sep 2025 20:14:30 +0000 (20:14 +0000)]
mgr, qa: clarify module checks in DaemonServer
The current check groups modules not being
enabled with failing to initialize. In this commit,
we reorder the checks:
1: Screen for a module being enabled. If it's not,
issue an EOPNOTSUPP with instructions on how
to enable it.
2. Screen for if a module is active. If a module
is enabled, then the cluster expects it to
be active to support commands. If the module
took too long to initialize though, we will
catch this and issue an ETIMEDOUT error with
a link for troubleshooting.
Now, these two separate issues are not grouped
together, and they are checked in the right order.
Laura Flores [Thu, 11 Sep 2025 22:13:51 +0000 (22:13 +0000)]
mgr, qa: add `pending_modules` to asock command
Now, the command `ceph tell mgr mgr_status` will show a
"pending_modules" field. This is another way for Ceph operators
to check which modules haven't been initalized yet (in addition
to the health error).
This command was also added to testing scenarios in the workunit.
Laura Flores [Tue, 29 Jul 2025 22:46:46 +0000 (22:46 +0000)]
mgr, common, qa, doc: issue health error after max expiration is exceeded
----------------- Enhancement to the Original Fix -----------------
During a mgr failover, the active mgr is marked available if:
1. The mon has chosen a standby to be active
2. The chosen active mgr has all of its modules initialized
Now that we've improved the criteria for sending the "active" beacon
by enforcing it to retry initializing mgr modules, we need to account
for extreme cases in which the modules are stuck loading for a very long
time, or even indefinitely. In these extreme cases where the modules might
never initialize, we don't want to delay sending the "active" beacon for
too long. This can result in blocking other important mgr functionality,
such as reporting PG availability in the health status. We want
to avoid sending warnings about PGs being unknown in the health status when
that's not ultimately the problem.
To account for an exeptionally long module loading time, I added a new
configurable `mgr_module_load_expiration`. If we exceed this maximum amount
of time (in ms) allotted for the active mgr to load the mgr modules before declaring
availability, the standby will then proceed to mark itself "available" and
send the "active" beacon to the mon and unblock other critical mgr functionality.
If this happens, a health error will be issued at this time, indicating
which mgr modules got stuck initializing (See src/mgr/PyModuleRegistry.cc). The
idea is to unblock the rest of the mgr's critical functionality while making it
clear to Ceph operators that some modules are unusable.
The workunit was rewritten so it tests for these scenarios:
1. Normal module loading behavior (no health error should be issued)
2. Acceptable delay in module loading behavior (no health error should be
issued)
3. Unacceptable delay in module loading behavior (a health error should be
issued)
This documentation explains the "Module failed to initialize"
cluster error.
Users are advised to try failing over
the mgr to reboot the module initialization process,
then if the error persists, file a bug report. I decided
to write it this way instead of providing more complex
debugging tips such as advising to disable some mgr modules
since every case will be different depending on which modules
failed to initialize.
In the bug report, developers can ask for the health detail
output to narrow down which module is causing a bottleneck,
and then ask the user to try disabling certain modules until
the mgr is able to fully initialize.
Laura Flores [Fri, 25 Apr 2025 22:11:19 +0000 (22:11 +0000)]
mgr: ensure that all modules have started before advertising active mgr
----------------- Explanation of Problem ----------------
When the mgr is restarted or failed over via `ceph mgr fail` or during an
upgrade, mgr modules sometimes take longer to start up (this includes
loading their class, commands, and module options, and being removed
from the `pending_modules` map structure). This startup delay can happen
due to a cluster's specific hardware or if a code bottleneck is triggered in
a module’s `serve()` function (each mgr module has a `serve()` function that
performs initialization tasks right when the module is loaded).
When this startup delay occurs, any mgr module command issued against the
cluster around the same time fails with error saying that the command is not
supported:
```
$ ceph mgr fail; ceph fs volume ls
Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
Module 'volumes' is not enabled/loaded (required by command 'fs volume ls'): use `ceph mgr module enable volumes` to enable it
```
We should try to lighten any bottlenecks in the mgr module `serve()`
functions wherever possible, but the root cause of this failure is that the
mgr sends a beacon to the mon too early, indicating that it is active before
the module loading has completed. Specifically, some of the mgr modules
have loaded their class but have not yet been deleted from the `pending_modules`
structure, indicating that they have not finished starting up.
--------------------- Explanation of Fix --------------------
This commit improves the criteria for sending the “active” beacon to the mon so
the mgr does not signal that it’s active too early. We do this through the following additions:
1. A new context `ActivePyModules::recheck_modules_start` that will be set if not all modules
have finished startup.
2. A new function `ActivePyModules::check_all_modules_started()` that checks if modules are
still pending startup; if all have started up (`pending_modules` is empty), then we send
the beacon right away. But if some are still pending, we pass the beacon task on to the new
recheck context `ActivePyModules::recheck_modules_start` so we know to send the beacon later.
3. Logic in ActivePyModules::start_one() that only gets triggered if the modules did not all finish
startup the first time we checked. We know this is the case if the new recheck context
`recheck_modules_start` was set from `nullptr`. The beacon is only sent once `pending_modules` is
confirmed to be empty, which means that all the modules have started up and are ready to support commands.
4. Adjustment of when the booleans `initializing` and `initialized` are set. These booleans come into play in
MgrStandby::send_beacon() when we check that the active mgr has been initialized (thus, it is available).
We only send the beacon when this boolean is set. Currently, we set these booleans at the end of Mgr::init(),
which means that it gets set early before `pending_modules` is clear. With this adjustment, the bools are set
only after we check that all modules have started up. The send_beacon code is triggered on mgr failover AND on
every Mgr::tick(), which occurs by default every two seconds. If we don’t adjust when these bools are set, we
only fix the mgr failover part, but the mgr still sends the beacon too early via Mgr::tick(). Below is the relevant
code from MgrStandby::send_beacon(), which is triggered in Mgr::background_init() AND in Mgr::tick():
```
// Whether I think I am available (request MgrMonitor to set me
// as available in the map)
bool available = active_mgr != nullptr && active_mgr->is_initialized();
auto addrs = available ? active_mgr->get_server_addrs() : entity_addrvec_t();
dout(10) << "sending beacon as gid " << monc.get_global_id() << dendl;
```
--------------------- Reproducing the Bug ----------------------
At face value, this issue is indeterministically reproducible since it
can depend on environmental factors or specific cluster workloads.
However, I was able to deterministically reproduce it by injecting a
bottleneck into the balancer module:
```
diff --git a/src/pybind/mgr/balancer/module.py b/src/pybind/mgr/balancer/module.py
index d12d69f..91c83fa8023 100644
--- a/src/pybind/mgr/balancer/module.py
+++ b/src/pybind/mgr/balancer/module.py
@@ -772,10 +772,10 @@ class Module(MgrModule):
self.update_pg_upmap_activity(plan) # update pg activity in `balancer status detail`
self.optimizing = False
+ # causing a bottleneck
+ for i in range(0, 1000):
+ for j in range (0, 1000):
+ x = i + j
+ self.log.debug("hitting the bottleneck in the balancer module")
self.log.debug('Sleeping for %d', sleep_interval)
self.event.wait(sleep_interval)
self.event.clear()
```
Then, the error reproduces every time by running:
```
$ ./bin/ceph mgr fail; ./bin/ceph telemetry show
Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
Module 'telemetry' is not enabled/loaded (required by command 'telemetry show'): use `ceph mgr module enable telemetry` to enable it
```
With this fix, the active mgr is marked as "initialized" only after all
the modules have started up, and this error goes away. The command may
take a bit longer to execute depending on the extent of the delay.
This commit adds a dev-only config that can inject a longer
loading time into the mgr module loading sequence so we can
simulate this scenario in a test.
The config is 0 ms by default since we do not add any delay
outside of testing scenarios. The config can be adjusted
with the following command:
`ceph config set mgr mgr_module_load_delay <ms>`
A second dev-only config also allows you to specify which
module you want to be delayed in loading time. You may change
this with the following command:
`ceph config set mgr mgr_module_load_delay_name <module name>`
The workunit added here tests a simulated slow loading module
scenario to ensure that this case is properly handled.
Oguzhan Ozmen [Thu, 19 Feb 2026 01:42:54 +0000 (01:42 +0000)]
rgw/beast: use strand executor for timeout timer to prevent concurrent socket access
The timeout_timer in AsioFrontend::on_accept() was constructed with
context.get_executor(), which dispatches the timeout handler to the
raw io_context. Since each connection's coroutine runs on a strand
(via make_strand(context)), this allowed the timeout handler to
execute concurrently with the coroutine on a different thread.
When the timeout fires, timeout_handler calls socket.cancel() and
socket.shutdown() racing with the coroutine's async_read_header(),
async_write(), or async_read_some() on the same socket and buffer.
This can corrupt the flat_static_buffer's internal pointers, leading
to SIGSEGV in the Beast HTTP parser.
Fix by constructing the timeout_timer with yield.get_executor(),
which returns the coroutine's strand executor. This ensures the
timeout handler is serialized with the coroutine, preventing
concurrent access to the socket and buffer.
Abhishek Desai [Tue, 17 Mar 2026 10:18:46 +0000 (15:48 +0530)]
mgr/dashboard : Restrict create storage class with existing name
fixes : Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>
(cherry picked from commit f144cf27ffb82c8386a606eed0019298364ad8be)
Ilya Dryomov [Thu, 15 Jan 2026 12:56:13 +0000 (13:56 +0100)]
librbd: avoid losing sparseness in read_parent()
When read_parent() constructs a read for image_ctx->parent, it employs
a thick bufferlist (either re-using the bufferlist on the object extent
or creating a temporary one inside of C_ObjectReadMergedExtents). This
forgoes any sparseness: even if the result obtained by ObjectRequest is
sparse, it's thickened by ReadResult's handler for Bufferlist type.
This behavior is very old and hasn't been a problem for regular clones
because the public API returns a thick bufferlist in the case of C++ or
equivalent char* buf/struct iovec iov[] buffers in the case of C anyway.
ObjectCacher isn't sparse-aware but it's also not used for caching reads
by default and reading from parent for the purposes of a copyup is done
in CopyupRequest in a way that preserves sparseness. However, when it
comes to migration, source image reads go through read_parent() and the
destination image gets thickened as an inadvertent side effect.
Fix this by introducing a new ChildObject type for ReadResult whose
handler would plant the result obtained by parent's ObjectRequest into
child's ObjectRequest, as if read_parent() wasn't even called.
Patrick Donnelly [Fri, 17 Apr 2026 21:47:14 +0000 (17:47 -0400)]
Merge PR #67511 into tentacle
* refs/pull/67511/head:
doc/_ext: fix ceph_commands.py for new decorator-based command system
pybind/mgr: add per-module CLICommand instances to remaining modules
pybind/mgr/dashboard: create DBCLICommand, use throughout
pybind/mgr/tests/test_object_format: update DecoDemo to use fresh CLICommand
pybind/mgr/smb: adapt SMBCommand to use CLICommandBase
pybind/orchestrator,cephadm: replace CLICommandMeta
pybind/mgr: mechanically fix simple users to not import CLI*Command
pybind/mgr/mgr_module: support per-module CLICommand instances and globals
pybind/.../dashboard: misc automatic linter fixes
Shilpa Jagannath [Tue, 27 Jan 2026 06:53:47 +0000 (01:53 -0500)]
rgw/auth: a forwarded CreateBucket request in case of multisite has an empty
payload hash computed for UNSIGNED-PAYLOAD. use that hash for the header
'x-amz-content-sha256' in AWSSignerV4::prepare()
Kefu Chai [Sun, 8 Feb 2026 12:34:15 +0000 (20:34 +0800)]
doc/_ext: fix ceph_commands.py for new decorator-based command system
After commit 4aa9e246f, mgr modules migrated from using a class-level
COMMANDS list to decorator-based command registration using per-module
CLICommand instances (e.g., @BalancerCLICommand.Read('balancer status')).
This broke the ceph_commands.py Sphinx extension which was hardcoded to
expect m.COMMANDS to be a list, causing documentation builds to fail.
But not all modules are using this per-module CLICommand. Some modules are
fully migrated (balancer, hello, etc.) and use decorators, while others
are partially migrated (volumes, progress, stats, influx, k8sevents,
osd_perf_query, osd_support) - they have CLICommand defined but still
use the old COMMANDS list.
This fix updates _collect_module_commands() to handle three scenarios:
1. Fully migrated modules: Check CLICommand.dump_cmd_list() and use it
if it returns commands
2. Partially migrated modules: Fall back to the old COMMANDS list if
dump_cmd_list() returns empty
3. Legacy modules: Use COMMANDS list if CLICommand doesn't exist
This ensures the Sphinx extension works with modules in any migration
state, maintaining backwards compatibility while supporting the new
decorator pattern.
Conflicts:
- src/pybind/mgr/prometheus/module.py
CLIReadCommand/CLIWriteCommand removed from imports, not yet in tentacle - kept tentacle version
- src/pybind/mgr/orchestrator/module.py
_cert_store_key_ls, _cert_store_cert_ls - signature changed in main, kept tentacle version
_cert_store_entity_ls - function renamed in main, kept tentacle name
_cert_store_entity_ls - function name changed in main but not in tentacle
- src/pybind/mgr/smb/cli.py
typing imports differ between main and tentacle - manually reconciled
error_wrapper not backported to tentacle - removed from cherry-pick
- src/pybind/mgr/mirroring/module.py
import differs between main and tentacle - manually kept tentacle imports
- src/pybind/mgr/pg_autoscaler/module.py
get_scaling_threshold not backported to tentacle - removed that change
qa/valgrind: generalize suppressions for gcc-14 MismatchedFree
the rgw suite is showing a handful of these false positives, and i
really don't want to micromanage suppressions for every little thing
that shows up here
so i chose to generalize the existing suppressions from
https://tracker.ceph.com/issues/74604 so that a single suppression would
cover this entire class of MismatchedFree warnings
projects / ceph.git / log