ceph-volume: skip internal raid mirror LVs in inventory
ceph-volume inventory started including all LVM mapper devices after c06bee965f1. On hosts with raid mirrored system volumes, that pulls in
hidden legs like var_rmeta_0 which have no /dev/vg/lv node and makes
cephadm's ceph-volume inventory call fail.
Skip those internal LVs in get_devices() and avoid rewriting the device
path to a missing lv_path in Device._parse().
ceph-volume: fix inventory without /dev/vg/lv (slashed paths)
Ths makes ceph-volume use UdevData.preferred_block_path() in
get_devices() so it keeps /dev/vg/lv (slashed path form) when
it exists, else /dev/mapper/<name> (dashed path form).
This is needed for thin-pool LVs and environments where udev
does not create slashed paths.
Kefu Chai [Sat, 13 Jun 2026 01:50:09 +0000 (09:50 +0800)]
python-common/cryptotools: stop using the removed X509Req API
pyOpenSSL deprecated OpenSSL.crypto.X509Req in 24.2.0 (2024-07-20) and
removed it in 26.3.0 (2026-06-12). as we don't pin pyopenssl, CI picked
up the new release, and create_self_signed_cert() started failing with:
AttributeError: module 'OpenSSL.crypto' has no attribute 'X509Req'
this took down run-tox-mgr, run-tox-mgr-dashboard-py3 and the mypy check.
we only used X509Req to build a subject name and then copied it into the
X509 cert. so drop it, and set the subject on the cert directly. the
resulting cert stays the same: subject from dname, issuer set to the same
subject, self-signed.
Kefu Chai [Tue, 23 Jun 2026 07:43:28 +0000 (15:43 +0800)]
mgr/dashboard: skip the table when an nvmeof cli result has no columns
The dashboard leaves prettytable unpinned. prettytable commit 2574492 ("Apply
some Pylint rules (PLR)", #436) rewrote _stringify_row()'s row_height as
`max(_get_size(c)[1] for c in row)`, which raises ValueError("max() iterable
argument is empty") on a row with no cells. The change is undocumented and
shipped in 3.18.0; get_string() trips on it when a table has a row but no
columns.
AnnotatedDataTextOutputFormatter builds such a table for an empty result, or
one whose only field is status or error_message, so NvmeofCLICommand.call()
returns -EINVAL and the command fails. This broke run-tox-mgr-dashboard-py3
once the tox virtualenv picked up prettytable 3.18.0.
Return an empty string when there are no columns instead of formatting a
degenerate table.
Conflicts:
src/pybind/mgr/dashboard/tests/test_nvmeof_cli.py
Resolution: tentacle has only the empty-result test case; updated its
expected stdout to ''
mgr/DaemonServer: Aggregate and globally sort OSDs for ok-to-upgrade
The 'ok-to-upgrade' command output sorting did not scale accurately
when target CRUSH buckets contained multiple child buckets (e.g., a
chassis containing multiple hosts). OSDs were previously sorted
individually per child bucket and appended sequentially. This created
fragmented, per-host sort segments rather than a globally sorted list
for the parent bucket.
Changes:
1. Fix the issue above by aggregating all child OSDs into a single vector prior
to executing a single, global sort operation based on PG counts. Additionally,
optimize memory efficiency and future-proof the logic by reserving continuous
vector blocks to avoid dynamic heap reallocations.
2. Add integration tests with chassis and rack based CRUSH hierarchies which
verifies the ok-to-upgrade functionality. In addition, the tests crucially
verify the order of OSDs returned is according to the ascending order of
acting PG count. Additionally, make minor fix-ups to lines that determine the
length of a list in JSON response by removing the redundant "| bc".
Alex Ainscow [Wed, 8 Apr 2026 10:49:58 +0000 (11:49 +0100)]
osd: Allow multiple objects with same version in missing list.
Most of the time, a single version in a PG can only correspond to a single object.
However, following a PG merge it is possible, even likely, that two objects will
have the same version. The PG Log works around this by discarding the log.
However, during backfill, it is possible for the missing list to be build with
these duplicate versions.
A recently added assert detected that this scenario was corrupting the reverse
missing list (rmissing). This behaviour has always existed, but was previously
unnoticed. It could cause some bugs and potentially loop-asserts on OSDs,
although mostly would not be noticed.
Here we fix this properly, by converting rmissing to a multimap. This is wrapped
in some insert functions, which assert that the rmissing list does not end up
with duplicate entries. The code is optimised for the case where there are no
duplicate versions.
Additionally, some of the old asserts have been rolled into the insert functions.
Fixes: https://tracker.ceph.com/issues/75778 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
(cherry picked from commit f3940400952b444a31f59b633fa3fa35437c87a9)
mgr: guard close_section calls in get_perf_schema_python
When a daemon exists in the daemon state map but has no perf counters
(e.g. an OSD that was running but is now down or destroyed), the loop
in get_perf_schema_python never executes, leaving prev_key_name empty
and no formatter sections opened. The unconditional close_section()
calls after the loop then trigger an assertion failure in PyFormatter
(cursor != root).
Add the same `if (!prev_key_name.empty())` guard that already protects
the close_section() calls inside the loop, so we only close sections
that were actually opened.
Nitzan Mordechai [Sun, 10 May 2026 06:51:08 +0000 (06:51 +0000)]
qa: ignore evicted client warnings for singletone bluestore
After adding mds client into singletone bluestore and increas debug
levels and bluestore_allocator=stupid client is slow enough that MDS
evicts it after ~302 seconds
mgr/DaemonServer: clarify ok-to-upgrade error message for CRUSH buckets
Refine the error string in DaemonServer.cc returned by the
ok-to-upgrade command when OSDs in a CRUSH bucket cannot be upgraded.
The original message is ambiguous. It fails to clearly convey that
stopping *any* individual OSD in that specific bucket will drop PGs
offline, meaning no OSDs within that bucket can be safely upgraded at
this time.
Update the phrasing to explicitly state that at least X PGs will go offline
if any OSD out of the total count in that CRUSH bucket is stopped. Also
standardize on capitalized acronyms (PG, OSD, CRUSH) and wrap the bucket
name in single quotes for better log readability.
ceph-volume: has_bluestore_label checks all bluestore label replica offsets
BlueStore replicates the block device label at fixed offsets (0 and
multiples of 1Gb up to 1000gb). has_bluestore_label() only read the
first 22 bytes, so disks with a wiped primary label but intact
replicas are missed.
with this commit, has_bluestore_label() scans each known offset with
seek/read and compares the ASCII prefix as bytes.
The thrashosds task is occasionally restarting OSDs and mon/mgr log
warnings are wrongly flagging this as a problem.
Fixes: https://tracker.ceph.com/issues/76747 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit a1430973f2c3e8758890447a1de3b906da77b60e)
Conflicts:
qa/suites/upgrade/reef-x/stress-split/1-start.yaml
qa/suites/upgrade/squid-x/stress-split/1-start.yaml
Resolution: different upgrade suites, just add the change manually
libcephsqlite: ensure atexit handlers are registered after openssl
When the sqlite3 executable encounters an error with .bail=on, it will
make a call to exit(). The atexit() handlers will execute in LIFO order.
We need to ensure that openssl (before OpenSSL 4.0 [1]) atexit handlers are
registered before libcephsqlite.
[1] http://github.com/openssl/openssl/commit/31659fe32673a6bd66abf3f8a7d803e81c6ffeed (OpenSSL 4.0 no longer arms `OPENSSL_cleanup()` function as an `atexit(3)`)
Fixes: https://tracker.ceph.com/issues/59335 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 7949cd5f12eb7cc0dc85fd1b5c1d795fad1df922)
Redouane Kachach [Fri, 29 May 2026 09:09:44 +0000 (11:09 +0200)]
qa/tasks: capture CommandCrashedError when running nvme list cmd
The safe_while retry loop does not catch exceptions, so a
CommandCrashedError from `nvme list` bypasses it entirely. Catch
CommandCrashedError and continue the retry loop instead.
ceph-volume: retry lvs after empty result and "devices file is missing" stderr
When LVM's devices file is out of sync with the runtime device view (common
in teuthology/container namespaces with multipath), `lvs` can exit 0 with
empty stdout and only stderr warnings about missing mapper entries.
It can leave get_lvs() empty and cause Device() to fall through to lsblk on a
vg/lv path which can produce a misleading "not a block device" error.
With this fix, ceph-volume retries once with 'use_devicesfile=0' when it
detects this specific pattern.
Bill Scales [Fri, 15 May 2026 14:39:25 +0000 (15:39 +0100)]
osd: Fix bug when calculating min_peer_features
PeeringState calculates the minimum set of features for the set
of OSDs within a PG. There is a bug when the peer info has
already been cached where these peers features are not included
in the calculation. This can lead to the min feature set
including features that not all OSDs have.
Previously this just made some asserts less aggressive than they
should have been. Pull request https://github.com/ceph/ceph/pull/57740
uses min_peer_features to decide how to encode messages to other OSDs.
Midway through an upgrade this bug can cause an OSD to send
the wrong version of a message to a downlevel OSD causing
it to abort.
Fixes: https://tracker.ceph.com/issues/76600 Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
(cherry picked from commit ce5882778db9b92b14a6bb5eca22e3cecf2be9dc)
Casey Bodley [Wed, 25 Mar 2026 16:33:40 +0000 (12:33 -0400)]
qa/rgw: remove ragweed from verify subsuite
it's currently broken with newer python on rocky 10 and ubuntu 24
(tracked in https://tracker.ceph.com/issues/72500) and doesn't provide
interesting test coverage outside of rgw/upgrade
ceph-volume: OSD mapper lifecycle (LVM + raw) for activate
This adds small helpers so activate can consistently bring the OSD device
stack online (LVM lvchange, optional mapper open) and tear it down again,
with refresh in between. Same idea for the raw path. Crypto is handled
inside that flow when the OSD is encrypted.
ceph-volume: raw activate should ignore lvm backed OSD devices
the generic activate (`ceph-volume activate`) runs the
raw path before LVM. Raw.activate was walking lsblk / raw
list entries and could hit block devices that are actually
logical volumes from `ceph-volume lvm prepare` or `lvm batch`
(with ceph lvm tags on the lv).
That made raw activation poke at LVM backed OSDs instead of
leaving it to `lvm activate`.
with this commit ceph-volume now builds the set of LV paths
that carry those tags once (`lvs` via ceph_volume_lvm_prepare_lv_paths)
and skip any candidate path that matches, so only real raw
OSDs go through the 'raw activate path'.
Also, we now pass `with_tpm` through luks_open() calls for db and
wal so encrypted metadata uses the same systemd-cryptsetup path
as the block LV when ceph.with_tpm is set.
tentacle: tentacle-p2p add centos9 to stress-split
Adds CentOS 9 Stream as a supported host OS for the tentacle-p2p
stress-split suite.
Rocky10 is not added here. This suite tests a bare-metal p2p upgrade
between Tentacle point releases. Rocky10 package support was added during
that release cycle, meaning there is no valid "FROM" package baseline to
install on Rocky10, making a bare-metal point-to-point upgrade path on Rocky10
impossible to test.
tentacle: upgrade/tentacle-p2p install pytest for rbd-python tests
The rbd-python workload runs test_librbd_python.sh, which invokes
'python3 -m pytest' to run the librbd Python API tests.
On a bare-metal install, python3-pytest is not pulled in by the
Ceph packages and is not present and the workunit fails.
Add it via extra_system_packages so the tests can run.
tentacle: suites/upgrade add centos to centos image upgrade
Previously, each suite had a single upgrade-sequence.yaml that targeted
only one image.
This commit splits each upgrade sequence into two variants teuthology
picks one per run
This applies to:
- reef-x/parallel
- reef-x/stress-split
- squid-x/parallel
- squid-x/stress-split
- telemetry/reef-x
- telemetry/squid-x
For the stress-split suites, the upgrade logic is split into a
first-half-sequence run concurrently with thrashosds,
and a second-half-sequence run after.
Both sequences contain the hardcoded target image, so each variant
needs its own copy. They were previously inlined in 1-start.yaml.
This commit extracts them into the upgrade-sequence$/ files so
each variant can target the right image.
Patrick Donnelly [Tue, 19 May 2026 23:15:29 +0000 (19:15 -0400)]
Merge PR #68803 into tentacle
* refs/pull/68803/head:
rgw/multisite: concurrency adjustment - consider the case caller provides 1
rgw/multisite: log concurrency state transitions in adj_concurrency
rgw/multisite: fix uninitialized LatencyMonitor average and use exponentially weighted moving average
rgw/multisite: expose lock latency as perf counter for data sync
Patrick Donnelly [Fri, 15 May 2026 17:27:08 +0000 (13:27 -0400)]
Merge PR #68776 into tentacle
* refs/pull/68776/head:
tentacle: test/neorados: Narrow Asio includes in `leak_watch_notify`
test/neorados: Don't leak watch handle
neorados: Avoid double cleanup in watch/notify
neorados: Actually enforce notification queue limit
neorados: Do not try to decode an empty response in notify
neorados: Go through linger_cancel on `io_context` shutdown
common/async: Fix removal from service list
osdc: remove implicit LingerOp reference between watch/unwatch
osdc: linger_register() returns intrusive_ptr<LingerOp>
neorados: NotifierHandler holds intrusive_ptr<LingerOp>
neorados: Notifier holds intrusive_ptr<LingerOp>
librados: aio_unwatch() delivers ENOTCONN to AioCompletion
osdc: Objecter::linger_by_cookie() for safe cast from uint64
librados: linger callbacks hold a reference to LingerOp
Adam C. Emerson [Wed, 26 Nov 2025 05:59:35 +0000 (00:59 -0500)]
test/neorados: Don't leak watch handle
Add missing unwatch call.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit ccc40eb69b4e8b66f7b9d32622bb22614b410166) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 26 Nov 2025 05:59:13 +0000 (00:59 -0500)]
neorados: Avoid double cleanup in watch/notify
An error coming in after `maybe_cleanup()` is called could trigger it
again. Add a flag to prevent that.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit d4ee87e985a6f2234050fd85d31564b7776bf1e6) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
We were adding an overflow marker on every message above capacity, not
just the first, vitiating the purpose of the bound. The shame, the
shame.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4cc2dbe316be9d8ac63ee00a6aff5455ecb4cd86) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 26 Nov 2025 03:15:13 +0000 (22:15 -0500)]
neorados: Do not try to decode an empty response in notify
The response can be empty on some errors. Attempting to decode an
empty one loses the error value on valid errors. Also swallow any
decode errors.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 89134791c16a2332457cfea85321a07e9ff7ca87) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Fri, 6 Mar 2026 00:53:17 +0000 (19:53 -0500)]
neorados: Go through linger_cancel on `io_context` shutdown
Rather than just dropping the reference, clean up the linger operation
properly within Objecter. Also, clear out handlers before
relinquishing reference to avoid use-after-free.
Fixes: https://tracker.ceph.com/issues/75164 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit f5affe06676bec2f32f856bce85b5d78932d807f)
Conflicts:
qa/workunits/rados/test.sh
- Just drop the `watch_leak` test on teuthology since it's of
marginal utility and not worth backporting all of
<https://github.com/ceph/ceph/pull/64219>.
Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 9 Mar 2026 22:03:42 +0000 (18:03 -0400)]
common/async: Fix removal from service list
Thanks to Seena Fallah <seenafallah@gmail.com> for this fix, part of a
larger commit.
Fixes: https://tracker.ceph.com/issues/75164 Co-authored-by: Seena Fallah <seenafallah@gmail.com> Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit b6a54e67cbcf11b23dcd5a5cd59795aeb7ff948e) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Casey Bodley [Thu, 11 Dec 2025 19:19:01 +0000 (14:19 -0500)]
osdc: remove implicit LingerOp reference between watch/unwatch
before this change set, linger_register() returned a raw LingerOp
pointer with an implicit reference for the caller. for librados,
this implicit reference is only dropped when the corresponding
unwatch() calls linger_cancel()
after commit 94f42b648feea77bd09dc3fdb48e6db2b48c7717 introduced
linger_by_cookie(), unwatch() no longer has a safe way to drop this
implicit reference. to prevent LingerOp leaks when unwatch() returns
ENOTCONN, we can't hold this implicit reference count until unwatch()
linger_register() now returns an explicit reference to the caller as
intrusive_ptr<LingerOp>. this helps to guarantee that this reference
count gets dropped before the completion of watch()/aio_watch()
because linger_register() no longer acquires an implicit reference for
the caller, linger_cancel() no longer drops it with info->put()
Reported-by: Ilya Dryomov <idryomov@gmail.com> Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1b0f873162d4bc357b230a78452531fdf39a6b25) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Casey Bodley [Thu, 11 Dec 2025 16:34:00 +0000 (11:34 -0500)]
librados: aio_unwatch() delivers ENOTCONN to AioCompletion
94f42b648feea77bd09dc3fdb48e6db2b48c7717 added a new error condition to
IoCtx::aio_unwatch() that callers aren't prepared to handle. instead of
returning that error directly, report it asynchronously to the
AioCompletion
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c0c146d37ae793fd1e1ab2a5118eac40149f2c6b) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
osdc: Objecter::linger_by_cookie() for safe cast from uint64
a `linger_ops_set` was added for `Objecter::handle_watch_notify()`
as a safety check before casting `uint64_t cookie` to `LingerOp*`
and deferencing it
neorados also made use of this set through `Objecter::is_valid_watch()`
checks. however, this approach was still susceptible to use-after-free,
because the callers didn't preserve a LingerOp reference between this
check and its use - and the Objecter lock is dropped in between. in
addition, `neorados::RADOS::unwatch_()` was missing its check for
`is_valid_watch()`
librados did not make use of this `is_valid_watch()` at all, so was
casting cookies directly to LingerOp* and dereferencing. this results
in use-after-free for any cookies invalidated by `linger_cancel()` -
for example when called by `CB_DoWatchError`
replace `is_valid_watch()` with a `linger_by_cookie()` function that
* performs the validity check with `linger_ops_set`,
* safely reinterpret_casts the cookie to LingerOp*, and
* returns a reference to the caller via intrusive_ptr<LingerOp>
`librados::IoCtxImpl::watch_check()`, `unwatch()` and `aio_unwatch()`
now call `linger_by_cookie()`, so have to handle the null case by
returning `-ENOTCONN` (this matches neorados' existing behavior)
Fixes: https://tracker.ceph.com/issues/72771 Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 94f42b648feea77bd09dc3fdb48e6db2b48c7717) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
projects / ceph.git / log