Alex Ainscow [Wed, 8 Apr 2026 10:49:58 +0000 (11:49 +0100)]
osd: Allow multiple objects with same version in missing list.
Most of the time, a single version in a PG can only correspond to a single object.
However, following a PG merge it is possible, even likely, that two objects will
have the same version. The PG Log works around this by discarding the log.
However, during backfill, it is possible for the missing list to be build with
these duplicate versions.
A recently added assert detected that this scenario was corrupting the reverse
missing list (rmissing). This behaviour has always existed, but was previously
unnoticed. It could cause some bugs and potentially loop-asserts on OSDs,
although mostly would not be noticed.
Here we fix this properly, by converting rmissing to a multimap. This is wrapped
in some insert functions, which assert that the rmissing list does not end up
with duplicate entries. The code is optimised for the case where there are no
duplicate versions.
Additionally, some of the old asserts have been rolled into the insert functions.
Fixes: https://tracker.ceph.com/issues/75778 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
(cherry picked from commit f3940400952b444a31f59b633fa3fa35437c87a9)
mgr: guard close_section calls in get_perf_schema_python
When a daemon exists in the daemon state map but has no perf counters
(e.g. an OSD that was running but is now down or destroyed), the loop
in get_perf_schema_python never executes, leaving prev_key_name empty
and no formatter sections opened. The unconditional close_section()
calls after the loop then trigger an assertion failure in PyFormatter
(cursor != root).
Add the same `if (!prev_key_name.empty())` guard that already protects
the close_section() calls inside the loop, so we only close sections
that were actually opened.
The thrashosds task is occasionally restarting OSDs and mon/mgr log
warnings are wrongly flagging this as a problem.
Fixes: https://tracker.ceph.com/issues/76747 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit a1430973f2c3e8758890447a1de3b906da77b60e)
Conflicts:
qa/suites/upgrade/reef-x/stress-split/1-start.yaml
qa/suites/upgrade/squid-x/stress-split/1-start.yaml
Resolution: different upgrade suites, just add the change manually
libcephsqlite: ensure atexit handlers are registered after openssl
When the sqlite3 executable encounters an error with .bail=on, it will
make a call to exit(). The atexit() handlers will execute in LIFO order.
We need to ensure that openssl (before OpenSSL 4.0 [1]) atexit handlers are
registered before libcephsqlite.
[1] http://github.com/openssl/openssl/commit/31659fe32673a6bd66abf3f8a7d803e81c6ffeed (OpenSSL 4.0 no longer arms `OPENSSL_cleanup()` function as an `atexit(3)`)
Fixes: https://tracker.ceph.com/issues/59335 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 7949cd5f12eb7cc0dc85fd1b5c1d795fad1df922)
Redouane Kachach [Fri, 29 May 2026 09:09:44 +0000 (11:09 +0200)]
qa/tasks: capture CommandCrashedError when running nvme list cmd
The safe_while retry loop does not catch exceptions, so a
CommandCrashedError from `nvme list` bypasses it entirely. Catch
CommandCrashedError and continue the retry loop instead.
Bill Scales [Fri, 15 May 2026 14:39:25 +0000 (15:39 +0100)]
osd: Fix bug when calculating min_peer_features
PeeringState calculates the minimum set of features for the set
of OSDs within a PG. There is a bug when the peer info has
already been cached where these peers features are not included
in the calculation. This can lead to the min feature set
including features that not all OSDs have.
Previously this just made some asserts less aggressive than they
should have been. Pull request https://github.com/ceph/ceph/pull/57740
uses min_peer_features to decide how to encode messages to other OSDs.
Midway through an upgrade this bug can cause an OSD to send
the wrong version of a message to a downlevel OSD causing
it to abort.
Fixes: https://tracker.ceph.com/issues/76600 Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>
(cherry picked from commit ce5882778db9b92b14a6bb5eca22e3cecf2be9dc)
tentacle: tentacle-p2p add centos9 to stress-split
Adds CentOS 9 Stream as a supported host OS for the tentacle-p2p
stress-split suite.
Rocky10 is not added here. This suite tests a bare-metal p2p upgrade
between Tentacle point releases. Rocky10 package support was added during
that release cycle, meaning there is no valid "FROM" package baseline to
install on Rocky10, making a bare-metal point-to-point upgrade path on Rocky10
impossible to test.
tentacle: upgrade/tentacle-p2p install pytest for rbd-python tests
The rbd-python workload runs test_librbd_python.sh, which invokes
'python3 -m pytest' to run the librbd Python API tests.
On a bare-metal install, python3-pytest is not pulled in by the
Ceph packages and is not present and the workunit fails.
Add it via extra_system_packages so the tests can run.
tentacle: suites/upgrade add centos to centos image upgrade
Previously, each suite had a single upgrade-sequence.yaml that targeted
only one image.
This commit splits each upgrade sequence into two variants teuthology
picks one per run
This applies to:
- reef-x/parallel
- reef-x/stress-split
- squid-x/parallel
- squid-x/stress-split
- telemetry/reef-x
- telemetry/squid-x
For the stress-split suites, the upgrade logic is split into a
first-half-sequence run concurrently with thrashosds,
and a second-half-sequence run after.
Both sequences contain the hardcoded target image, so each variant
needs its own copy. They were previously inlined in 1-start.yaml.
This commit extracts them into the upgrade-sequence$/ files so
each variant can target the right image.
Patrick Donnelly [Tue, 19 May 2026 23:15:29 +0000 (19:15 -0400)]
Merge PR #68803 into tentacle
* refs/pull/68803/head:
rgw/multisite: concurrency adjustment - consider the case caller provides 1
rgw/multisite: log concurrency state transitions in adj_concurrency
rgw/multisite: fix uninitialized LatencyMonitor average and use exponentially weighted moving average
rgw/multisite: expose lock latency as perf counter for data sync
Patrick Donnelly [Fri, 15 May 2026 17:27:08 +0000 (13:27 -0400)]
Merge PR #68776 into tentacle
* refs/pull/68776/head:
tentacle: test/neorados: Narrow Asio includes in `leak_watch_notify`
test/neorados: Don't leak watch handle
neorados: Avoid double cleanup in watch/notify
neorados: Actually enforce notification queue limit
neorados: Do not try to decode an empty response in notify
neorados: Go through linger_cancel on `io_context` shutdown
common/async: Fix removal from service list
osdc: remove implicit LingerOp reference between watch/unwatch
osdc: linger_register() returns intrusive_ptr<LingerOp>
neorados: NotifierHandler holds intrusive_ptr<LingerOp>
neorados: Notifier holds intrusive_ptr<LingerOp>
librados: aio_unwatch() delivers ENOTCONN to AioCompletion
osdc: Objecter::linger_by_cookie() for safe cast from uint64
librados: linger callbacks hold a reference to LingerOp
Adam C. Emerson [Wed, 26 Nov 2025 05:59:35 +0000 (00:59 -0500)]
test/neorados: Don't leak watch handle
Add missing unwatch call.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit ccc40eb69b4e8b66f7b9d32622bb22614b410166) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 26 Nov 2025 05:59:13 +0000 (00:59 -0500)]
neorados: Avoid double cleanup in watch/notify
An error coming in after `maybe_cleanup()` is called could trigger it
again. Add a flag to prevent that.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit d4ee87e985a6f2234050fd85d31564b7776bf1e6) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
We were adding an overflow marker on every message above capacity, not
just the first, vitiating the purpose of the bound. The shame, the
shame.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 4cc2dbe316be9d8ac63ee00a6aff5455ecb4cd86) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 26 Nov 2025 03:15:13 +0000 (22:15 -0500)]
neorados: Do not try to decode an empty response in notify
The response can be empty on some errors. Attempting to decode an
empty one loses the error value on valid errors. Also swallow any
decode errors.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 89134791c16a2332457cfea85321a07e9ff7ca87) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Fri, 6 Mar 2026 00:53:17 +0000 (19:53 -0500)]
neorados: Go through linger_cancel on `io_context` shutdown
Rather than just dropping the reference, clean up the linger operation
properly within Objecter. Also, clear out handlers before
relinquishing reference to avoid use-after-free.
Fixes: https://tracker.ceph.com/issues/75164 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit f5affe06676bec2f32f856bce85b5d78932d807f)
Conflicts:
qa/workunits/rados/test.sh
- Just drop the `watch_leak` test on teuthology since it's of
marginal utility and not worth backporting all of
<https://github.com/ceph/ceph/pull/64219>.
Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 9 Mar 2026 22:03:42 +0000 (18:03 -0400)]
common/async: Fix removal from service list
Thanks to Seena Fallah <seenafallah@gmail.com> for this fix, part of a
larger commit.
Fixes: https://tracker.ceph.com/issues/75164 Co-authored-by: Seena Fallah <seenafallah@gmail.com> Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit b6a54e67cbcf11b23dcd5a5cd59795aeb7ff948e) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Casey Bodley [Thu, 11 Dec 2025 19:19:01 +0000 (14:19 -0500)]
osdc: remove implicit LingerOp reference between watch/unwatch
before this change set, linger_register() returned a raw LingerOp
pointer with an implicit reference for the caller. for librados,
this implicit reference is only dropped when the corresponding
unwatch() calls linger_cancel()
after commit 94f42b648feea77bd09dc3fdb48e6db2b48c7717 introduced
linger_by_cookie(), unwatch() no longer has a safe way to drop this
implicit reference. to prevent LingerOp leaks when unwatch() returns
ENOTCONN, we can't hold this implicit reference count until unwatch()
linger_register() now returns an explicit reference to the caller as
intrusive_ptr<LingerOp>. this helps to guarantee that this reference
count gets dropped before the completion of watch()/aio_watch()
because linger_register() no longer acquires an implicit reference for
the caller, linger_cancel() no longer drops it with info->put()
Reported-by: Ilya Dryomov <idryomov@gmail.com> Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1b0f873162d4bc357b230a78452531fdf39a6b25) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Casey Bodley [Thu, 11 Dec 2025 16:34:00 +0000 (11:34 -0500)]
librados: aio_unwatch() delivers ENOTCONN to AioCompletion
94f42b648feea77bd09dc3fdb48e6db2b48c7717 added a new error condition to
IoCtx::aio_unwatch() that callers aren't prepared to handle. instead of
returning that error directly, report it asynchronously to the
AioCompletion
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c0c146d37ae793fd1e1ab2a5118eac40149f2c6b) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
osdc: Objecter::linger_by_cookie() for safe cast from uint64
a `linger_ops_set` was added for `Objecter::handle_watch_notify()`
as a safety check before casting `uint64_t cookie` to `LingerOp*`
and deferencing it
neorados also made use of this set through `Objecter::is_valid_watch()`
checks. however, this approach was still susceptible to use-after-free,
because the callers didn't preserve a LingerOp reference between this
check and its use - and the Objecter lock is dropped in between. in
addition, `neorados::RADOS::unwatch_()` was missing its check for
`is_valid_watch()`
librados did not make use of this `is_valid_watch()` at all, so was
casting cookies directly to LingerOp* and dereferencing. this results
in use-after-free for any cookies invalidated by `linger_cancel()` -
for example when called by `CB_DoWatchError`
replace `is_valid_watch()` with a `linger_by_cookie()` function that
* performs the validity check with `linger_ops_set`,
* safely reinterpret_casts the cookie to LingerOp*, and
* returns a reference to the caller via intrusive_ptr<LingerOp>
`librados::IoCtxImpl::watch_check()`, `unwatch()` and `aio_unwatch()`
now call `linger_by_cookie()`, so have to handle the null case by
returning `-ENOTCONN` (this matches neorados' existing behavior)
Fixes: https://tracker.ceph.com/issues/72771 Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 94f42b648feea77bd09dc3fdb48e6db2b48c7717) Fixes: https://tracker.ceph.com/issues/76434 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
mds: add retry request to MDSRank wait queue rather via finisher
C_MDS_RetryRequest inherits from MDSInternalContext which does not
acquire mds_lock by itself. Adding to MDSRank wait queue will process
this via the progress thread which completes the context with mds_lock
acquired.
Adam Kupczyk [Wed, 13 May 2026 04:50:04 +0000 (06:50 +0200)]
extblkdev: Fix FCM plugin asserting on multivolume devices
The issue is that FCM does not work properly with BlueStore "block"
that spans over multiple devices, even FCM ones.
The current code detects it, and issues health warning.
But by coding mistake the check for it applies to all deployments,
and is an assert-grade error.
The original problem was introduced with:
https://github.com/ceph/ceph/pull/68024
(as backport of https://github.com/ceph/ceph/pull/66318)
It was supposed to be solved with:
https://github.com/ceph/ceph/pull/68663
(as backport of https://github.com/ceph/ceph/pull/68416)
But it was not, since the actual offending lines were removed
by this temporary solution: https://github.com/ceph/ceph/pull/68391
This commit now properly removes the lines were not removed. Fixes: https://tracker.ceph.com/issues/76581 Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
Patrick Donnelly [Wed, 13 May 2026 12:40:50 +0000 (08:40 -0400)]
Merge PR #68476 into tentacle
* refs/pull/68476/head:
mgr/dashboard: Update permissions for pool-manager role
mgr/dashboard : Select replicated rule by default in pools form
mgr/dashboard : Fix application names in pools form
mgr/dashboard : add stretch cluster validation for pools form
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Kyr Shatskyy [Thu, 30 Oct 2025 11:58:05 +0000 (12:58 +0100)]
qa/cephfs: lua to respect missing kernel in yaml
When teuthology-suite is called with '-k none' option, which is valid,
there is no kernel record in job config created.
However at some test cases the lua premerge dies with exception:
KeyError: 'kernel'
and when branch is not set for '-k none' and kernel client is
overridden:
KeyError: 'branch'
so teuthology-suite quits unexpectedly without scheduling any jobs.
Remove 'success_message_template' from NvmeofCLICommand decorator of
NVMeoFSubsystem.add_network() and NVMeoFSubsystem.del_network().
This is because 'success_message_template' feature introduction PR
hasn't been backported to tentacle.
This commit can be reverted later in tentacle branch.
Gil Bregman [Mon, 13 Apr 2026 21:41:25 +0000 (00:41 +0300)]
mgr/dashboard: Add port and secure-listeners to subsystem add NVMeoF CLI command
Fixes: https://tracker.ceph.com/issues/75998 Signed-off-by: Gil Bregman <gbregman@il.ibm.com>
(cherry picked from commit 624adc09431dc2fdfa617940161f188c0831bf97)
Conflicts:
src/pybind/mgr/dashboard/controllers/nvmeof.py
Resolve conflict to use "traddr" instead of "server_address"
in NVMeoFSubsystem().create() parameters.
Main branch renamed the param ("traddr") to "server_address".
Tentacle
Vallari Agrawal [Thu, 12 Mar 2026 13:50:00 +0000 (19:20 +0530)]
mgr/dashboard: Add 'network_mask' to nvmeof cli
This commit add the following to nvmeof cli:
0. Add new param `--network-mask` to 'subsystem add' cmd
It's a list parameter so we can pass multiple netmask by
`subsystem add --network-mask <subnet1> --network-mask <subnet2>`
1. Add new cli `subsystem add_network --network-mask <subnet>`
2. Add new cli `subsystem del_network --network-mask <subnet>`
3. Add column 'network_mask' to `subsystem list` output
4. Add column 'manual' to `listener list` output
Conflicts:
src/pybind/mgr/dashboard/controllers/nvmeof.py
NVMeoFSubsystem controller uses param name "traddr"
in tentacle branch and its renamed to "server_address"
in main branch. Since its a breaking change, it would be
changed to "server_address" in next major version.
So in this backport commit, we use "traddr" in create(),
add_network(), and del_network().
PR #67318 (boto3 migration) cherry-picked onto tentacle conflicted with
PR #66168 which had added test_suspended_delete_marker_incremental_sync
using the old boto API. The conflict resolution added the boto3-rewritten
version of the function but left the original old-API version in place,
resulting in dup definitions with the same name.
Remove the stale old-API duplicate; keep the boto3 version added by PR #67318.
* refs/pull/68505/head:
rgw: reenable 'bucket stats' on indexless buckets
rgw: 'bucket stats' omits usage for buckets on other zonegroups
rgw/rados: pass SiteConfig into bucket_stats()
rgw: bucket_stats() uses local variable 'index'
* refs/pull/66769/head:
test/rgw/logging: run teuthology on erasure coded pool
rgw/bucket-logging: support for EC pools
rgw/logging: do not create empty temporary objects
rgw/logging: deleteting the object holding the temp object name on cleanup
rgw/logging: make sure source bucket is in the target's list
rgw/logging: removed unused APIs from header
rgw/logging: fix race condition when name update returns ECANCELED
rgw/logging: add error message when log_record fails
rgw/logging: rollover objects when conf changes
rgw/logging: allow committing empty objects
rgw/logging: verify http method exists
rgw/logging: fix/remove/add bucket logging op names
rgw/logging: refactor canonical_name()
rgw/logging: fix canonical names
rgw: RGWPostBucketLoggingOp uses yield context
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com> Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
* refs/pull/67923/head:
rgw/test [tentacle]: add missing ceph and call_ceph shell functions
RGW/test_multi/RGWBucketFullSyncCR: test bucket full sync while source bucket is deleted in the middle
RGW/multisite/RGWListRemoteBucketCR: clear reused bucket_list_result to avoid stale listings
RGW/multisite: bucket_list_result object provides a method to reset its entries
RGW/multisite: add some more debug logs to sync codepath
RGW/test_multi: remove unused import
RGW/test_multi: allow Cluster object to run ceph admin commands
RGW: add delay injection options for integration testing
RGW: make SSTR macro safe against variable name collisions
* refs/pull/68374/head:
qa: enforce centos9 for test
qa: rename distro
qa/suites/fs/bugs: use centos9 for squid upgrade test
qa: remove unused variables
qa: use centos9 for fs suites using k-testing
qa: update fs suite to rocky10
qa: skip dashboard install due to dependency noise
qa: only setup nat rules during bridge creation
qa: correct wording of comment
qa: use nft instead iptables
qa: use py3 builtin ipaddress module
* refs/pull/68569/head:
tentacle: only add package tasks if rocky is the final distro
qa/distros: add centos 9 stream back to supported distros
qa/distros: re-install nvme-cli package in rocky tests
qa: allowlist bpf podman denials on Rocky 10
qa/distros: bump rocky to 10.1
qa/distros: add rocky_10 as supported container host
qa/distros: bump rpm_latest.yaml to rocky_10.yaml
qa/distros: rename centos_latest.yaml to rpm_latest.yaml
qa/distros: add rocky_9 and rocky_10
projects / ceph.git / log