The readable.sh script has forward incompat checks, but no
backward incompat checks.
This fix will:
1. Add check for backward_incompat directory for each type for specific
objects or all objects with the same type and skip those objects from being tested.
2. Add version comparison helper functions (version_lt, version_le, version_ge,
versions_span) for robust version handling
3. Replace 'sort -n' with 'sort -V' for proper version number sorting
4. Add CORPUS_PATH environment variable to allow teuthology tests to execute this script
5. Improve readability of the script
The difference between backward and forward incompat:
- forward_incompat: Marks objects from older versions that newer ceph-dencoder
versions cannot read. Example: Version 19.2.x objects marked incompat at version 20.2.x
means ceph-dencoder v20.2.x+ can't decode them. Skip when testing old objects
with a new ceph-dencoder.
- backward_incompat: Marks objects from newer versions that older ceph-dencoder
versions cannot read. Example: Version 19.2.x objects marked backward_incompat at v19.2.x
means ceph-dencoder < v19.2.x can't decode them. Skip when testing new objects
with an old ceph-dencoder.
NitzanMordhai [Mon, 2 Feb 2026 07:34:24 +0000 (07:34 +0000)]
workunits/dencoder: use readable.sh script instead of python script
The python script test_readable.py was added for backword and forward
compability. maintaining 2 scripts that finally doing the same is west,
reverting and using readable.sh and leave the python out.
Patrick Donnelly [Wed, 18 Mar 2026 00:22:23 +0000 (20:22 -0400)]
Merge PR #67623 into squid
* refs/pull/67623/head:
mgr/orchestrator: make group parameter optional for nvmeof (squid)
pybind/mgr/orchestrator/module.py: NvmeofServiceSpec service_id
The reason we had a slow-requests is because during the test, 16 concurrent 4 MB writes were running while recovery and backfill were disabled. At the same time, osd.0 was marked out and then back in, causing PG remapping. Because recovery/backfill was disabled, some PGs could not restore their replicas after the remap, leaving them in degraded/remapped states. As a result, a batch of writes remained stuck in the replicated write path, leading to IO stall and slow ops being reported. Solution is to ignore this as we are testing the progress module, not the write paths of OSDs. We intentionally disable backfill and recovery in order to prevent the recovery event to finish quickly. We wanted to prolong it until the progress event pops up.
qa: make test_progress atomically capture OSD marked in/out events
Problem:
Test had a race condition where events could complete and disappear
between checking the event count and fetching the event, causing
test failures.
Solution:
Refactor to atomically capture events during the wait condition check.
Added helper methods _wait_for_osd_marked_out_event() and
_wait_for_osd_marked_in_event() that capture events at the moment
they're detected, eliminating the race window.
Kefu Chai [Tue, 3 Mar 2026 04:51:32 +0000 (12:51 +0800)]
mgr/orchestrator: make group parameter optional for nvmeof (squid)
Add default value for group parameter in nvmeof commands to maintain
backward compatibility with existing squid tests and deployments.
Context:
--------
On main branch, when commit 6bee4e10f7f added the group parameter, the
tests were subsequently updated to provide the group argument explicitly:
Main test: ceph orch apply nvmeof foo default
Expected: nvmeof.foo.default
However, on squid branch, the existing tests still use the older syntax
without specifying a group:
The previous cherry-pick (e1612d048a1) fixed the service_id construction
logic to handle empty groups correctly, but the group parameter was still
required without a default value, causing "ceph orch apply nvmeof foo" to
fail with EINVAL (missing required argument).
This commit adds the missing default value (group: str = '') to make the
parameter optional, maintaining backward compatibility with existing squid
tests and user scripts that don't specify a group.
With both changes:
1. Cherry-picked e1612d048a1: service_id logic handles empty group
2. This commit: group parameter has default value ''
Result:
"ceph orch apply nvmeof foo" works (creates nvmeof.foo)
"ceph orch apply nvmeof foo mygroup" also works (creates nvmeof.foo.mygroup)
Test: qa/suites/orch/cephadm/smoke-roleless/2-services/nvmeof.yaml
Fixes job 50373 failure from test run dgalloway-2026-02-13_23:06:25
Please note, this change was not cherry-picked from main branch, because
main intentionally still requires the CLI group argument for arch
apply/add nvmeof, and its tests were updated accordingly.
On squid, however, the earlier cherry-pick 6bee4e10 introduced the
required group parameter, but squid still has the old test/behavior
(ceph orch apply nvmeof foo expecting nvmeof.foo) and does not contain
the later main commits 3e5e85aadc1 and b377085c302.
- make service_id better alligned with default/empty group
(https://github.com/ceph/ceph/commit/f6d552d7c777f1160545188dcffa6b685b05ca8a)
- fix service_id in nvmeof daemon add
Ilya Dryomov [Tue, 24 Feb 2026 11:46:35 +0000 (12:46 +0100)]
librbd/mirror: detect trashed snapshots in UnlinkPeerRequest
If two instances of UnlinkPeerRequest race with each other (e.g. due
to rbd-mirror daemon unlinking from a previous mirror snapshot and the
user taking another mirror snapshot at same time), the snapshot that
UnlinkPeerRequest was created for may be in the process of being removed
(which may mean trashed by SnapshotRemoveRequest::trash_snap()) or fully
removed by the time unlink_peer() grabs the image lock. Because trashed
snapshots weren't handled explicitly, UnlinkPeerRequest could spuriously
fail with EINVAL ("not mirror snapshot" case) instead of the expected
ENOENT ("missing snapshot" case). This in turn could lead to spurious
ImageReplayer failures with it stopping prematurely.
ImageUpdateWatchers::flush() requests aren't tracked with
m_in_flight-like mechanism the way ImageUpdateWatchers::send_notify()
requests are, but in both cases callbacks that represent delayed work
that is very likely to (indirectly) reference ImageCtx are involved.
When the image is getting closed, ImageUpdateWatchers::shut_down() is
called before anything that belongs to ImageCtx is destroyed. However,
the shutdown can complete prematurely in the face of a pending flush if
one gets sent shortly before CloseRequest is invoked. The callback for
that flush will then race with CloseRequest and may execute after parts
of or even the entire ImageCtx is destroyed, leading to use-after-free
and various segfaults.
Ilya Dryomov [Thu, 19 Feb 2026 14:45:39 +0000 (15:45 +0100)]
test: disable known flaky tests in run-rbd-unit-tests
The failures seem to be more frequent on newer hardware. In the
absence of immediate fixes, disable a few tests that have been known to
be flaky for a long time to avoid disrupting "make check" runs.
Patrick Donnelly [Mon, 21 Apr 2025 15:20:46 +0000 (11:20 -0400)]
qa/workunits/fs/misc: remove data pool cleanup
This cleanup is at the very least incorrect as it can cause the MDS to throw
read-only errors because the data pool is removed before it can write out
backtraces.
We've not yet finalized a truly safe workflow to remove a data pool -- even
flushing the MDS journals first may not be enough (considering a large purge
queue).
Fixes: https://tracker.ceph.com/issues/70919 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit f45cf2d53729b95ed1257241efaaa97d3d63abe1)
Ville Ojamo [Mon, 5 Jan 2026 06:10:45 +0000 (13:10 +0700)]
doc: Remove sphinxcontrib-seqdiag Python package from RTD builds
This is a proactive PR to avoid breaking docs builds when Setuptools 81
starts to be used in the RTD builds process.
The sphnixcontrib-seqdiag Python package is not compatible with
Setuptools 81 or later due to use of pkg_resources:
https://setuptools.pypa.io/en/latest/pkg_resources.html
Setuptools 81 release should be imminent, with the Python deprecation
warning stating pkg_resources "removal as early as 2025-11-30".
Seqdiag seems to be unmaintained with the latest update at Pypi in
the year 2021 and also no updates to the seqdiag git repo.
There are no seqdiag directives left in the docs after last seqdiags
were removed in PR #52308.
Two other options would exist for fixing the situation (see PR for
discussion) but this seems to be the suitable one.
Patrick Donnelly [Wed, 18 Feb 2026 15:41:32 +0000 (10:41 -0500)]
Merge PR #64815 into squid
* refs/pull/64815/head:
The compilation of ISAL compress in the current code depends on the macro HAVE_NASM_X64_AVX2. However, the macro HAVE_NASM_X64_AVX2 has been removed, resulting in the compression not using ISAL even if the compressor_zlib_isal parameter is set to true.
Ilya Dryomov [Tue, 13 Jan 2026 19:32:14 +0000 (20:32 +0100)]
osd/PrimaryLogPG: encode an empty data_bl for empty sparse reads
Commit 0cf383da0741 ("ReplicatedPG: clamp SPARSE_READ to object size
for ec pool") didn't handle the case of a sparse read that ends up
being empty correctly: the OSD encodes only an empty extent map whereas
clients (both userspace and kernel) also expect to see an empty data
buffer. IOW the reply contains one 32-bit zero instead of the expected
two.
Ilya Dryomov [Tue, 11 Nov 2025 20:39:58 +0000 (21:39 +0100)]
qa/valgrind.supp: make gcm_cipher_internal suppression more resilient
gcm_cipher_internal() and ossl_gcm_stream_final() make it to the stack
trace only on CentOS Stream 9. On Ubuntu 22.04 and Rocky 10, it looks
as follows:
Thread 4 msgr-worker-1:
Conditional jump or move depends on uninitialised value(s)
at 0x70A36D4: ??? (in /usr/lib64/libcrypto.so.3.2.2)
by 0x70A39A1: ??? (in /usr/lib64/libcrypto.so.3.2.2)
by 0x6F8A09C: EVP_DecryptFinal_ex (in /usr/lib64/libcrypto.so.3.2.2)
by 0xB498C1F: ceph::crypto::onwire::AES128GCM_OnWireRxHandler::authenticated_decrypt_update_final(ceph::buffer::v15_2_0::list&) (crypto_onwire.cc:271)
by 0xB4992D7: ceph::msgr::v2::FrameAssembler::disassemble_preamble(ceph::buffer::v15_2_0::list&) (frames_v2.cc:281)
by 0xB482D98: ProtocolV2::handle_read_frame_preamble_main(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int) (ProtocolV2.cc:1149)
by 0xB475318: ProtocolV2::run_continuation(Ct<ProtocolV2>&) (ProtocolV2.cc:54)
by 0xB457012: AsyncConnection::process() (AsyncConnection.cc:495)
by 0xB49E61A: EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) (Event.cc:492)
by 0xB49EA9D: UnknownInlinedFun (Stack.cc:50)
by 0xB49EA9D: UnknownInlinedFun (invoke.h:61)
by 0xB49EA9D: UnknownInlinedFun (invoke.h:111)
by 0xB49EA9D: std::_Function_handler<void (), NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (std_function.h:290)
by 0xBB11063: ??? (in /usr/lib64/libstdc++.so.6.0.33)
by 0x4F17119: start_thread (in /usr/lib64/libc.so.6)
The proposal to amend the existing suppression so that it's tied to the
specific callsite rather than libcrypto internals [1] received a thumbs
up from Radoslaw.
The existing StandardPolicy that exposed as RBD_LOCK_MODE_EXCLUSIVE
argument to rbd_lock_acquire() disables automatic exclusive lock
transitions with "permanent" semantics: any request to release the lock
causes the peer to error out immediately. Such a lock owner can
perform maintenance operations that are proxied from other peers, but
any write-like I/O issued by other peers will fail with EROFS.
This isn't suitable for use cases where one of the peers wants to
manage exclusive lock manually (i.e. rbd_lock_acquire() is used) but
the lock is acquired only for very short periods of time. The rest of
the time the lock is expected to be held by other peers that stay in
the default "auto" mode (AutomaticPolicy) and run as usual, completely
unconcerned with each other or the manual-mode peer. However, these
peers get acutely aware of the manual-mode peer because when it grabs
the lock with RBD_LOCK_MODE_EXCLUSIVE their I/O gets disrupted: higher
layers translate EROFS into generic EIO, filesystems shut down, etc.
Add a new TransientPolicy exposed as RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT
to allow disabling automatic exclusive lock transitions with semantics
that would cause the other peers to block waiting for the lock to be
released by the manual-mode peer. This is intended to be a low-level
interface -- no attempt to safeguard against potential misuse causing
e.g. indefinite blocking is made.
It's possible to switch between RBD_LOCK_MODE_EXCLUSIVE and
RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT modes of operation both while the
lock is held and after it's released.
Ilya Dryomov [Mon, 19 Jan 2026 16:43:41 +0000 (17:43 +0100)]
librbd: prepare lock_acquire() for changing between policies
In preparation for adding a new TransientPolicy, get rid of the check
implemented in terms of exclusive_lock::Policy::may_auto_request_lock()
that essentially makes it so that exclusive lock policy on a given
image handle can be changed from the default AutomaticPolicy only once.
In order to effect another change a new image handle would have been
needed which is pretty suboptimal.
Ilya Dryomov [Mon, 22 Dec 2025 18:07:27 +0000 (19:07 +0100)]
librbd: fix RequestLockPayload log message in ImageWatcher
exclusive_lock::Policy::lock_requested() isn't guaranteed to queue
the release of exclusive lock (and in fact only one of the two existing
implementations does that). Instead of talking about the lock, log the
response to the notification.
Ilya Dryomov [Mon, 22 Dec 2025 16:22:53 +0000 (17:22 +0100)]
librbd: amend error message in lock_acquire()
... since it went stale with commit 2914eef50d69 ("rbd: Changed
exclusive-lock implementation to use the new managed-lock"). In the
context of exclusive lock, requesting the lock refers to a specific
action which may or may not be performed as part of acquiring the lock
and lock_acquire() doesn't get visibility into that.
Kyr Shatskyy [Fri, 21 Nov 2025 21:20:04 +0000 (22:20 +0100)]
qa/workunits/rgw: drop netstat usage
The `netstat` is deprecated now in modern Linux and usually
requires an extra package dependency to be installed.
Usually it is `net-tools`, however, for example, opensuse,
`netstat` does not present in it. Thus, let us use `ss` as
an alternative.
When using `netstat -nltp` we get lines like:
'tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 25156/valgrind.bin \ntcp6 0 0 :::443 :::* LISTEN 25156/valgrind.bin \n'
When using `ss -nltp` we get lines like:
'LISTEN 0 4096 0.0.0.0:443 0.0.0.0:* users:(("memcheck-amd64-",pid=66045,fd=72))'
so we need to filter processes by `memcheck`. However further
parsing code works equivalently as for netstat.
Ilya Dryomov [Fri, 30 Jan 2026 15:32:35 +0000 (16:32 +0100)]
qa/tasks/rbd_mirror_thrash: don't use random.randrange() on floats
This stopped working in Python 3.12:
Changed in version 3.12: Automatic conversion of non-integer types
is no longer supported. Calls such as randrange(10.0) and
randrange(Fraction(10, 1)) now raise a TypeError.
Ilya Dryomov [Thu, 29 Jan 2026 20:41:03 +0000 (21:41 +0100)]
qa/workunits/rbd: reduce randomized sleeps in live import tests
These tests were tuned for slower hardware than what we have now.
Currently "rbd migration execute" always finishes (successfully) before
the NBD server is killed.
Conflicts:
qa/workunits/rbd/cli_migration.sh [ commit afc89fdde80f
("qa/workunits/rbd: add test_import_nbd_stream_disconnected()")
was originally skipped due to NBD stream not being in squid at
the time ]
Ilya Dryomov [Wed, 28 Jan 2026 09:41:13 +0000 (10:41 +0100)]
qa/workunits/rbd: drop randomized sleeps in "big image" tests
These tests were tuned for slower hardware than what we have now.
Even without these the image is often 25-30% synced by the time the
test gets to the "non-primary snapshot in question is still being
synced" assert.
Ilya Dryomov [Tue, 27 Jan 2026 20:56:23 +0000 (21:56 +0100)]
qa/workunits/rbd: avoid unnecessary sleeping in stop_mirror()
There is no need to wait for anything if -KILL is passed for sig
because the process would disappear immediately. In teuthology runs
where multiple rbd-mirror daemons are deployed (and therefore need to
be stopped when stop_mirrors() is called by the test), it causes
gratuitous delays of 4+ seconds.
Ilya Dryomov [Fri, 23 Jan 2026 13:48:53 +0000 (14:48 +0100)]
qa: don't assume that /dev/sda or /dev/vda is present in unmap.t
Instead of hard-coding the block device name, use the block device that
is backing the filesystem that the test is running on. We can be quite
sure it won't be an RBD device ;)
Ilya Dryomov [Wed, 21 Jan 2026 18:41:41 +0000 (19:41 +0100)]
qa: krbd_blkroset.t: eliminate a race in the open_count test
Even at QD=1, dd may take less than 10 seconds to work its way to the
end of a 10M image, producing "No space left on device" error instead
of the expected "Operation not permitted" error which is supposed to
arise from the device getting marked read-only while opened.
Disable OSD bench from benchmarking the OSDs for teuthology tests. This is to
help prevent a cluster warning pertaining to the IOPS value not lying within
a typical threshold range from being raised.
The tests can rely on the built-in static values as defined by
osd_mclock_max_capacity_iops_[ssd|hdd] which should be good enough.
Aashish Sharma [Wed, 17 Dec 2025 09:21:14 +0000 (14:51 +0530)]
monitoring: make cluster matcher backward compatible for pre-7.1 metrics
Ceph 18.* adds a `cluster` label to all Prometheus metrics. When
upgrading from earlier releases, historical metrics lack this label
and are excluded by Grafana queries that strictly match on `cluster`.
Update the shared Grafana matcher logic to use a regex matcher that
also matches series without the `cluster` label, restoring visibility
of pre-upgrade metrics while preserving multi-cluster behavior.
Conflicts:
monitoring/ceph-mixin/dashboards_out/ceph-application-overview.json (file not in squid)
monitoring/ceph-mixin/dashboards_out/ceph-cluster-advanced.json
(conflicts with $rate_interval in main)
monitoring/ceph-mixin/dashboards_out/ceph-cluster.json (missing
cluster label in metrics)
monitoring/ceph-mixin/dashboards_out/cephfsdashboard.json (file
not in squid)
monitoring/ceph-mixin/dashboards_out/multi-cluster-overview.json
(file not in squid)
Ville Ojamo [Mon, 19 Jan 2026 13:06:46 +0000 (20:06 +0700)]
doc/cephadm: remove sections not apply to Squid in rgw.rst
4949311 backported changes that do not apply to Squid.
PR #63073 body and the commit referenced therein as cherry-pick do not
correspond to the diff. Remove the additions that do not apply to Squid:
- Wildcard SAN feature in 3c24753 only since Tentacle.
- Shutdown delay feature in b84bb72 only since Tentacle.
The third feature doc addition is valid, d620ba6 was backported to Squid
in PR #61350 for disable multisite sync traffic, commit 59b3f28. This
backport cherry-picked only the feature addition and missed the docs
commit 8878619. Leave this section in.
Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
Conflicts:
monitoring/ceph-mixin/dashboards_out/ceph-nvmeof.json (file not
in squid)
monitoring/ceph-mixin/dashboards_out/multi-cluster-overview.json
(file not in squid)
src/python-common/ceph/cephadm/images.py (file not in squid,
need to change the image name in constants.py instead)
Dongdong Tao [Wed, 21 May 2025 08:48:52 +0000 (17:48 +0900)]
common: drop stack singleton object of temp messenger for foreground ceph daemons
During the initialization, OSD needs to create a temporary messenger to read config db from the ceph-mon.
This temporary messenger will need to create async messenger threads according to the local/default value of "ms_async_op_threads" .
If this option is not specified in ceph.conf, it will by default create 3 threads, then use these threads to read config db from ceph-mon.
Those threads are associated to a stack singletion object.
Now here is the difference between OSD running in foreground and background:
a. In background mode, this singleton object will be dropped before forking the child process in function notify_pre_fork,
then the new ms_async_op_threads fetched from mon config db will be used to create later messenger threads, this is what we want.
b. In foreground mode, this singleton object will not be dropped and will be reused by all later messengers,
thus the number of threads doesn't change.
Fixes: https://tracker.ceph.com/issues/71401 Signed-off-by: dongdong tao <dongdong.tao@canonical.com>
(cherry picked from commit 30d66ff075ca72c0c3759bfccee09302b221b25f)
The alert CephPGImbalance doesn't take any device classes configured into account. As a result, there can be false positives when using mixed-size OSD disks.
Ref: https://github.com/rook/rook/discussions/13126#discussioncomment-10043490
Conflicts:
monitoring/ceph-mixin/prometheus_alerts.yml (remove cluster
label from alert since its not there in squid)
monitoring/ceph-mixin/tests_alerts/test_alerts.yml (remove
cluster label from the alert since its not there in squid)
Kefu Chai [Wed, 24 Dec 2025 05:55:26 +0000 (13:55 +0800)]
debian/control: add iproute2 to build dependencies
Test scripts like qa/tasks/cephfs/mount.py expect the ip command to be
available in the container environment. Without it, tests fail with:
```
/bin/bash: line 1: ip: command not found
File "/ceph/qa/tasks/cephfs/mount.py", line 96, in cleanup_stale_netnses_and_bridge
p = remote.run(args=['ip', 'netns', 'list'],
...
teuthology.exceptions.CommandFailedError: Command failed with status 127: 'ip netns list'
```
Add iproute2 to the debian package build dependencies when the
<pkg.ceph.check> build profile is enabled. This ensures the package is
available during container-based builds, since buildcontainer-setup.sh
→ script/run-make.sh → install-deps.sh → debian/control → generated
dependency package chain respects build profiles configured via
`FOR_MAKE_CHECK` and `WITH_CRIMSON` environment variables set in
Dockerfile.build.
David Galloway [Tue, 16 Dec 2025 22:08:00 +0000 (17:08 -0500)]
install-deps: Replace apt-mirror
apt-mirror.front.sepia.ceph.com has happened to always work because we set up CNAMEs to gitbuilder.ceph.com.
That host is making its way to a new home upstate (literally and figuratively) so we'll get rid of the front subdomain since it's publicly accessible anyway and add TLS while we're at it.
Ilya Dryomov [Tue, 9 Dec 2025 14:22:02 +0000 (15:22 +0100)]
librbd: fix ExclusiveLock::accept_request() when !is_state_locked()
To accept an async request, two conditions must be met: a) exclusive
lock must be a firm STATE_LOCKED state and b) async requests shouldn't
be blocked or if they are blocked there should be an exception in place
for a given request_type. If a) is met but b) isn't, ret_val is set
to m_request_blocked_ret_val, as expected -- the reason for denying
the request is that async requests are blocked. However, if a) isn't
met, ret_val also gets set to m_request_blocked_ret_val. This is wrong
because the reason for denying the request in this case isn't that
async requests are blocked (they may or may not be) but a much heavier
circumstance of exclusive lock being in a transient state or not held
at all.
In such scenarios, whether async requests are blocked or not isn't
relevant and ExclusiveLock::accept_request() behaving otherwise can
lead to bogus "duplicate lock owners detected" errors getting raised
during an attempt to handle any maintenance operation notification in
ImageWatcher::handle_operation_request(). This error isn't considered
retryable so the entire operation that needed the exclusive lock would
be spuriously failed with EINVAL.
projects / ceph.git / log