msgr/async: make msgr_active_connections counter a gauge
msgr_active_connections tracks the current number of active connections
rather than a monotonic total. Register it as a gauge so perf reset does
not zero it while live connections may still later decrement the value.
John Mulligan [Thu, 16 Apr 2026 17:47:04 +0000 (13:47 -0400)]
ceph.spec.in: add new --with pypkg to be passed on to cmake
Add a new --with pypkg option that passes WITH_PYPKG to cmake.
This allows building with the new (experimental) python packaging
support. If this proves useful a future change can consider enabling
by default in some conditions.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 13 Apr 2026 21:24:39 +0000 (17:24 -0400)]
python-common: update CMakeLists.txt to optionally use new packaging
Add support for invoking the new pep517 based packaging mode added
in a previous commit. Because this approach will not work on older
distros and there seems to be spotty support for the new packging
form on debian/ubuntu (when nested within a additional layer like
CMake) I am choosing not to enable the new stuff by default.
View with `git diff -w`
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 13 Apr 2026 21:24:46 +0000 (17:24 -0400)]
python-common: add a pyproject.toml file
In order to support the current python packaging standards we need
to have a pyproject.toml [1] file. This file defines the project's
metadata and build tool.
For continuity, I have left setuptools in place as the build backend
so the existing steup.py is still in play. I also experimented with
flit as a back-end. Flit seemed to work OK but I was a bit unsure
how distro support for it would be when we started to roll out this
option. Thus, to be safe I decided to stay with setuptools for now.
John Mulligan [Mon, 13 Apr 2026 21:24:24 +0000 (17:24 -0400)]
cmake/modules: add PythonPackage.cmake
Time is marching on and the state of the art with python packaging has
not stood still. In Python 3.12, distutils has been removed after being
deprecated for a couple of versions. According to the Python Packaging
User Guide [1]: "However, `python setup.py` and the use of `setup.py` as a
command line tool are deprecated."
Currently, ceph provides a decent sized and growing library of python
code in `src/python-common/ceph`. It currently relies on `setup.py` and
the deprecated `python setup.py install` command. This change aims to be
the first step in moving toward a more contemporary approach so that we
don't get caught late when the older approaches really stop working.
Because ceph's primary diver of "build stuff" is CMake, there was an
existing `cmake/modules/Distutils.cmake` that invokes a `python setup.py
install` command. Rather than risk breaking older distros we add a new
`cmake/modules/PythonPackage.cmake` file that uses the PEP 517/518
[2][3] style
of packaging. I could not find some existing CMake support for this
so unfortunately I had to write this.
The approach taken is loosely based on what the rpm build process does.
It invokes pip's wheel subcommand to build a wheel (during the build
phase) and then uses pip to install the wheel to install the content
to the system.
A future commit will add conditional support for using this approach
in src/python-common.
Jamie Pryde [Fri, 1 May 2026 09:45:42 +0000 (10:45 +0100)]
cmake: Fix ISA-L build on arm
A typo in CFLAGS means we're passing an empty string to configure_cmd.
We are then overwriting the build environment CFLAGS with our empty string CFLAGS,
which can result in build failures in certain environments, as seen in the tracker.
This fix gets any build environment CFLAGS and appends the other flags
we want to use when building ISA-L 2.32.0
mgr/dashboard: "Access Denied" being shown on overview page for read-only user
Fix: https://tracker.ceph.com/issues/76293 Signed-off-by: Devika Babrekar <devika.babrekar@ibm.com>
Alex Ainscow [Fri, 24 Apr 2026 14:57:55 +0000 (15:57 +0100)]
osd: Fix incorrect rollback logic for partial write OI
Before this fix, when rolling back an OI, the system used the OI
from the primary to rollback to. This is wrong if the previous
write was a partial write. This may have a few consequences
during recovery (although its not clear any are serious) and in
EC direct reads, where a false-positive version mismatch will be
detected.
The test provided recreates the issue.
The fix provided modifies the rollback as it is being written.
Fixes: https://tracker.ceph.com/issues/76213 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
# Conflicts:
# src/test/osd/TestECFailoverWithPeering.cc
Alex Ainscow [Mon, 27 Apr 2026 16:46:40 +0000 (17:46 +0100)]
osd: Avoid assertion on empty object read when reading multiple objects
Tracker 75432 hits an assert which is attempting to protect the system
against hanging, due to generating a read request which sends no messages.
The assert fired because recovery was attempting to read multiple objects
in a single read request. One object did not require any further shard
reads in order to recover, while the other did. The consequence is that
the assert fired on one of the objects.
The problem is simply that the assert is in the wrong place.
Fixes: https://tracker.ceph.com/issues/75432 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Kefu Chai [Wed, 6 May 2026 03:32:03 +0000 (11:32 +0800)]
test/osd: add perf test for calc_pg_upmaps with mutual overfull upmap pairs
Reproduces the scenario from https://tracker.ceph.com/issues/63137: 500
cohort OSDs all at deviation +2, linked by pg_upmap_items rings with R=5
pairs each. Before the fix, every incoming pair triggered a test_change
call that got rejected -- calc_pg_upmaps took ~11 minutes. After, it
exits in ~200ms.
Hu-Yuxuan [Mon, 9 Oct 2023 09:49:54 +0000 (17:49 +0800)]
osd/OSDMap: skip upmap-items drop when um_from would become equally overfull
Improves pg-upmap balancer convergence speed on clusters where many OSDs
are mutually overfull and linked by pg_upmap_items -- a common pattern
after a scale-up that increases replica count.
When try_drop_remap_overfull finds an incoming [um_from -> osd] pair on an
overfull osd, it called test_change to check whether dropping the pair would
improve distribution. If um_from carries the same excess load as osd,
though, dropping just returns the PG to an equally crowded OSD -- test_change
rejects the move anyway. With many such mutual pairs, the wasted calls add
up to minutes before the balancer gives up.
This adds an early check: skip test_change when um_from would end up at
least as overfull as osd after receiving the PG back. This eliminates
O(n_cohort * R) wasted test_change calls per balancer pass and reduces
calc_pg_upmaps from minutes to milliseconds in the affected scenario.
This commit removes centos9 from crimson's supported distros. This is in
line with the wider ceph moving on to rocky10 from centos9. We have
established that crimson is compatible with rocky10. More details can be
found in this tracker: https://tracker.ceph.com/issues/75823. Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
Ronen Friedman [Sat, 2 May 2026 15:53:49 +0000 (15:53 +0000)]
crimson/osd: defer snap trimming while scrubbing
Classic OSD enforces mutual exclusion between scrubbing and snap
trimming via the WaitScrub state in the snap trim state machine.
Crimson was missing this, allowing both to run concurrently on the
same PG (visible as active+clean+scrubbing+deep+snaptrim), which
could prevent snap trimming from completing within the expected
timeout.
Defer snap trim initiation while PG_STATE_SCRUBBING is set, and
re-trigger it from notify_scrub_end() via kick_snap_trim().
This is a temporary fix until the full scrub scheduling code,
including is_scrub_queued_or_active(), is merged.
Matty Williams [Fri, 1 May 2026 11:08:30 +0000 (12:08 +0100)]
osd: Use coroutines to perform synchronous reads in EC without blocking any threads
Coroutines.h file added containing type and struct definitions.
Coroutine spawned in PrimaryLogPG::do_op if the message contains a CEPH_OSD_OP_CALL op.
ECBackend::objects_read_sync yields while running ECBackend::objects_read_async. Coroutine is resumed once the read is complete.
When a coroutine op is being processed, all other ops to the same PG are blocked.
Fixes: https://tracker.ceph.com/issues/74531 Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Matty Williams [Fri, 1 May 2026 11:07:36 +0000 (12:07 +0100)]
osd: Adjust lock type selection in PrimaryLogPG::get_rw_locks
All writes in EC are asynchronous (even if they are only updating metadata), so Cls reads in EC must use the RWEXCL lock to eliminate the possibility of race conditions.
This resolves failures with Cls queue tests on EC pools where read operations were observed overtaking in-flight writes.
Fixes: https://tracker.ceph.com/issues/74531 Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
John Mulligan [Fri, 1 May 2026 19:14:34 +0000 (15:14 -0400)]
script: move llvm.sh logic to a function
Move llvm.sh logic to a function and call that function in a way that
lets newer systems skip using it and install llvm packages (that
are new enough) from the distro repositories.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 30 Apr 2026 19:56:27 +0000 (15:56 -0400)]
script: remove call to set ccache option run_second_cpp
Remove the call to ccache to set run_second_cpp=true because ccache
no longer supports this option. See
https://ccache.dev/releasenotes.html#_ccache_4_12
for more information.
My reading of the ccache release notes & documentation is that when this
option was available the default was true. This is what we were setting
so removing it should not change behavior for ccache, at least most
recent versions. Commit 4cb5a590537a9caaf61db42ce8ea123d2ab961f3 seems
to be the source of this line that sets run_second_cpp=true but that
change does not clearly say why it is being explicitly set, so I think
it's safe to remove.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 30 Apr 2026 19:45:23 +0000 (15:45 -0400)]
debian: remove xmlsec1 deps from control file
These deps should be included as part of the dependencies of the
python-onelogin-saml2 package. The names of the packages have changed
between jammy (22.04) and resolute (26.04) and it's not convenient (or
possible?) in debian packaging to have different dependencies for
different versions of the OS. Since they'll be pulled in as needed by
onelogin package, we can just remove these.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add a handler for Transaction::OP_MERGE_COLLECTION in SeaStore's
_do_transaction_step() and the corresponding _merge_collection()
implementation. Since coll_t is not part of the onode key, no
onode re-keying is needed: the operation updates the destination
collection's split_bits and removes the source collection from the
collection B-tree, all within a single transaction.
Add a handler for Transaction::OP_MERGE_COLLECTION in CyanStore's
do_transaction_no_callbacks() and the corresponding _merge_collection()
implementation. Moves all objects from the source collection into the
destination, updates the destination's split_bits, and removes the
source collection from the store's collection map.
crimson/os/seastore: handle enoent in SeaStore::Shard::stat
stat() aborts via assert_all when get_onode() returns ENOENT for a
non-existent object. But get_onode() explicitly declares enoent as a
valid error (onode_manager.h:36-40), and replicated_recovery_backend
calls stat() during push-based recovery on objects that haven't been
pulled yet.
Handle enoent by returning an empty struct stat — same pattern as
exists() (line 1207) and the !store_active early return (line 1384).
The recovery code in replicated_recovery_backend.cc:831-840 already
handles st_size=0 correctly.
Add a unit test that stats a non-existent object to verify it returns
an empty struct instead of aborting.
osd/SnapMapper::update_snaps() to handle a missing OBJ_ record
by falling back to add_oid() instead of silently creating an
inconsistent state (OBJ_ without matching SNA_ entries). This
was observed on replicas that had recently recovered objects:
the snap mapper entries created during recovery were not visible
to a subsequent snap-trim repop's update_snaps() call, leaving
the clone with no snap mapper entries. Scrub would then detect
and report the inconsistency as an error.
Promote snap mapper remove_oid/clear_snaps logging to dout(10)
and add apply_op_stats tracing to aid diagnosis of any remaining
stat or snap mapper drift.
Gil Bregman [Tue, 5 May 2026 08:53:25 +0000 (11:53 +0300)]
mgr/dashboard: Allow empty port value when adding a listener in NVMEoF CLI Fixes: https://tracker.ceph.com/issues/76410 Signed-off-by: Gil Bregman <gbregman@il.ibm.com>
doc: warn against default cephadm shell for ceph-volume
This commit documents that ceph-volume should not be run in a plain
cephadm shell session because default bind mounts are insufficient.
It describes the enriched -m example, how to retrieve the bootstrap-osd
keyring, and mentions that manual ceph-volume use outside ceph orch is
mainly for debug or dev.
Kushal Deb [Tue, 5 May 2026 05:30:12 +0000 (11:00 +0530)]
mgr/cephadm: allow nvmeof group assignment for NVMe-oF services
NVMe-oF services created in older releases may have a service_id that does
not include the gateway group name, because those services could be created
without a group.
The current validation requires an NVMe-oF service_id to end with
the configured group name when spec.group is set. This is correct for new
services, but it blocks the documented upgrade flow for existing legacy
services where the user exports the existing spec, adds spec.group, and
applies it back.
Allow this narrow update path when the service already exists, the stored
spec has no group, and the incoming spec keeps the same service_id while
adding a non-empty group. Keep the existing validation for new services and
for services that already have a group, so duplicate or misleading group
configurations are still rejected.
rgw/test: add Journal mode support to bucket logging test suite
Add --logging-type flag to run the Python bucket logging test suite
in either Standard or Journal mode. The same tests run against both
logging types with no changes to test logic or assertions.
- Add --logging-type pytest CLI option (Standard default, Journal opt-in)
- Detect boto3 LoggingType extension availability at session startup
- Thread logging_type through helpers and test functions
- Add teuthology task YAML for Journal mode suite runs
- Install service-2.sdk-extras.json in the teuthology task when
logging_type is Journal (s3tests cleans it up after its own run,
so the file isn't available by the time our Journal job runs)
- Document Journal mode local usage in the test suite README
rgw: fix assert crash in RGWDeleteMultiObj end_response()
remove the close_section() call from end_response() and rely
on output_footer() inside rgw_flush_formatter_and_reset() to close the
DeleteResult section safely.
Previously, applying oauth2-proxy without required configuration
resulted in misleading field-level validation errors.
This change introduces:
- Early validation for missing spec in CLI path
- Aggregated missing field validation in OAuth2ProxySpec
- Clear and actionable error messages
Fixes:
- ceph orch apply oauth2-proxy without spec
- ceph orch apply -i <yaml> with missing required fields
The LibRadosAio.PoolEIOFlag test was unstable, skip due to:
- Test Unreliability (Timing Dependency): The test used a fixed
iteration count for IO submission and finished too quickly, missing the
target error.
* Fix: The submission loop now runs continuously (time-bounded)
until the EIO error is reliably caught, eliminating the timing issue.
Nitzan Mordechai [Wed, 26 Nov 2025 14:36:42 +0000 (14:36 +0000)]
aio_cxx: Fix mutual deadlock in PoolEIOflag test
The LibRadosAio.PoolEIOFlag test was unstable, it was hanging due to:
- Deadlock: The main thread held the shared mutex ('my_lock') while
calling thread join. This created a mutual deadlock.
* Fix: Mutex is unlocked before thread join using RAII scopes.
- also convert to std::jthread and drop the join
seastore/omap_manager/btree: change omap manager funcs to coroutines
This commit changes funcs in BTree OMap manager to coroutines. Apart
from cleaner code that's easier to follow this is done to fix ASan
heap-use-after-free asserts.
Example QA job with the error: https://pulpito.ceph.com/shraddhaag-2026-04-20_07:04:25-crimson-rados-main-distro-debug-trial/164374/ Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
Dhairya Parmar [Thu, 6 Nov 2025 19:54:05 +0000 (01:24 +0530)]
src/vstart.sh: fix start_ganesha() to avoid crashing nfs-ganesha server
inclusion of "allow_set_io_flusher_fail = true" is needed otherwise the service won't start cause of EPERM,
referencing cephadm commit http://github.com/ceph/ceph/commit/3e3b7fa8ea22a2dd7f5d5c97d6e096a5e30585f4 which
does the same. Also, ensure double quotes around some config options like RecoveryBackend, some options in
RADOS_KV block and some in RADOS_URLS.
seastore/omap_manager/btree: change node insert/del funcs to coroutines
This commit changes OMapLeafNode and OMapInnerNode funcs to coroutines
to improve readability and prevent any ASan heap-use-after-free asserts. Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
* refs/pull/67536/head:
qa/multisite: enable the multisite test for oidc.
rgw/oidc: plumb RGWObjVersionTracker through load/store for race detection
rgw/oidc rados: add rgwrados::oidcs namespace abstraction for cls_user for accounts.
rgw/rest-oidc: Forward all oidc mutation request to master zone.
rgw/oidc: add rgwrados::oidc interface to support multisite.
mds: add retry request to MDSRank wait queue rather via finisher
C_MDS_RetryRequest inherits from MDSInternalContext which does not
acquire mds_lock by itself. Adding to MDSRank wait queue will process
this via the progress thread which completes the context with mds_lock
acquired.
Jon Bailey [Tue, 27 Jan 2026 16:59:28 +0000 (16:59 +0000)]
osd: Reduce the amount of status invalidations when rolling shards forwards during peering
Currently stats invalidations happen during peering when rolling forward shards.
We can reduce this so we only invalidate the stats when we don't have any other shards at the version we want to roll the stats forwards to.
In the cases where we have a shard with the stats at the correct version, we use those stats instead of invalidating.
If we do not have any shards with the correct version of stats, we do the invalidate as before.
We also add a check to ensure that we don't use stats from a non-primary shard if the head of the log is a partial write, to prevent incorrect stats in fast ec.
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>
Yonatan Zaken [Fri, 1 May 2026 08:41:46 +0000 (11:41 +0300)]
mgr/cephadm: redirect browser to correct port by identity provider
After authentication, the external identity provider was redirecting
to the correct dashboard address but omitting the external port causing
the browser to redirect to the default https port (443) which isn't
used since an external port was configured in mgmt-gateway spec file.
The nginx external_server_conf.j2 file was changed to use $http_host
instead of $host in order for the oauth2-proxy service to correctly
construct the dashboard URL including the non-standard port.
Matthew N. Heler [Wed, 17 Dec 2025 02:53:20 +0000 (20:53 -0600)]
qa/rgw: add teuthology support for target_by_bucket cloud transition
Add cloud_target_by_bucket and cloud_target_by_bucket_prefix options
to rgw_cloudtier.py and s3tests.py. Create new test suite to run
target_by_bucket-specific s3-tests.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
projects / ceph.git / log