cephfs-tool: wait for async callbacks before freeing contexts
bench_async_write_worker destroyed AsyncIOContext while the finisher
still ran async_io_callback after setting completed. Track in-flight
callbacks per worker and drain before tearing down the io_queue.
Edwin Rodriguez [Mon, 22 Jun 2026 15:56:54 +0000 (11:56 -0400)]
cephfs-tool: Add support for `--async-write-fsync` and `--shared-file` options
Add --async-write-fsync and --shared-file benchmark options.
--async-write-fsync sets ceph_ll_io_info.fsync on each async write to model
Ganesha-style write+commit and skips the post-phase ceph_ll_fsync when enabled.
--shared-file directs all worker threads at one file to stress client_lock
contention on a single inode. Both options are recorded in JSON/text output.
Edwin Rodriguez [Thu, 18 Jun 2026 14:14:43 +0000 (10:14 -0400)]
cephfs-tool: Add filesystem statistics reporting before and after benchmark runs
Added filesystem stats reporting to cephfs-tool.cc.
Helper functions
• fetch_statvfs() — uses ceph_ll_statfs on the root inode when --async-io
is set, otherwise ceph_statfs on mount_root
• report_fs_stats() — prints and optionally JSON-encodes:
• Total / free blocks
• File count
When it runs
• Before the iteration loop (after the bench directory is created)
• After all iterations finish (before the final report and optional cleanup)
Edwin Rodriguez [Thu, 21 May 2026 13:21:32 +0000 (09:21 -0400)]
cephfs-tool: set thread names for read/write workers
Set pthread names on all benchmark worker threads (wr-worker-N, rd-worker-N)
via ceph_pthread_setname so they are visible in top, perf, and lockstat
output during profiling.
Edwin Rodriguez [Thu, 28 May 2026 13:50:40 +0000 (09:50 -0400)]
cephfs-tool: add async I/O support with configurable queue depth
Add --async-io to use ceph_ll_nonblocking_readv_writev for read and write
workers instead of synchronous ceph_read/ceph_write. Add --queue-depth to
limit outstanding async I/Os per thread via a counting semaphore and
pre-allocated AsyncIOContext pool. Async write uses ceph_ll_create; async
read uses ceph_ll_walk + ceph_ll_open and defers inode release until all
async ops including readahead complete. Reject --async-io with
--per-thread-mount since async workers require a shared mount.
Edwin Rodriguez [Thu, 14 May 2026 16:11:00 +0000 (12:11 -0400)]
cephfs-tool: add lockstat collection and dumping support
When built with CEPH_LOCKSTAT, add --lockstat-dump and --lockstat-threshold
to collect lock contention data during benchmark phases. Start collection
at benchmark begin, reset counters before each read/write phase, and dump
per-iteration/per-phase JSON files (e.g. base_iter1_write.json). Requires
a lockstat-instrumented libcephfs build.
Edwin Rodriguez [Thu, 7 May 2026 13:36:51 +0000 (09:36 -0400)]
cephfs-tool: Add support for overriding 'ms_async_op_threads' configuration
Add --msgr-workers to override ms_async_op_threads before mount (1-24;
0 keeps the ceph.conf default). Controls async messenger I/O threads
independently of benchmark worker thread count. Validate the range at
startup and include the resolved value in console and JSON output.
cephfs-tool: Add progress tracking support for benchmarking phases
Add --progress and --progress-interval to display live bandwidth, IOPS,
and ETA during read/write phases. A background progress_reporter thread
aggregates per-thread stats every 100ms and prints a single updating
status line. Progress percentage is based on elapsed time when --duration
is set, otherwise on files completed. Make ThreadStats fields atomic so
workers and the reporter can access them concurrently.
cephfs-tool: Add support for overriding the 'client_oc' options
Add --client-oc and --client-oc-size to override libcephfs object cacher
settings before ceph_init. Enables benchmarking with the object cacher
enabled or disabled and with a configurable cache size. Resolved values
are shown in console and JSON output.
cephfs-tool: Add support for dumping performance counters
Add --perf-dump to write libcephfs client performance counters to a file
at the end of the benchmark. Uses ceph_get_perf_counters() while the
mount is still active so counters reflect the full benchmark run.
cephfs-tool: Add duration-based phase limit for benchmarking
Add --duration to cap each read and write phase at N seconds instead of
running until all files are processed. Workers loop until the time limit
expires, cycling through their per-thread file set in duration mode.
Pass phase start time into worker threads for consistent cross-thread
duration checks. Only count each unique file once in stats even when
filenames are reused across cycles.
cephfs-tool: Add JSON output support for benchmarking results
Add --json to write structured benchmark results (configuration,
per-iteration read/write metrics, and cross-iteration summary statistics)
to a file using JSONFormatter. Extend print_statistics() to optionally
mirror mean, stddev, min, and max into the JSON document. Improve
parse_size() to accept both SI (1000-based, e.g. "4MB") and binary
(1024-based, e.g. "4MiB") size suffixes.
* refs/pull/66558/head:
qa/cephfs: minor fix in comment
qa/cephfs: give more time to tests in test_clone_stats.py
qa/cephfs: increase number of files to cloned in test_clone_stats.py
volumes/stats_util: improve log messages
Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Jos Collin <jcollin@redhat.com>
Igor Golikov [Wed, 27 May 2026 13:19:25 +0000 (16:19 +0300)]
client: fix Dentry UAF by holding DentryRef across unlink
Hold a function-scoped DentryRef in Client::unlink() to keep the dentry
alive through the entire teardown sequence.
Without the O_DIRECTORY dentry pin (PR #60909, merged after v19.2.1),
opening a directory does not call _ll_get(), so the dentry stays at
ref=1 while inode->dir is still active. When trim_cache() evicts this
dentry, Dentry::unlink() calls put() for the dir pin, ref drops to 0,
and the dentry is freed while Client::unlink() still needs it for
detach/lru_remove.
A function-scoped DentryRef guarantees the dentry survives until after
it has been properly removed from the dir and LRU, regardless of pin
state.
Signed-off-by: Igor Golikov <igolikov@ibm.com> Fixes: https://tracker.ceph.com/issues/74625
Igor Golikov [Wed, 27 May 2026 13:18:44 +0000 (16:18 +0300)]
client: add assert to catch dentry ref drop during unlink
Add ceph_assert(dn->ref > 0) after Dentry::unlink() in Client::unlink()
to catch the case where internal put() calls drop the dentry reference
to zero before we proceed to detach/lru_remove.
This turns a silent use-after-free into a clean assertion failure with
a full stack trace, making it possible to catch the exact conditions
in a test environment.
Signed-off-by: Igor Golikov <igolikov@ibm.com> Fixes: https://tracker.ceph.com/issues/74625
Anthony M [Tue, 30 Jun 2026 22:11:39 +0000 (17:11 -0500)]
doc/foundation: Removed members section
These updates will allow us to update a single page rather than two. The detailed member lists and categories have been removed. I've added a redirect for visitors to this page to https://ceph.io/en/foundation/members/
Signed-off-by: Anthony M <amiddleton@linuxfoundation.org>
Sun Yuechi [Tue, 30 Jun 2026 11:50:09 +0000 (19:50 +0800)]
librbd: fix use-after-free in trash purge on image open error
When ImageState::open() fails with an error other than -ENOENT, it
asynchronously deletes the ImageCtx before returning. The trash purge
loop only skipped the -ENOENT case and fell through on every other
error, dereferencing the already-freed ictx in diff_iterate() and
state->close(). Add the missing continue so a failed open is skipped.
The fall-through dates back to 504f4e78, which split `if (r < 0)
continue` to log non-ENOENT errors but dropped the continue.
Fixes: https://tracker.ceph.com/issues/77836 Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
Sun Yuechi [Tue, 30 Jun 2026 03:50:20 +0000 (20:50 -0700)]
blk/aio: fix mismatched parenthesis in POSIX AIO submit path
The condition `if ((cur->n_aiocb == 1) {` has an unbalanced
parenthesis and fails to compile. This code is only built on the
HAVE_POSIXAIO (BSD) path, so the breakage is normally hidden.
Xuehan Xu [Fri, 5 Jun 2026 09:19:13 +0000 (17:19 +0800)]
crimson/os/seastore/journal/circular_bounded_journal: find the real
journal tail on startup
Recover the correct journal tail during replay by scanning
JOURNAL_TAIL records instead of relying solely on the on-disk
header, which may be stale if a crash occurs between record
write and header update.
The journal header is stored *separately* from the (append-only)
journal and is not updated atomically with journal writes,
so it may lag behind after a crash.
Treat the journal as the source of truth and rebuild the tail from persisted records.
Replay is reorganized into three passes:
1) discover latest journal tail
2) rebuild allocation map
3) replay deltas
Kefu Chai [Mon, 29 Jun 2026 01:19:02 +0000 (09:19 +0800)]
crimson/test: stop the messenger-thrash config on every exit path
test_messenger_thrash put sharded_conf().stop() in the success
continuation, so an exception from the tests skipped it and left the
static sharded<ConfigProxy> holding live instances. its destructor then
asserts. a tester hit this when the run went OOM at --memory 256M:
ERROR [shard 0:main] test - Test failed: got exception
ceph::buffer::v15_2_0::bad_alloc (Bad allocation [buffer:1])
/ceph/src/seastar/include/seastar/core/sharded.hh:573:
seastar::sharded<Service>::~sharded()
[with Service = crimson::common::ConfigProxy]:
Assertion `_instances.empty()' failed.
terminate called without an active exception
run the body in seastar::async and stop the config through
seastar::deferred_stop, as crimson/osd/main.cc already does. the guard
is constructed only after start() succeeds, and its destructor stops the
config on any exit path, so a failing test exits cleanly instead of
aborting.
Ronen Friedman [Sun, 28 Jun 2026 14:42:23 +0000 (14:42 +0000)]
cmake: add WITH_MOLD option for Mold linker support
Add a WITH_MOLD cmake option that auto-configures the build for
the Mold linker. When enabled, cmake finds mold, sets CMAKE_LINKER,
injects -fuse-ld=mold into all linker flag variables, and sets
USING_MOLD_LINKER which gates the Mold-specific workarounds added
in the previous commit.
Usage: cmake -DWITH_MOLD=ON ...
Assisted-by: Claude <claude.ai> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Ronen Friedman [Sun, 28 Jun 2026 14:42:07 +0000 (14:42 +0000)]
build: add Mold linker compatibility
Mold handles several linker features differently from ld.bfd:
- --exclude-libs,ALL hides .symver-aliased symbols (e.g. rados_*)
even when the version script lists them as global. Detect Mold
(via USING_MOLD_LINKER) and disable --exclude-libs; version
scripts already control symbol visibility.
- Duplicate symbols across static archives are treated as errors.
Merge crimson-alien-common into crimson-alienstore as a single
archive. For crimson-osd, link crimson-alienstore before
crimson-common with --allow-multiple-definition so the alien
thread's non-crimson definitions win (Mold picks first definition).
- Python/Cython extension builds (via Distutils.cmake) invoked the
system default linker. Pass -fuse-ld= via MOLD_FUSE_LD_FLAG so
extensions link with the configured linker.
- Add rados_* and _rados_* to librados.map global section for Mold
compatibility with .symver aliases.
Co-authored-by: Mark Kogan <mkogan@redhat.com> Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Yael Azulay [Thu, 21 May 2026 11:38:28 +0000 (14:38 +0300)]
cephadm: keep mgr ports in sync when modules are enabled/disabled
ceph orch ps - was showing stale or incorrect port information for `mgr` daemons after enabling or disabling mgr modules (e.g., dashboard, prometheus).
This fix ensures the displayed ports always reflect the actual active module endpoints.
Code Fixes:
-----------
Changes in src/pybind/mgr/cephadm/services/cephadmservice.py:
- Extract and refactor _get_mgr_service_ports:
The logic for reading active mgr module ports was inline inside prepare_create, using check_mon_command({'prefix': 'mgr services'}). It was extracted into a static helper method and refactored to read directly from mgr.get('mgr_map')['services'] instead -- which is faster and consistent with how get_dependencies reads the same data. The method is now shared by both prepare_create (existing use) and get_dependencies (new use).
- Extract `_get_mgr_service_ports`:
The logic for reading active mgr module ports from `mgr_map['services']` was inline inside `prepare_create`. It was extracted into a static helper method so it can be reused by both `prepare_create` (existing use) and `get_dependencies` (new use). Returns the port numbers currently registered by active mgr modules (e.g., `[8443, 9283]`).
- Override `get_dependencies` in `MgrService`:
The base class returns `[]` for all Ceph daemons. The serve loop compares `get_dependencies()` output against `last_deps` (saved after the previous reconfig). If they differ, a reconfig is triggered. By overriding this method in `MgrService`, we make the serve loop aware of mgr port changes. Returns a sorted list such as `['port:8443', 'port:9283', 'sd_port:8765']`.
- Override `generate_config` in `MgrService`:
After a reconfig completes, the result of `generate_config` is saved as `last_deps`. The inherited implementation returns `[]` as deps. If `[]` is saved but `get_dependencies` returns `['port:8443', 'sd_port:8765']`, they would always differ, causing an infinite reconfig loop. This override ensures that what is saved as `last_deps` matches what `get_dependencies` will return in the next iteration.
Changes in `src/pybind/mgr/cephadm/serve.py`:
- Update mgr cache on reconfig:
The existing logic skipped the cache update for all Ceph daemon reconfigs. A new `elif` branch for mgr reconfigs fetches the existing cached `DaemonDescription`, updates only its `ports` field, and saves it back. This makes `ceph orch ps` reflect the correct ports immediately after a reconfig, without showing a misleading "starting" status.
Changes in `src/cephadm/cephadm.py`:
- Update `unit.meta` on reconfig:
`unit.meta` is a per-daemon file on the host that stores metadata including ports. It is read periodically by `cephadm ls` and fed back into the orchestrator cache. Previously it was only written on initial deployment, so every periodic cache refresh would overwrite the corrected ports with stale data from `unit.meta`. This change updates the `ports` field in `unit.meta` during reconfig, ensuring the cache refresh confirms the correct state rather than reverting it.
Changes to src/pybind/mgr/cephadm/tests/services/test_mgr.py:
- Updated _prepare helper:
The original mocked check_mon_command to feed services data to prepare_create. Since our _get_mgr_service_ports now reads from mgr.get('mgr_map') instead of check_mon_command, the mock was replaced with mock_store_set('_ceph_get', 'mgr_map', ...) which sets the data in the right place.
- Added test_get_dependencies_changes_when_module_enabled:
A new test that verifies get_dependencies returns different dep strings before and after a module is enabled. This directly tests the core sync mechanism -- the serve loop compares get_dependencies output against last_deps to detect port changes and trigger a reconfig.
to run new tests:
cd src/pybind/mgr
tox -e py3 -- cephadm/tests/services/test_mgr.py -v
Ilya Dryomov [Fri, 26 Jun 2026 10:56:27 +0000 (12:56 +0200)]
script/ptl-tool: fix redmine_linkage_correct case in _get_active_issues()
If redmine_linkage_correct is True, _get_active_issues() is supposed to
ignore issues with "Multiple Source PRs" category. However that doesn't
happen because the category string is compared against (category, text)
pair.
Sun Yuechi [Thu, 25 Jun 2026 11:50:36 +0000 (19:50 +0800)]
cmake: build fio from upstream 3.42
fio-3.42 is the latest stable release. We switched to the ceph/fio fork in 10baab3fc8293b8c30ca90a4acd76f70d011f1b5 to pick up a clang build fix; that
fix has since landed upstream, so switch back. This also gives riscv64
proper arch support (arch-riscv64.h, added in fio-3.36), which the fork's
fio-3.27-cxx lacks.
projects / ceph.git / log