Ilya Dryomov [Fri, 26 Jun 2026 10:56:27 +0000 (12:56 +0200)]
script/ptl-tool: fix redmine_linkage_correct case in _get_active_issues()
If redmine_linkage_correct is True, _get_active_issues() is supposed to
ignore issues with "Multiple Source PRs" category. However that doesn't
happen because the category string is compared against (category, text)
pair.
rbd-mirror: Remove old non-primary demoted image snapshots on the local cluster
When an image is demoted on primary cluster and later promoted again,
a non-primary demoted image snapshot is created on the peer (secondary)
cluster. Each such promote/demote cycle on same cluster results in an
additional non-primary demoted snapshot being created on peer.
As a result, repeated promote/demote cycles on the same cluster can
lead to accumulation of stale non-primary demoted snapshots on the
secondary cluster. These snapshots are not removed immediately because
cleanup is only triggered when peer (secondary) is promoted and then
demoted.
The proposed changes ensures that such snapshots are proactively identified
and removed, preventing unbounded buildup. Also added test coverage to
verify cleanup behavior.
Ronen Friedman [Thu, 11 Jun 2026 12:32:34 +0000 (12:32 +0000)]
crimson/osd: return ENOENT from EC getxattr for non-existent objects
PGBackend::getxattr() for EC pools reads from the attr_cache and
returns ENODATA when an attr is not found, without distinguishing
between "object exists but attr not set" and "object doesn't exist".
This causes cls methods like cls_refcount_get() to proceed as if the
object exists, enter the EC RMW pipeline, and hang.
Add an os.exists check in the EC path so non-existent objects return
ENOENT, matching the behavior of the replicated path which goes to
the object store and gets ENOENT directly.
Do not mark an upgrade complete until daemons in the filtered scope
match the target image. Limited (--limit) upgrades still complete
when remaining_count reaches zero.
Sun Yuechi [Tue, 23 Jun 2026 06:16:38 +0000 (14:16 +0800)]
test/bufferlist: shrink benchmark iterations under ASan
The pure-timing *_bench/BenchAlloc loops repeat a stateless path millions of
times. Under ASan that adds no memory-error coverage but inflates the run from
minutes to hours, so cut the iteration counts when built with ASan.
Sun Yuechi [Tue, 23 Jun 2026 10:18:25 +0000 (18:18 +0800)]
test/bufferlist: avoid BenchDeref iterator overrun past end
Drive the deref loop by get_remaining() >= step instead of comparing to
end(), so a total length that is not a multiple of step no longer lets
`iter += step` run past the end and throw "End of buffer".
The original counts were always a multiple of step, so the overrun only
showed up once the following commit shrinks the rounds under ASan -- a
false alarm rather than a real regression, which is why it went unnoticed
until now.
Collect FCM stats locally from NVMe drives (vendor log page 0xCA)
and expose them via node-proxy, the cephadm agent, and
`ceph orch hardware status --category fcm`.
Introduce a node backend that aggregates Redfish data with
node-local collectors, since FCM metrics are not available
from the BMC.
Afreen Misbah [Tue, 23 Jun 2026 13:30:22 +0000 (19:00 +0530)]
mgr/dashboard: fix bind address regression from CherryPy isolation
The CherryPy isolation refactor (PR #67227) accidentally changed the
dashboard bind address from wildcard (*:8443) to mon_ip:8443. The
get_mgr_ip() replacement was originally only for URI generation, but
the refactor passed the mutated address to CherryPyMgr.mount() as the
actual socket bind address.
This breaks the management gateway when its VIP is not on the same
interface as mon_ip, as the dashboard becomes unreachable on other
interfaces.
Preserve the original wildcard address for binding and only use
get_mgr_ip() for the advertised URI. Add regression test to prevent
future confusion between bind_addr and server_addr.
The python binaries on some CI images and dev boxes ship stripped, so the
allocator frames in an interpreter-shutdown leak come through unsymbolised
(/usr/bin/python3.13+0x...) and the function-name matches above cannot apply.
leak:python3.10 already handled this for 3.10, and a stale comment claimed 3.12
does not leak.
Add leak:python3.12 and leak:python3.13, mirroring the 3.10 entry, so the
interpreter globals are suppressed whatever CPython the build uses.
Kefu Chai [Tue, 23 Jun 2026 08:14:19 +0000 (16:14 +0800)]
ceph.in: load asan/lsan suppressions on WITH_ASAN builds
bin/ceph from a WITH_ASAN build aborts at exit, with LeakSanitizer reporting
CPython and Cython module-init allocations as leaks:
==2577940==ERROR: LeakSanitizer: detected memory leaks
Direct leak ... in PyObject_Malloc (/usr/bin/python3.12+...)
#4 __pyx_pymod_exec_rados rados_processed.c
SUMMARY: AddressSanitizer: 32113 byte(s) leaked in 30 allocation(s).
These are interpreter globals that live for the process lifetime. qa/lsan.supp
already suppresses them, but bin/ceph never loaded it: vstart.sh sets
LSAN_OPTIONS for the daemons it spawns, while a bin/ceph invoked separately
(ceph-api runs ./bin/ceph fsid once vstart.sh returns) inherits none and exits
non-zero. It stayed hidden until radosgw-admin stopped crashing in vstart and
the run reached that call.
ceph.in already re-execs with the ASan runtime preloaded under WITH_ASAN. Set
ASAN_OPTIONS and LSAN_OPTIONS first, from the CEPH_ASAN_OPTIONS and
CEPH_LSAN_OPTIONS that CMake also feeds add_ceph_test(), so the re-exec'd
interpreter starts with the suppressions loaded. Use setdefault so a value
from the caller still wins.
Kefu Chai [Tue, 23 Jun 2026 08:14:19 +0000 (16:14 +0800)]
cmake: factor the ASan/LSan test options into cache variables
add_ceph_test() spelled out the suppression-file paths and sanitizer flags
inline. bin/ceph needs the same options, so lift them into CEPH_ASAN_OPTIONS
and CEPH_LSAN_OPTIONS and have add_ceph_test() consume those. The environment
the tests run with is unchanged.
David Galloway [Tue, 23 Jun 2026 14:11:34 +0000 (10:11 -0400)]
.github/workflows/releng-audit.yaml: Check main for upstream commit
This check was failing to correctly identify whether backported commits
had actually been cherry-picked because it was checking the target branch
instead of the main branch. This change checks out the main branch
to look for the upstream cherry-picked commit.
Signed-off-by: David Galloway <david.galloway@ibm.com>
Kotresh HR [Tue, 23 Jun 2026 15:13:18 +0000 (20:43 +0530)]
qa: Add mgr snapshot mirror status tests
Add teuthology coverage for `ceph fs snapshot mirror status`:
parity with asok peer_status, default idle metrics, stale omap
handling, error paths, filter scopes, daemon restart, and cache TTL.
Reuse existing peer_dir_status and metrics assertion helpers.
Adjust stale test wait for InstanceWatcher timeout and set a
short cache TTL in the cache test. Pass peer_uuid via --peer_uuid=
in the test helper for peer-only scope queries.
Kotresh HR [Sun, 21 Jun 2026 17:59:43 +0000 (23:29 +0530)]
doc/cephfs: Document fs snapshot mirror status mgr command
Describe the mirroring module command that reads persisted omap
metrics, including syntax, output layout, stale detection, caching,
and comparison with the admin socket peer status interface.
Document --peer_uuid as a named argument for peer-only and
directory/peer filtering.
Kotresh HR [Sun, 21 Jun 2026 17:52:29 +0000 (23:22 +0530)]
mgr/mirroring: make snapshot mirror metrics cache optional
Add snapshot_mirror_metrics_cache_enabled (default true). When disabled,
metrics_status reads omap directly and skips complete and partial caches.
When enabled, behavior is unchanged.
Kotresh HR [Sun, 21 Jun 2026 17:50:28 +0000 (23:20 +0530)]
mgr/mirroring: make snapshot mirror metrics cache TTL configurable
Add snapshot_mirror_metrics_cache_ttl as a runtime mgr/mirroring module
option (default 15 seconds) instead of a hard-coded CACHE_TTL_SECS.
Both complete and partial lru_cache_timeout wrappers read the value
when caching omap metrics so operators can tune cache freshness without
code changes.
Kotresh HR [Sun, 21 Jun 2026 17:48:15 +0000 (23:18 +0530)]
mgr/mirroring: handle missing cephfs_mirror object in metrics status
When snapshot mirroring is not enabled for a filesystem, the
cephfs_mirror RADOS object does not exist and "ceph fs snapshot mirror
status" fails reading sync stat omap. Map rados ENOENT to a clear
MirrorException so the CLI reports that snapshot mirroring must be
enabled instead of a generic omap read failure or an uncaught exception.
Also catch unexpected errors in metrics_status like other mirror CLI
handlers, so the mgr module does not crash on failure. Return
-errno.EINVAL from the generic error path instead of the exception
message as exit code.
Kotresh HR [Sun, 21 Jun 2026 17:28:38 +0000 (22:58 +0530)]
mgr/mirroring: detect stale snapshot mirror sync metrics in omap
Persisted metrics in the cephfs_mirror omap can outlive the writing
daemon when cephfs-mirror stops or a directory is reshuffled to another
instance; mgr would keep reporting stale progress until the owning
daemon writes again.
Extend format_and_order_sync_stat_for_display() to compare persisted
_instance_id against InstanceWatcher live instances (via
FSPolicy.get_live_instance_ids()) and the directory's tracked instance
(via Policy.get_tracked_instance_id()). Mark metrics stale when the
persisted writer is no longer live (any state), or when it does not
match the tracked instance while persisted state is not "idle". Show
state "stale" with current_syncing_snap omitted.
Pass policy and live instance ids through load_sync_stat_metrics() and
fetch_sync_stat_metrics(), and into sync_stat_complete_cache and
sync_stat_partial_cache loaders. Cache hits serve already-formatted
(stale-marked) entries until TTL expiry without re-checking instance
liveness.
Kotresh HR [Tue, 16 Jun 2026 10:42:41 +0000 (16:12 +0530)]
mgr/mirroring: cache fs snapshot mirror status omap metrics
Add a short-lived in-memory cache for metrics returned by
"ceph fs snapshot mirror status", which reads persisted sync stats
from the cephfs_mirror object omap. Omap walks are relatively
expensive; caching reduces repeated reads when the CLI or multiple
clients poll at short intervals.
Two TTL-bucketed LRU caches (CACHE_TTL_SECS) are used instead of one
unified cache. The complete cache always holds every mirrored
directory and peer for a filesystem; the partial cache holds one
directory. A single directory-granularity cache cannot prove it has
all directories, so full-scan queries rely on the complete cache
contract.
Cache implementation (lru_cache_timeout decorator backed by
_TimedLRUCache, not functools.lru_cache):
- lru_cache has no TTL and no peek-on-hit API. Single-directory
queries must read the complete cache without loading on miss;
lru_cache.cache is also unavailable on Python 3.14+.
- complete (sync_stat_complete_cache): full omap prefix scan via
load_sync_stat_metrics. Cache key: (time_token, filesystem).
COMPLETE_CACHE_MAX limits filesystem entries per TTL window.
Bind cache_peek/cache_info/cache_clear via a CachedMethod descriptor
so TTL lookups receive (self, ...); otherwise single-dir status fails
with 'str' object has no attribute 'mgr'.
Serve logic (under the existing lock):
- status <fs> [--peer_uuid=<uuid>]: load complete cache on miss;
filter by peer when requested.
- status <fs> <dir>: peek complete cache (no load on miss); if the
entry contains the directory and peers, serve from complete.
Otherwise load partial cache (omap on miss).
Kotresh HR [Mon, 1 Jun 2026 09:29:31 +0000 (14:59 +0530)]
mgr/mirroring: Reorder and format metrics output
Reorder and format status output to match the output of asok
interface peer_status command. Return metrics under
metrics/<dir>/peer/<uuid> to match the asok peer_status layout,
including {"metrics": {}} when no peers are configured.
mgr/mirroring: Add new interface to expose mirroring metrics
Add the following new interface to expose mirroring metrics
ceph fs snapshot mirror status <fsname> [<mirrored_dir_path>] [--peer_uuid=<peer_uuid>]
The cmd loads the persisted directory sync metrics from the
cephfs_mirror object's omap. Metrics are grouped by mirrored
directory and peer.
When --peer_uuid is specified, only metrics for that peer are
returned. peer_uuid is a named CLI argument (_end_positional_) so
peer-only filtering does not require a mirrored directory path.
Kotresh HR [Sat, 20 Jun 2026 07:58:56 +0000 (13:28 +0530)]
tools/cephfs_mirror: Remove persisted dir stats
When a directory is removed from mirroring, the persisted directory
stats need to be removed. This patch handles the cleanup.
Omap keys must not be removed when mirrored directories are reshuffled
across cephfs-mirror daemons. The mgr release notify now carries a
purging flag (set only during permanent removal, not reshuffle), and
the daemon removes persisted stats only when purging is true. On
reshuffle with an in-progress sync, clear live current_syncing_snap
state and persist idle metrics so the acquiring daemon does not inherit
stale syncing omap entries.
Kotresh HR [Sat, 20 Jun 2026 06:21:42 +0000 (11:51 +0530)]
tools/cephfs_mirror: Load persisted mirror metrics from omap
Load last_synced_snap metadata from the cephfs_mirror object omap on
PeerReplayer initialization and when a mirrored directory is added.
Live current_syncing_snap metrics are not restored; they are rebuilt
when synchronization starts.
When the daemon restarts after a snapshot was synced on the remote but
metrics were not yet written to omap, loaded metadata may belong to an
older snapshot. Add reconcile_last_synced_snap() to compare against
the remote snap map, clear stale last-sync fields, and update
last_synced_snap id/name in memory.
Treat snaps_synced, snaps_deleted, and snaps_renamed as per-session
counters. Do not load them from omap; they start at zero for each
daemon session and are still reported via the admin socket. Persist
omap metrics unconditionally after reconcile so the mgr picks up the
new instance id and cleared session counters on restart.
Kotresh HR [Sat, 20 Jun 2026 05:33:49 +0000 (11:03 +0530)]
tools/cephfs_mirror: Persist metrics to omap
Persist snapshot mirroring metrics to the cephfs_mirror object omap
for the mgr/mirroring module to support status command.
The tick thread only keeps omap up to date for in-progress syncs on
registered directories. Persist explicitly when stats change so omap
is updated before a directory unregisters:
- snap delete/rename propagation (inc_deleted_snap / inc_renamed_snap)
- after each successful snap sync (following set_last_synced_stat)
- after sync_snaps failure (following _inc_failed_count)
- after sync_perms failure (following _inc_failed_count)
Without the explicit call sites, omap can keep stale live or idle state
after sync completes or fails because directories leave m_registered
before the next tick.
Kotresh HR [Sun, 17 May 2026 21:01:58 +0000 (02:31 +0530)]
tools/cephfs_mirror: Adds capability to persist metrics
Adds the capability to persist mirroring metrics.
The metrics are persisted in the omap of the cephfs-mirror
object. Metrics are persisted asynchronously.
Each mirrored directory path stores the corresponding
metrics as the value of a unique omap key representing
the mirrored directory. The omap key is as below.
Kefu Chai [Tue, 23 Jun 2026 12:14:10 +0000 (20:14 +0800)]
Revert "script/run-make: enable ASan"
in 8ac962c6, we enabled ASan in script/run-make, which is part of
our CI workflow to build the tree. The goal was to enable us to identify
memory related issues early, but this introduced two kinds of problems:
- it's observed that some tests take around 4x time to complete in
comparison to the test time without ASan enabled
- api and dashboard e2e tests are failing because the ASan supppression
rules are not populated to them
rgw: restore constant-time GCM tag comparison in ISA-L path
a8ed43bfc05 replaced ct_memeq with memcmp in the ISA-L GCM accelerator,
making tag verification and the key-cache compare non-constant-time.
Restore ct_memeq for both; the OpenSSL and EVP paths already compare in
constant time.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Matthew Heler [Fri, 5 Jun 2026 15:48:32 +0000 (10:48 -0500)]
rgw: fix AES-256-GCM key/IV reuse on multipart part re-upload
Re-uploading the same part number in a GCM multipart upload encrypted the new
data under the same key and IV as the first upload, since the IV is
part_number||chunk_index and the part key came from the part number alone. GCM
requires a unique IV per key; reusing one to encrypt different data weakens its
confidentiality and integrity guarantees.
Generate a random 16-byte salt on each UploadPart and fold it into the part key,
HMAC(ObjectKey, BE32(part) || salt), so every upload gets a fresh key. The salt
rides RGWUploadPartInfo, and complete stores the selected part's salt in
RGW_ATTR_CRYPT_PART_NUMS, which now holds (part, salt) pairs. GET reads it back
to re-derive the key, and an empty salt reproduces the old derivation so unsalted
parts still decrypt.
Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
Gil Bregman [Mon, 22 Jun 2026 09:06:00 +0000 (12:06 +0300)]
mgr/cephadm: Add degraded namespace flag to NVMEoF spec file Fixes: https://tracker.ceph.com/issues/77556 Signed-off-by: Gil Bregman <gbregman@il.ibm.com>
Emmanuel Ameh [Tue, 23 Jun 2026 10:23:14 +0000 (11:23 +0100)]
doc/cephfs: clarify deprecated commands and remove obsolete upgrade path
Several CephFS pages described deprecated commands and features without
clear status or replacements, and one page documented an upgrade path that
predates every supported release.
* cephfs-mirroring: correct the peer_add deprecation note to reference the
actual replacement command, peer_bootstrap create, and link the Bootstrap
Peers section
* kernel-features, experimental-features: state that inline data has been
deprecated since the Octopus release and that enabling it triggers a
health warning
* upgrading: remove the "Upgrading pre-Firefly file systems past Jewel"
section; the tmap_upgrade command it referenced was removed in Kraken
Emmanuel Ameh [Tue, 23 Jun 2026 09:34:33 +0000 (10:34 +0100)]
doc/rados/operations: refine cache-tiering deprecation note per review
Reduce the note to point existing users to the removal procedure, add a
ref target for the removal section, and rephrase the dm-cache mention to
reflect community adoption without implying official endorsement.
Kefu Chai [Tue, 23 Jun 2026 07:43:28 +0000 (15:43 +0800)]
mgr/dashboard: skip the table when an nvmeof cli result has no columns
The dashboard leaves prettytable unpinned. prettytable commit 2574492 ("Apply
some Pylint rules (PLR)", #436) rewrote _stringify_row()'s row_height as
`max(_get_size(c)[1] for c in row)`, which raises ValueError("max() iterable
argument is empty") on a row with no cells. The change is undocumented and
shipped in 3.18.0; get_string() trips on it when a table has a row but no
columns.
AnnotatedDataTextOutputFormatter builds such a table for an empty result, or
one whose only field is status or error_message, so NvmeofCLICommand.call()
returns -EINVAL and the command fails. This broke run-tox-mgr-dashboard-py3
once the tox virtualenv picked up prettytable 3.18.0.
Return an empty string when there are no columns instead of formatting a
degenerate table.
Sun Yuechi [Mon, 22 Jun 2026 17:38:54 +0000 (01:38 +0800)]
test/encoding: disable LeakSanitizer in readable.sh and check-generated.sh
readable.sh forks two short-lived ceph-dencoder processes per corpus
object(tens of thousands of processes overall). With ASAN's
LeakSanitizer enabled each process runs a whole-heap leak check at exit
that dominates runtime (~10s vs ~1s per process, measured on riscv64).
This test only validates encode/decode behaviour, not leaks, so disable
leak detection while keeping ASAN's other checks active.
check-generated.sh forks ceph-dencoder the same way, so apply the same
fix there.
Ilya Dryomov [Mon, 22 Jun 2026 16:58:47 +0000 (18:58 +0200)]
Merge pull request #69641 from ceph/ljflores-patch-1
doc: update email address for Laura Flores
Reviewed-by: Gregory Farnum <gfarnum@redhat.com> Reviewed-by: Dan van der Ster <dan.vanderster@clyso.com> Reviewed-by: Casey Bodley <cbodley@redhat.com>
projects / ceph.git / log