Karthik U S [Sat, 16 May 2026 03:15:50 +0000 (08:45 +0530)]
qa/test_mirroring: Add tests for mirroring checkpoints
Adding integration tests for validating the cephfs mirroring
checkpoints feature
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit fe7fa7b0761720181cf8735ac7b5f17080c1d8a7)
Karthik U S [Thu, 4 Jun 2026 11:47:05 +0000 (17:17 +0530)]
doc: Add docs and pending release notes for mirroring checkpoints
Adding documentations and pending release notes for the mirroring
checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 12bd34d69ac7e136d14d25ea274e99727b559870)
Karthik U S [Wed, 10 Jun 2026 05:48:25 +0000 (11:18 +0530)]
mgr/mirroring,tools/cephfs_mirror: Handle checkpoint state transition
When a new checkpoint is being added or when the daemon gets restarted,
it will check whether the newly created checkpoint or any other old
checkpoints have already been mirrored onto the remote peer. If so, it
will transition to the correct state by checking for the highest snap id
present on the remote, and setting all the checkpoints which have snap id
lesser than or equal to that of the remote to COMPLETE. This is done by
sending an acquire notification from the mirroring module to the mirror
daemon, which is handled in the add_directory path, by adding the directory
to be checked for the state transition in the tick thread.
This path gets triggered:
a) when the daemon gets restarted
b) when the peer mapping changes
c) by sending the acquire notification from checkpoint add/now CLIs.
d) when mirroring module restarts
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit c64fe9636514d9e9b1c286a0b21b84569e266067)
Karthik U S [Sat, 16 May 2026 00:50:03 +0000 (06:20 +0530)]
tools/cephfs_mirror: update checkpoint status during snapshot sync
When the sync completes or fails for a checkpointed snapshot, transition
the status of that checkpoint from CREATED/FAILED to COMPLETE/FAILED in
the do_sync_snaps() along with the timestamp of the event.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 37163436cc87a2be691202bb565c7b5007df026f)
Karthik U S [Tue, 12 May 2026 23:21:53 +0000 (04:51 +0530)]
tools/cephfs_mirror: Helper functions for mirroring checkpoints
Implementation of helper functions and data structures for the
snapshot based mirroring checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 0a394ef438abd0248d078ea72e8ae5ce4f3d5e84)
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 959bc2eb07cacf70a4da09170e1c9bbed4b9423f)
Kotresh HR [Tue, 23 Jun 2026 15:13:18 +0000 (20:43 +0530)]
qa: Add mgr snapshot mirror status tests
Add teuthology coverage for `ceph fs snapshot mirror status`:
parity with asok peer_status, default idle metrics, stale omap
handling, error paths, filter scopes, daemon restart, and cache TTL.
Reuse existing peer_dir_status and metrics assertion helpers.
Adjust stale test wait for InstanceWatcher timeout and set a
short cache TTL in the cache test. Pass peer_uuid via --peer_uuid=
in the test helper for peer-only scope queries.
Kotresh HR [Sun, 21 Jun 2026 17:59:43 +0000 (23:29 +0530)]
doc/cephfs: Document fs snapshot mirror status mgr command
Describe the mirroring module command that reads persisted omap
metrics, including syntax, output layout, stale detection, caching,
and comparison with the admin socket peer status interface.
Document --peer_uuid as a named argument for peer-only and
directory/peer filtering.
Kotresh HR [Sun, 21 Jun 2026 17:52:29 +0000 (23:22 +0530)]
mgr/mirroring: make snapshot mirror metrics cache optional
Add snapshot_mirror_metrics_cache_enabled (default true). When disabled,
metrics_status reads omap directly and skips complete and partial caches.
When enabled, behavior is unchanged.
Kotresh HR [Sun, 21 Jun 2026 17:50:28 +0000 (23:20 +0530)]
mgr/mirroring: make snapshot mirror metrics cache TTL configurable
Add snapshot_mirror_metrics_cache_ttl as a runtime mgr/mirroring module
option (default 15 seconds) instead of a hard-coded CACHE_TTL_SECS.
Both complete and partial lru_cache_timeout wrappers read the value
when caching omap metrics so operators can tune cache freshness without
code changes.
Kotresh HR [Sun, 21 Jun 2026 17:48:15 +0000 (23:18 +0530)]
mgr/mirroring: handle missing cephfs_mirror object in metrics status
When snapshot mirroring is not enabled for a filesystem, the
cephfs_mirror RADOS object does not exist and "ceph fs snapshot mirror
status" fails reading sync stat omap. Map rados ENOENT to a clear
MirrorException so the CLI reports that snapshot mirroring must be
enabled instead of a generic omap read failure or an uncaught exception.
Also catch unexpected errors in metrics_status like other mirror CLI
handlers, so the mgr module does not crash on failure. Return
-errno.EINVAL from the generic error path instead of the exception
message as exit code.
Kotresh HR [Sun, 21 Jun 2026 17:28:38 +0000 (22:58 +0530)]
mgr/mirroring: detect stale snapshot mirror sync metrics in omap
Persisted metrics in the cephfs_mirror omap can outlive the writing
daemon when cephfs-mirror stops or a directory is reshuffled to another
instance; mgr would keep reporting stale progress until the owning
daemon writes again.
Extend format_and_order_sync_stat_for_display() to compare persisted
_instance_id against InstanceWatcher live instances (via
FSPolicy.get_live_instance_ids()) and the directory's tracked instance
(via Policy.get_tracked_instance_id()). Mark metrics stale when the
persisted writer is no longer live (any state), or when it does not
match the tracked instance while persisted state is not "idle". Show
state "stale" with current_syncing_snap omitted.
Pass policy and live instance ids through load_sync_stat_metrics() and
fetch_sync_stat_metrics(), and into sync_stat_complete_cache and
sync_stat_partial_cache loaders. Cache hits serve already-formatted
(stale-marked) entries until TTL expiry without re-checking instance
liveness.
Kotresh HR [Tue, 16 Jun 2026 10:42:41 +0000 (16:12 +0530)]
mgr/mirroring: cache fs snapshot mirror status omap metrics
Add a short-lived in-memory cache for metrics returned by
"ceph fs snapshot mirror status", which reads persisted sync stats
from the cephfs_mirror object omap. Omap walks are relatively
expensive; caching reduces repeated reads when the CLI or multiple
clients poll at short intervals.
Two TTL-bucketed LRU caches (CACHE_TTL_SECS) are used instead of one
unified cache. The complete cache always holds every mirrored
directory and peer for a filesystem; the partial cache holds one
directory. A single directory-granularity cache cannot prove it has
all directories, so full-scan queries rely on the complete cache
contract.
Cache implementation (lru_cache_timeout decorator backed by
_TimedLRUCache, not functools.lru_cache):
- lru_cache has no TTL and no peek-on-hit API. Single-directory
queries must read the complete cache without loading on miss;
lru_cache.cache is also unavailable on Python 3.14+.
- complete (sync_stat_complete_cache): full omap prefix scan via
load_sync_stat_metrics. Cache key: (time_token, filesystem).
COMPLETE_CACHE_MAX limits filesystem entries per TTL window.
Bind cache_peek/cache_info/cache_clear via a CachedMethod descriptor
so TTL lookups receive (self, ...); otherwise single-dir status fails
with 'str' object has no attribute 'mgr'.
Serve logic (under the existing lock):
- status <fs> [--peer_uuid=<uuid>]: load complete cache on miss;
filter by peer when requested.
- status <fs> <dir>: peek complete cache (no load on miss); if the
entry contains the directory and peers, serve from complete.
Otherwise load partial cache (omap on miss).
Kotresh HR [Mon, 1 Jun 2026 09:29:31 +0000 (14:59 +0530)]
mgr/mirroring: Reorder and format metrics output
Reorder and format status output to match the output of asok
interface peer_status command. Return metrics under
metrics/<dir>/peer/<uuid> to match the asok peer_status layout,
including {"metrics": {}} when no peers are configured.
mgr/mirroring: Add new interface to expose mirroring metrics
Add the following new interface to expose mirroring metrics
ceph fs snapshot mirror status <fsname> [<mirrored_dir_path>] [--peer_uuid=<peer_uuid>]
The cmd loads the persisted directory sync metrics from the
cephfs_mirror object's omap. Metrics are grouped by mirrored
directory and peer.
When --peer_uuid is specified, only metrics for that peer are
returned. peer_uuid is a named CLI argument (_end_positional_) so
peer-only filtering does not require a mirrored directory path.
Kotresh HR [Sat, 20 Jun 2026 07:58:56 +0000 (13:28 +0530)]
tools/cephfs_mirror: Remove persisted dir stats
When a directory is removed from mirroring, the persisted directory
stats need to be removed. This patch handles the cleanup.
Omap keys must not be removed when mirrored directories are reshuffled
across cephfs-mirror daemons. The mgr release notify now carries a
purging flag (set only during permanent removal, not reshuffle), and
the daemon removes persisted stats only when purging is true. On
reshuffle with an in-progress sync, clear live current_syncing_snap
state and persist idle metrics so the acquiring daemon does not inherit
stale syncing omap entries.
Kotresh HR [Sat, 20 Jun 2026 06:21:42 +0000 (11:51 +0530)]
tools/cephfs_mirror: Load persisted mirror metrics from omap
Load last_synced_snap metadata from the cephfs_mirror object omap on
PeerReplayer initialization and when a mirrored directory is added.
Live current_syncing_snap metrics are not restored; they are rebuilt
when synchronization starts.
When the daemon restarts after a snapshot was synced on the remote but
metrics were not yet written to omap, loaded metadata may belong to an
older snapshot. Add reconcile_last_synced_snap() to compare against
the remote snap map, clear stale last-sync fields, and update
last_synced_snap id/name in memory.
Treat snaps_synced, snaps_deleted, and snaps_renamed as per-session
counters. Do not load them from omap; they start at zero for each
daemon session and are still reported via the admin socket. Persist
omap metrics unconditionally after reconcile so the mgr picks up the
new instance id and cleared session counters on restart.
Kotresh HR [Sat, 20 Jun 2026 05:33:49 +0000 (11:03 +0530)]
tools/cephfs_mirror: Persist metrics to omap
Persist snapshot mirroring metrics to the cephfs_mirror object omap
for the mgr/mirroring module to support status command.
The tick thread only keeps omap up to date for in-progress syncs on
registered directories. Persist explicitly when stats change so omap
is updated before a directory unregisters:
- snap delete/rename propagation (inc_deleted_snap / inc_renamed_snap)
- after each successful snap sync (following set_last_synced_stat)
- after sync_snaps failure (following _inc_failed_count)
- after sync_perms failure (following _inc_failed_count)
Without the explicit call sites, omap can keep stale live or idle state
after sync completes or fails because directories leave m_registered
before the next tick.
Kotresh HR [Sun, 17 May 2026 21:01:58 +0000 (02:31 +0530)]
tools/cephfs_mirror: Adds capability to persist metrics
Adds the capability to persist mirroring metrics.
The metrics are persisted in the omap of the cephfs-mirror
object. Metrics are persisted asynchronously.
Each mirrored directory path stores the corresponding
metrics as the value of a unique omap key representing
the mirrored directory. The omap key is as below.
Kotresh HR [Tue, 16 Jun 2026 17:31:13 +0000 (23:01 +0530)]
qa: Add tests for cephfs_mirror_directory perf counters
Verify counter dump registration, current-sync gauges while syncing
(full/delta), last-sync and summary counters after idle sync, and
extend mirror stats and remote-snap failure tests for per-directory
snaps_* and dir_state.
Describe per-directory labeled perf counters, labels, update behavior,
mapping to peer status, and counter reference tables. Document the
per-peer tick thread and cephfs_mirror_tick_interval, which refreshes
current-sync gauges on each tick. Add a PendingReleaseNotes entry for
the new cephfs_mirror_directory perf counter group.
Kotresh HR [Tue, 16 Jun 2026 16:57:12 +0000 (22:27 +0530)]
cephfs_mirror: update per-directory last sync and summary perf counters
Wire the remaining cephfs_mirror_directory labeled counters that were
registered in the prior commit but not yet refreshed. Live
current_syncing_snap gauges continue to be updated from the per-peer tick
thread; this commit updates last_synced_snap and per-directory snap
summary counters at the points where SnapSyncStat is actually modified.
Depends-on the cephfs_mirror_directory PerfCounters schema (dir_state,
current_*, last_*, snaps_*) added when each mirrored directory is
registered.
These counters appear under "cephfs_mirror_directory" in "counter dump" with
the same labels as current-sync metrics (source_fscid, source_filesystem,
peer_uuid, peer_cluster_name, peer_cluster_filesystem, directory). ceph-exporter
exposes them as e.g. ceph_cephfs_mirror_directory_last_sync_bytes and
ceph_cephfs_mirror_directory_snaps_synced.
Unlike cephfs_mirror_peers, values are per (peer_uuid, directory) rather than
aggregated across all directories on the peer. Peer-level counters are
unchanged.
Kotresh HR [Tue, 16 Jun 2026 17:22:21 +0000 (22:52 +0530)]
tools/cephfs_mirror: Expose per-directory snap metrics via perf counters
Introduce a new labeled perf counter group, cephfs_mirror_directory, so
per-directory snapshot mirror progress can be scraped via "counter dump" and
exported to Prometheus by ceph-exporter (e.g.
ceph_cephfs_mirror_directory_current_sync_bytes).
Design
------
* One PerfCounters instance per mirrored directory on a peer, keyed in
m_directory_perf_counters and registered on the daemon-wide
PerfCountersCollection.
* Labels on each instance (flat counter dump array entries):
- source_fscid, source_filesystem
- peer_uuid, peer_cluster_name, peer_cluster_filesystem
- directory (dir_root, e.g. "/parent/d1")
The peer_uuid label disambiguates the same directory path mirrored to
different peers.
* Counters are created in init() and add_directory(), removed in
remove_directory() and the PeerReplayer destructor.
* Priority follows cephfs_mirror_perf_stats_prio (same as
cephfs_mirror_peers).
Update path
-----------
Live / current_syncing_snap gauges are refreshed from
update_directory_current_sync_perf_counters(), called by
refresh_directory_current_sync_perf_counters() from the per-peer tick
thread (run_tick()). Each cephfs_mirror_tick_interval seconds (default 5)
the tick thread updates counters for each registered (actively syncing)
directory.
All of the following are added to the builder in this commit. Only the
"current sync" and dir_state fields listed under "Updated in this commit"
are written here; last_synced_snap and per-directory snap summary counters
are registered for a follow-up commit that updates them when stats change.
Current syncing snapshot (peer_status "current_syncing_snap")
[Updated in this commit]
current_snap_id - snapshot id being synchronized
current_sync_mode - 0 = full, 1 = delta (snapdiff)
current_read_bps - bytes/sec read (raw, not formatted)
current_write_bps - bytes/sec written
crawl_state - 0 = N/A, 1 = in-progress, 2 = completed
crawl_duration_seconds - crawl duration; in-progress uses now - start
datasync_wait_state - 0 = none, 1 = waiting, 2 = complete
datasync_wait_duration_seconds
current_sync_bytes - bytes synced so far for this snap
current_total_bytes - total bytes for this snap
current_sync_bytes_percent - basis points (1745 = 17.45%)
current_sync_files
current_total_files
current_sync_files_percent - basis points
current_eta_valid - 0 = calculating, 1 = ETA available
current_eta_seconds - ETA in seconds when valid
Per-directory snapshot summary (peer_status snaps_*)
[Registered only; not updated in this commit]
snaps_synced, snaps_deleted, snaps_renamed
Last synced snapshot (peer_status "last_synced_snap")
[Registered only; not updated in this commit]
last_snap_id
last_crawl_duration_seconds
last_datasync_wait_duration_seconds
last_sync_duration_seconds
last_sync_timestamp - utime_t / seconds since epoch
last_sync_bytes
last_sync_files
When idle or failed, current_* counters are zeroed and dir_state reflects
0 or 2 respectively.
Kotresh HR [Tue, 16 Jun 2026 16:38:15 +0000 (22:08 +0530)]
tools/cephfs_mirror: Add per-peer tick thread with configurable interval
Introduce a per-peer tick thread controlled by cephfs_mirror_tick_interval
(default 5 seconds). The interval is re-read each iteration so configuration
changes take effect without restarting the daemon. The thread provides a
generic hook for future periodic mirroring work.
Kotresh HR [Mon, 25 May 2026 18:22:29 +0000 (23:52 +0530)]
doc: Update the mirroring doc with new metrics fields
Update the mirroring documentation and also the
release notes with new metrics introduced and it's
availability via 'fs mirror peer status' asok
interface.
Kotresh HR [Fri, 5 Jun 2026 14:23:14 +0000 (19:53 +0530)]
tools/cephfs_mirror: Nest peer_status metrics by dir path and peer uuid
Restructure peer_status output so mirrored directory paths can be
shared by multiple peers without key collisions. Metrics are grouped
as metrics/<dir_path>/peer/<peer_uuid>/ instead of flat dir keys.
Kotresh HR [Sat, 28 Mar 2026 11:23:33 +0000 (16:53 +0530)]
tools/cephfs_mirror: Add eta metrics
Add estimate time of completion for the current
syncing snapshot. The calculation takes into
account the average read/write throughput from
the start of snapshot sync and not the current
read/write throughput. So the ETA is affected
accordingly.
Kotresh HR [Sat, 28 Mar 2026 10:57:02 +0000 (16:27 +0530)]
tools/cephfs_mirror: Add read/write throughput
The read throughput added measures the bytes
read per second from the source ceph filesystem.
Similarly, the write throughput added measures
the bytes written per second to the remote ceph
filesystem. It's derived from the time spent
in preadv and pwritev calls.
sync-mode:
---------
The 'sync-mode: full/delta' is added to peer status.
The 'delta' means, blockdiff along with snapdiff is
being used to sync the files where as 'full' means
full directory is crawled and each file is synced
entirely.
crawl:
-----
The state can be in-progress/completed. This
identifies whether the crawler thread is done
queuing the files for data sync threads.
The time taken for the duration is also shown.
If the crawl is in-progress, the duration
would show the time taken till then from the
start of the crawl. If the crawl state is
completed, then duration indicates total
time taken for the crawl.
The crawl duration is shown in "d h m s" format.
The existing 'sync_duration' in last_synced_snap
is also formatted
The values are as below. When crawl state is
completed, the 'total_files' metric doesn't
grow anymore.
crawl_duration:
--------------
The crawl_duration of last snapshot is saved in last_synced_snap
section as well.
Kotresh HR [Mon, 16 Feb 2026 10:59:31 +0000 (16:29 +0530)]
tools/cephfs_mirror: Add inprogress bytes and files metric
Add following mirroring progress metrics to current_syncing_snap
as below
bytes:
sync_bytes - bytes synced till now
total_bytes - total bytes to be synced
sync_percent - Percentage of bytes synced till now
files:
total_files - Total files to be synced
sync_files - files synced till now
sync_percent - Percentage of files synced till now
sync_files and sync_bytes are also stored in last_synced_snap section
after the snapshot is synced.
ceph-volume: skip internal raid mirror LVs in inventory
ceph-volume inventory started including all LVM mapper devices after c06bee965f1. On hosts with raid mirrored system volumes, that pulls in
hidden legs like var_rmeta_0 which have no /dev/vg/lv node and makes
cephadm's ceph-volume inventory call fail.
Skip those internal LVs in get_devices() and avoid rewriting the device
path to a missing lv_path in Device._parse().
Collect FCM stats locally from NVMe drives (vendor log page 0xCA)
and expose them via node-proxy, the cephadm agent, and
`ceph orch hardware status --category fcm`.
Introduce a node backend that aggregates Redfish data with
node-local collectors, since FCM metrics are not available
from the BMC.
Abhishek Desai [Tue, 26 May 2026 07:48:40 +0000 (13:18 +0530)]
mgr/dashboard : Support wildcard sans and zonegroup hostnames
fixes : https://tracker.ceph.com/issues/76795 Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>
(cherry picked from commit eee6a15dcd845914595dbb1d76470bd4b73947c1)
Aashish Sharma [Thu, 11 Jun 2026 09:19:04 +0000 (14:49 +0530)]
mgr/dashboard: fix zone creation in rgw service creation form
The zone creation request from the rgw service creation form was missing
the tier_type, sync_from and sync_from_all properties as a result the
zone creation was failing. This PR tends to fix this issue.
Afreen Misbah [Tue, 23 Jun 2026 13:30:22 +0000 (19:00 +0530)]
mgr/dashboard: fix bind address regression from CherryPy isolation
The CherryPy isolation refactor (PR #67227) accidentally changed the
dashboard bind address from wildcard (*:8443) to mon_ip:8443. The
get_mgr_ip() replacement was originally only for URI generation, but
the refactor passed the mutated address to CherryPyMgr.mount() as the
actual socket bind address.
This breaks the management gateway when its VIP is not on the same
interface as mon_ip, as the dashboard becomes unreachable on other
interfaces.
Preserve the original wildcard address for binding and only use
get_mgr_ip() for the advertised URI. Add regression test to prevent
future confusion between bind_addr and server_addr.
Kefu Chai [Sat, 13 Jun 2026 01:50:09 +0000 (09:50 +0800)]
python-common/cryptotools: stop using the removed X509Req API
pyOpenSSL deprecated OpenSSL.crypto.X509Req in 24.2.0 (2024-07-20) and
removed it in 26.3.0 (2026-06-12). as we don't pin pyopenssl, CI picked
up the new release, and create_self_signed_cert() started failing with:
AttributeError: module 'OpenSSL.crypto' has no attribute 'X509Req'
this took down run-tox-mgr, run-tox-mgr-dashboard-py3 and the mypy check.
we only used X509Req to build a subject name and then copied it into the
X509 cert. so drop it, and set the subject on the cert directly. the
resulting cert stays the same: subject from dname, issuer set to the same
subject, self-signed.
Alex Ainscow [Tue, 9 Jun 2026 14:45:25 +0000 (15:45 +0100)]
osd/ECTransaction: fix truncate+write planning for EC shard sizes
There are multiple problems fixed here, which are caused by operations
which perform a truncate-then-write in a single transaction.
NOTE: The only known scenario for these operations are CLS-sparsify operations.
I recommend backporting these changes to Tentacle, however, as they are
regressions in the RADOS API which can lead to data corruption.
PROBLEM: Cache not invalidated correctly:
If projected_size >= orig_size, the invalidate_cache flag is not set in the plan
This means there is potentially data in the RMW extent cache, which
may cause data corruption in subsequent writes (although not the write
being made). No actual use case for this operation is known, so it may not be as serious as
it sounds.
FIX: Invalidate cache on any truncate or "delete_first"
PROBLEM: Parity writes permitted with truncates:
Currently parity delta writes do not support operations which may have
truncates. However, if the object ended up the same size as pre-truncate, a
parity delta write may have been attemtped. This may lead to data corruption.
FIX: Block parity writes in this case.
NOTE: It is not worth the development effort to support PDW in this scenario, as
performance benefit would be minimal overall.
PROBLEM: RMW Reads can occur beyond first truncate point.
Such reads reflect the pre-truncate data (which is incorrect) and be
preserved, even if the invalidate cache flag is set (since the cache is
invalidated before the reads). This can lead to similar corruption as
the earlier "cache not invalidated"
FIX: trim the read to the lower truncate size.
PROBLEM:
When performing a truncate, the parity shards may need to be updated to
reflect truncates on other shards. These truncate writes in the plan were
overriding, rather than adding to, other writes in the op. The result was
a potentially truncated coding shard. This leads to assertions on reads.
FIX: Replace = with insert()
PROBLEM: Some shards not set to correct size on truncate-then-write
If projected_size is smaller or equal to the original size, then the
code which attempts to correctly size a shard will not run. However if
an operation performs a truncate then a partial write and does not
write to a particular shard AND the shard ends up smaller, then this
can lead to an incorrectly sized shard. This leads to assertion on reads.
FIX: Execute the shard-resize code on all truncates.
AI assistance was mainly used to write unit test. However, I cannot rule out a
contribution to the simple fixes found in this commit, so out of caution,
I place the Assisted-by tag.
Fixes: https://tracker.ceph.com/issues/77276 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com> Assisted-by: IBM-Bob:ClaudeSonnet/GPT
(cherry picked from commit 51d8c5c489ba3e664209fb3316f8d6e03e257e28)
kmip tests are failing again on ubuntu 24 because PyKMIP doesn't support
python 3.12. we'll be removing ubuntu 22 from main, so can't just pin the
test to that distro in the meantime
we're expecing the nvmeof team to add python 3.12 support in our
ceph/PyKMIP fork, and can reenable kmip testing once that happens
Fixes: https://tracker.ceph.com/issues/76995 Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d27261d0c2641e92365cf590684567c58bbfb905) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 17 Jun 2026 18:33:45 +0000 (14:33 -0400)]
qa/rgw: Remove 'force-branch' from s3tests configs
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 5bc34f8ed84aa2f205ccfe04fff9e3ec5ced8727) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 29 Sep 2025 21:11:09 +0000 (17:11 -0400)]
qa/rgw: Run s3-tests from within the Ceph repo
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit d0c88cb53c6848e9bcd7c4ea90f21f1aa6912242) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Thu, 11 Jun 2026 22:46:45 +0000 (18:46 -0400)]
test/rgw: Include s3-tests in Ceph repo
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 868d20182a08685265e0b5831bad703fdcf3189b) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Kefu Chai [Tue, 23 Jun 2026 07:43:28 +0000 (15:43 +0800)]
mgr/dashboard: skip the table when an nvmeof cli result has no columns
The dashboard leaves prettytable unpinned. prettytable commit 2574492 ("Apply
some Pylint rules (PLR)", #436) rewrote _stringify_row()'s row_height as
`max(_get_size(c)[1] for c in row)`, which raises ValueError("max() iterable
argument is empty") on a row with no cells. The change is undocumented and
shipped in 3.18.0; get_string() trips on it when a table has a row but no
columns.
AnnotatedDataTextOutputFormatter builds such a table for an empty result, or
one whose only field is status or error_message, so NvmeofCLICommand.call()
returns -EINVAL and the command fails. This broke run-tox-mgr-dashboard-py3
once the tox virtualenv picked up prettytable 3.18.0.
Return an empty string when there are no columns instead of formatting a
degenerate table.
rgw: restore constant-time GCM tag comparison in ISA-L path
a8ed43bfc05 replaced ct_memeq with memcmp in the ISA-L GCM accelerator,
making tag verification and the key-cache compare non-constant-time.
Restore ct_memeq for both; the OpenSSL and EVP paths already compare in
constant time.
Matthew Heler [Fri, 5 Jun 2026 15:48:32 +0000 (10:48 -0500)]
rgw: fix AES-256-GCM key/IV reuse on multipart part re-upload
Re-uploading the same part number in a GCM multipart upload encrypted the new
data under the same key and IV as the first upload, since the IV is
part_number||chunk_index and the part key came from the part number alone. GCM
requires a unique IV per key; reusing one to encrypt different data weakens its
confidentiality and integrity guarantees.
Generate a random 16-byte salt on each UploadPart and fold it into the part key,
HMAC(ObjectKey, BE32(part) || salt), so every upload gets a fresh key. The salt
rides RGWUploadPartInfo, and complete stores the selected part's salt in
RGW_ATTR_CRYPT_PART_NUMS, which now holds (part, salt) pairs. GET reads it back
to re-derive the key, and an empty salt reproduces the old derivation so unsalted
parts still decrypt.
rgw/restore: take the hash mod HASH_PRIME when picking a shard
choose_oid fed ceph_str_hash_linux straight into % max_objs, and the
low bits of that hash are weak enough that similar object names keep
landing on the same shard. LC takes the hash mod HASH_PRIME first for
exactly this reason, do the same here.
Matthew N. Heler [Mon, 18 May 2026 01:57:01 +0000 (20:57 -0500)]
mon: add monitor RocksDB backup and restore
Implements an opt-in backup mechanism for the monitor using
rocksdb::BackupEngine. Backups run on a schedule when
mon_backup_interval is set, or are triggered manually via
`ceph tell mon.* backup`. Cleanup keeps the last N, hourly,
and daily snapshots, with a free-space guard. Off by default.
Restore is offline: stop the mon and run
ceph-mon --restore-backup <dir> --yes-i-really-mean-it
optionally with --backup-version (BackupEngine logical version,
as shown by --list-backups). The mon keyring is stashed alongside
the RocksDB backup so a wiped mon_data is recovered end-to-end,
and kv_backend is stamped back when missing.
Co-authored-by: Daniel Poelzleithner <poelzleithner@b1-systems.de> Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
(cherry picked from commit 3a9ae41e2a8fd614d67e3dac39d28ddf5dd6ca4a)
mgr/DaemonServer: Aggregate and globally sort OSDs for ok-to-upgrade
The 'ok-to-upgrade' command output sorting did not scale accurately
when target CRUSH buckets contained multiple child buckets (e.g., a
chassis containing multiple hosts). OSDs were previously sorted
individually per child bucket and appended sequentially. This created
fragmented, per-host sort segments rather than a globally sorted list
for the parent bucket.
Changes:
1. Fix the issue above by aggregating all child OSDs into a single vector prior
to executing a single, global sort operation based on PG counts. Additionally,
optimize memory efficiency and future-proof the logic by reserving continuous
vector blocks to avoid dynamic heap reallocations.
2. Add integration tests with chassis and rack based CRUSH hierarchies which
verifies the ok-to-upgrade functionality. In addition, the tests crucially
verify the order of OSDs returned is according to the ascending order of
acting PG count. Additionally, make minor fix-ups to lines that determine the
length of a list in JSON response by removing the redundant "| bc".
Patrick Donnelly [Fri, 18 Nov 2022 19:13:01 +0000 (14:13 -0500)]
umbrella: doc: add releases links to toc
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 8cf9ad62949516666ad0f2c0bb7726ef68e4d666) Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Conflicts:
doc/index.rst: index changes
Patrick Donnelly [Wed, 10 Jun 2026 18:30:59 +0000 (14:30 -0400)]
Merge PR #66726 into main
* refs/pull/66726/head:
doc: Update documentation to reflect new functionality
test: Add integration tests for EC Omap operations and recovery
osd: Hook up omap operations in EC pools
osd: Allow for recovery of OMAP header and entries in EC pools
doc: Write design document to explain the reasoning behind implementing this feature
osd: Introduce functions required for EC OMAP support
osd: Add ECOmapJournal class and relocate OmapUpdateType enum class
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com> Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
projects / ceph.git / log