Sun Yuechi [Tue, 7 Jul 2026 02:03:59 +0000 (19:03 -0700)]
rbd-mirror: take MirrorStatusUpdater lock by value in queue_update_task
queue_update_task() took the lock by rvalue reference, so its early
returns never released it. In handle_update_task() that left m_lock
held across the on_finish->complete() calls; during shutdown a
completion can re-enter try_remove_mirror_image_status() and re-lock
m_lock, deadlocking.
Take the unique_lock by value so it unlocks on every return path.
Fixes: https://tracker.ceph.com/issues/78047 Signed-off-by: Sun Yuechi <sunyuechi@iscas.ac.cn>
(cherry picked from commit b450195246cfeb7eabcaf8da53a433b98f279fee)
Patrick Donnelly [Mon, 27 Jul 2026 14:21:10 +0000 (10:21 -0400)]
Merge PR #69765 into umbrella
* refs/pull/69765/head:
rbd-mirror: Remove old non-primary demoted image snapshots on the local cluster
rbd-mirror: prune obsolete primary mirror snapshots after relocation
rbd-mirror: fix missing initialization of Peer UUID
Patrick Donnelly [Mon, 27 Jul 2026 14:01:13 +0000 (10:01 -0400)]
Merge PR #70287 into umbrella
* refs/pull/70287/head:
doc/rbd: elaborate on key-ref syntax (existing in S3Stream and new in NativeFormat)
qa/suites/rbd: add mon_host + key[-ref] coverage to migration-external
qa/suites/rbd: use client.0 entity in migration-external tests
librbd/migration/NativeFormat: support specifying mon_host and key via spec
Patrick Donnelly [Mon, 27 Jul 2026 13:51:53 +0000 (09:51 -0400)]
Merge PR #70176 into umbrella
* refs/pull/70176/head:
mgr/dashboard: fixing manual deployment option for osd creation flow
mgr/dashboard: fix cancel button
mgr/dashboard: Converting Create OSDs tearsheet type from Full to Wide
mgr/dashboard: carbonized OSD form component
qa/tests: drop parens from all health-code ignorelist entries
Monitor.cc logs a "Health check cleared: <CODE> (was: ...)" line
for any health check clearing mid-test, with the code bare (no
parens) -- while the "Health check failed"/"Health check failed
(unmute)" lines wrap the code in parens. Every \(CODE\) entry in
this file therefore only matches the raise/failed form and misses
the corresponding cleared form, the same gap just fixed for
POOL_FULL.
Drop the escaped parens from the remaining entries so each matches
both log forms, consistent with the bare entries already present
(OSD_ROOT_DOWN, MDS_INSUFFICIENT_STANDBY, POOL_FULL).
qa/tests: fix POOL_FULL ignorelist pattern in upgrade tests
\(POOL_FULL\) only matches the "Health check failed/update" cluster-log
format (code wrapped in parens). It misses the "Health check cleared"
format, where the code is not parenthesized (see src/mon/Monitor.cc),
so a pool clearing its full state mid-upgrade-test was not ignorelisted.
Drop the escaped parens so the pattern matches both forms, consistent
with other bare entries already in this file (OSD_ROOT_DOWN,
MDS_INSUFFICIENT_STANDBY).
Patrick Donnelly [Mon, 20 Jul 2026 18:28:42 +0000 (14:28 -0400)]
Merge PR #69553 into umbrella
* refs/pull/69553/head:
doc: add PendingReleaseNotes entry for rgw multisite DNS endpoint resolution
rgw/rest: add TODO for concurrent endpoint DNS resolution
rgw/multisite: fix endpoint unreachable detection in RGWRESTConn sync paths
rgw: store RGWEndpoint URL as boost::urls::url
doc/radosgw: expose rgw_rest_conn_connect_to_resolved_ips and rgw_rest_conn_ip_fail_timeout_secs
rgw: rename round-robin counters for brevity
rgw/zone: increase visibility into zone connections via admin socket
rgw: make CONN_STATUS_EXPIRE_SECS a cfg option
rgw/rest: track connection failures per-IP instead of per-endpoint
rgw/rest: remove unused headers
rgw/rest: consolidate endpoint_urls and resolved_endpoints into single vector
rgw: add operator<< for RGWEndpoint and simplify logging
rgw: track original URL within RGWEndpoint instead of separate member (refactor)
rgw: fix incomplete RGWRESTConn move constructor/assignment
rgw/rest: consolidate endpoint status tracking into ResolvedEndpoint
rgw/http: apply RGWEndpoint connect_to via libcurl CURLOPT_CONNECT_TO
rgw/rest: round-robin resolved endpoint IPs into curl CONNECT_TO mapping
rgw/rest: resolve multisite endpoints to all A/AAAA records (optional)
rgw: add rgw_resolve_endpoints_into_all_addresses config option
rgw/http: introduce RGWEndpoint to carry url + connect_to (refactor)
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Patrick Donnelly [Mon, 20 Jul 2026 18:27:59 +0000 (14:27 -0400)]
Merge PR #69741 into umbrella
* refs/pull/69741/head:
tests/neorados: fix ceph_test_neorados_completions being not installed
common/async: async_cond::notify/cancel must post to handler's associated executor
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Patrick Donnelly [Mon, 20 Jul 2026 18:26:49 +0000 (14:26 -0400)]
Merge PR #69723 into umbrella
* refs/pull/69723/head:
mgr/dashboard: Add "gw refresh_network" cmd
mgr/dashboard: bump nvmeof submodule to 1.8.2
mgr/dashboard: align nvmeof cli with missing parameters and functions from the old nvmeof cli
monitoring: fix NVMeoFMultipleNamespacesOfRBDImage
Patrick Donnelly [Mon, 20 Jul 2026 18:11:35 +0000 (14:11 -0400)]
Merge PR #70267 into umbrella
* refs/pull/70267/head:
21.1.0
debian/rules: Also exclude librgw and radosgw from dwz
debian/rules: exclude ceph-osd-crimson from dwz compression
script/buildcontainer-setup: fix package install on debian trixie
David Galloway [Fri, 17 Jul 2026 18:34:44 +0000 (14:34 -0400)]
container: default FROM_IMAGE to Rocky Linux 10
Since tentacle, the preferred base image for ceph containers has been
Rocky Linux 10, and the CI tag-naming logic in build.sh already assumes
rockylinux-10 is the default fromtag for every branch except reef and
squid. The actual build default was never flipped, though: anything
that ran build.sh without FROM_IMAGE set (e.g. the release container
job in ceph-build) still got a CentOS Stream 9 base.
Flip the Containerfile ARG and the build.sh fallback to
docker.io/rockylinux/rockylinux:10 so umbrella and later build from
Rocky 10 by default. Builds that want a different base can still pass
FROM_IMAGE explicitly, as the CI pipeline does.
Ilya Dryomov [Mon, 29 Jun 2026 11:39:50 +0000 (13:39 +0200)]
qa/suites/rbd: use client.0 entity in migration-external tests
Currently client.admin is passed for client_name and that doesn't
exercise client_name handling much as client.admin is the default.
When deploying multiple clusters the ceph task distributes keyrings
with only the key for the initial monitor and client.admin key. Other
keys (e.g. client.0) are present only on their respective clusters, so
client.0's key for cluster2 needs to be obtained on cluster1 explicitly
with "ceph auth get".
Leonid Chernin [Mon, 2 Feb 2026 06:01:14 +0000 (08:01 +0200)]
librbd/migration/NativeFormat: support specifying mon_host and key via spec
migration:
-take secret_key from spec
-take mon_host from the spec
-ignore source keyring, source ceph.conf - bypass them
-added validations of the new way schema
-get key from the KV DB if key in spec is actually key-ref
Fixes: https://tracker.ceph.com/issues/68177 Signed-off-by: Leonid Chernin <leonidc@il.ibm.com> Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 705d865ae77f4d3e80a9d3da3281ba6a824d7f29)
Kefu Chai [Fri, 23 Jan 2026 07:04:47 +0000 (15:04 +0800)]
debian/rules: exclude ceph-osd-crimson from dwz compression
When building with DWZ enabled, the debian packaging fails with:
```
dh_dwz: error: Aborting due to earlier error
```
Running the dwz command manually reveals the root cause:
```
$ dwz -mdebian/ceph-osd-crimson/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug \
-M/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug -- \
debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson \
debian/ceph-osd-crimson/usr/bin/crimson-store-nbd
dwz: debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson: Too many DIEs, not optimizing
dwz: Too few files for multifile optimization
```
The dwz tool has a limit on the number of DWARF DIEs (Debug Information
Entries) it can process. The ceph-osd-crimson binary, being a large C++
executable with extensive template usage, exceeds this limit, causing
dwz to exit with status 1 and fail the build.
This change excludes only ceph-osd-crimson from dwz processing using
the -X flag, allowing other binaries in the package to still benefit
from DWARF compression while avoiding the build failure.
Please note, we always disable DWZ in ceph-build's ceph-dev-pipeline.
Kefu Chai [Fri, 23 Jan 2026 07:04:47 +0000 (15:04 +0800)]
debian/rules: exclude ceph-osd-crimson from dwz compression
When building with DWZ enabled, the debian packaging fails with:
```
dh_dwz: error: Aborting due to earlier error
```
Running the dwz command manually reveals the root cause:
```
$ dwz -mdebian/ceph-osd-crimson/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug \
-M/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug -- \
debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson \
debian/ceph-osd-crimson/usr/bin/crimson-store-nbd
dwz: debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson: Too many DIEs, not optimizing
dwz: Too few files for multifile optimization
```
The dwz tool has a limit on the number of DWARF DIEs (Debug Information
Entries) it can process. The ceph-osd-crimson binary, being a large C++
executable with extensive template usage, exceeds this limit, causing
dwz to exit with status 1 and fail the build.
This change excludes only ceph-osd-crimson from dwz processing using
the -X flag, allowing other binaries in the package to still benefit
from DWARF compression while avoiding the build failure.
Please note, we always disable DWZ in ceph-build's ceph-dev-pipeline.
pybind/rbd: handle non-existent groups properly in list_snaps()
Calling `list_snaps()` on a non-existent group crashed the entire
Python process with a `free(): invalid pointer` error. This happened
because an unexpected `-ENOENT` error returned by `rbd_group_snap_list2`
and the exception was raised without updating the num_group_snaps to 0
from 10. Later the Cython `__dealloc__` wrapper attempted to free garbage
of uninitialized pointer spaces resulting in a crash.
Fix this behavior by resetting `self.num_group_snaps` to 0 inside
`GroupSnapIterator` before raising exception to prevent memory
corruption and accessing uninitialized pointers during cleanup call
Now `list_snaps()` consistently raise an `ObjectNotFound` error when
invoked on a non-existent group.
Also Extended the `TestGroups` test fixture with `self.dne_group` to
validate the expected error path for `list_snaps()`.
Fixes: https://tracker.ceph.com/issues/78034 Signed-off-by: VinayBhaskar-V <vvarada@redhat.com>
(cherry picked from commit 56bcf706cf91f0734899ae9de378c459a9b5af50)
David Galloway [Fri, 10 Jul 2026 20:12:01 +0000 (16:12 -0400)]
script/buildcontainer-setup: fix package install on debian trixie
Debian removed software-properties-common from the archive in trixie,
so the flat apt-get install list fails there. The package was only
needed to provide add-apt-repository for llvm.sh, which installs
clang-19 from apt.llvm.org. Trixie ships clang-19 natively, so install
it from the distro instead; run-make.sh's prepare() then finds clang-19
and skips llvm.sh entirely. Ubuntu and older Debian releases still have
software-properties-common and keep the previous behavior.
mgr/dashboard: Converting Create OSDs tearsheet type from Full to Wide Fixes: https://tracker.ceph.com/issues/77132 Signed-off-by: Devika Babrekar <devika.babrekar@ibm.com>
(cherry picked from commit 4eb4e25e5e6f14ed70fee5c41b2669cc0c557e19)
Fixes: https://tracker.ceph.com/issues/68265 Signed-off-by: Syed Ali Ul Hasan <syedaliulhasan19@gmail.com>
(cherry picked from commit 78dad340ab180532082c39a52efc0dfce244ec6e)
Nizamudeen A [Wed, 8 Jul 2026 09:27:33 +0000 (14:57 +0530)]
mgr/dashboard: fix unncessary traceback when bucket not exist
UI has an async validator which calls the GET bucket API to make sure
the bucket name doesn't exist, but the proxy
was not properly handling the http_status_codes which results in raising
a massive traceback in logs whenever you type things in the bucket name
field. So handling that gracefully by capturing the proper status codes
for both RequestException and DashboardException
BEFORE
```
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
return handler(*args, **kwargs)
File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in _call_
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper
return func(*vpath, **params)
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 357, in get
result = self.proxy(daemon_name, 'GET', 'bucket', {'bucket': bucket})
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 213, in proxy
raise DashboardException(e, http_status_code=http_status_code, component='rgw')
dashboard.exceptions.DashboardException: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000f14e08c1af0d5615-006'
b'71f54a7-3bc6-default","HostId":"3bc6-default-default"}')
2024-10-28T09:08:55.990+0000 7f89e045b640 0 [dashboard INFO request] [::ffff:10.74.18.122:51853] [GET] [500] [0.007s] [admin] [200.0B] /api/rgw/bucket/bucket-das
```
AFTER
```
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.services.exception] Dashboard Exception: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","RequestId":"tx00000c029a81e5d154844-006'
b'a4e183c-14251-default","HostId":"14251-default-default"}')
Jul 08 09:28:28 ceph-node-00 ceph-mgr[2243]: [dashboard INFO dashboard.tools] [::ffff:192.168.100.1:60408] [GET] [404] [0.078s] [admin] [200.0B] /api/rgw/bucket/testsa
```
qa/suites/rbd/valgrind: pin to centos_9.stream instead of rpm_latest
This used to be the case before commit d4b977afdc59 ("qa/distros:
rename centos_latest.yaml to rpm_latest.yaml") and subsequent changes
to enable Rocky 10. When paired with valgrind, python_api_tests* jobs
fail persistently on Ubuntu and Rocky, see [1]. Switch back until that
is resolved (or for as long as centos_9.stream remains in the mix).
David Galloway [Fri, 10 Jul 2026 20:12:01 +0000 (16:12 -0400)]
script/buildcontainer-setup: fix package install on debian trixie
Debian removed software-properties-common from the archive in trixie,
so the flat apt-get install list fails there. The package was only
needed to provide add-apt-repository for llvm.sh, which installs
clang-19 from apt.llvm.org. Trixie ships clang-19 natively, so install
it from the distro instead; run-make.sh's prepare() then finds clang-19
and skips llvm.sh entirely. Ubuntu and older Debian releases still have
software-properties-common and keep the previous behavior.
Fixes: https://tracker.ceph.com/issues/78111 Signed-off-by: David Galloway <david.galloway@ibm.com>
(cherry picked from commit b5697987ae86f51d7f16a1409ce9f2b542f3de52)
Super User [Wed, 8 Jul 2026 12:05:16 +0000 (17:35 +0530)]
doc/rbd: clarify mirror resync snapshot behavior
Add documentation explaining that rbd mirror resync only copies
image state and data up to the last mirror-snapshot for
snapshot-based mirroring. Document the requirement to create
a new mirror-snapshot on current primary so that new changes
(including resize operation) gets reflected on secondary after
resync.
Nizamudeen A [Wed, 8 Jul 2026 05:21:50 +0000 (10:51 +0530)]
mgr/dashboard: teardown http requests for feature-toggle
angular 19.2.latest is aggressively tearing down the test env which
forces the active rxjs timer to emit one execution as it collapses which
produces a ghost http in the case of request wrapped in the timer
service. so for now flushing all the requests manually to prevent it
Karthik U S [Sat, 16 May 2026 03:15:50 +0000 (08:45 +0530)]
qa/test_mirroring: Add tests for mirroring checkpoints
Adding integration tests for validating the cephfs mirroring
checkpoints feature
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit fe7fa7b0761720181cf8735ac7b5f17080c1d8a7)
Karthik U S [Thu, 4 Jun 2026 11:47:05 +0000 (17:17 +0530)]
doc: Add docs and pending release notes for mirroring checkpoints
Adding documentations and pending release notes for the mirroring
checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 12bd34d69ac7e136d14d25ea274e99727b559870)
Karthik U S [Wed, 10 Jun 2026 05:48:25 +0000 (11:18 +0530)]
mgr/mirroring,tools/cephfs_mirror: Handle checkpoint state transition
When a new checkpoint is being added or when the daemon gets restarted,
it will check whether the newly created checkpoint or any other old
checkpoints have already been mirrored onto the remote peer. If so, it
will transition to the correct state by checking for the highest snap id
present on the remote, and setting all the checkpoints which have snap id
lesser than or equal to that of the remote to COMPLETE. This is done by
sending an acquire notification from the mirroring module to the mirror
daemon, which is handled in the add_directory path, by adding the directory
to be checked for the state transition in the tick thread.
This path gets triggered:
a) when the daemon gets restarted
b) when the peer mapping changes
c) by sending the acquire notification from checkpoint add/now CLIs.
d) when mirroring module restarts
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit c64fe9636514d9e9b1c286a0b21b84569e266067)
Karthik U S [Sat, 16 May 2026 00:50:03 +0000 (06:20 +0530)]
tools/cephfs_mirror: update checkpoint status during snapshot sync
When the sync completes or fails for a checkpointed snapshot, transition
the status of that checkpoint from CREATED/FAILED to COMPLETE/FAILED in
the do_sync_snaps() along with the timestamp of the event.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 37163436cc87a2be691202bb565c7b5007df026f)
Karthik U S [Tue, 12 May 2026 23:21:53 +0000 (04:51 +0530)]
tools/cephfs_mirror: Helper functions for mirroring checkpoints
Implementation of helper functions and data structures for the
snapshot based mirroring checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 0a394ef438abd0248d078ea72e8ae5ce4f3d5e84)
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 959bc2eb07cacf70a4da09170e1c9bbed4b9423f)
Kotresh HR [Tue, 23 Jun 2026 15:13:18 +0000 (20:43 +0530)]
qa: Add mgr snapshot mirror status tests
Add teuthology coverage for `ceph fs snapshot mirror status`:
parity with asok peer_status, default idle metrics, stale omap
handling, error paths, filter scopes, daemon restart, and cache TTL.
Reuse existing peer_dir_status and metrics assertion helpers.
Adjust stale test wait for InstanceWatcher timeout and set a
short cache TTL in the cache test. Pass peer_uuid via --peer_uuid=
in the test helper for peer-only scope queries.
Kotresh HR [Sun, 21 Jun 2026 17:59:43 +0000 (23:29 +0530)]
doc/cephfs: Document fs snapshot mirror status mgr command
Describe the mirroring module command that reads persisted omap
metrics, including syntax, output layout, stale detection, caching,
and comparison with the admin socket peer status interface.
Document --peer_uuid as a named argument for peer-only and
directory/peer filtering.
Kotresh HR [Sun, 21 Jun 2026 17:52:29 +0000 (23:22 +0530)]
mgr/mirroring: make snapshot mirror metrics cache optional
Add snapshot_mirror_metrics_cache_enabled (default true). When disabled,
metrics_status reads omap directly and skips complete and partial caches.
When enabled, behavior is unchanged.
Kotresh HR [Sun, 21 Jun 2026 17:50:28 +0000 (23:20 +0530)]
mgr/mirroring: make snapshot mirror metrics cache TTL configurable
Add snapshot_mirror_metrics_cache_ttl as a runtime mgr/mirroring module
option (default 15 seconds) instead of a hard-coded CACHE_TTL_SECS.
Both complete and partial lru_cache_timeout wrappers read the value
when caching omap metrics so operators can tune cache freshness without
code changes.
Kotresh HR [Sun, 21 Jun 2026 17:48:15 +0000 (23:18 +0530)]
mgr/mirroring: handle missing cephfs_mirror object in metrics status
When snapshot mirroring is not enabled for a filesystem, the
cephfs_mirror RADOS object does not exist and "ceph fs snapshot mirror
status" fails reading sync stat omap. Map rados ENOENT to a clear
MirrorException so the CLI reports that snapshot mirroring must be
enabled instead of a generic omap read failure or an uncaught exception.
Also catch unexpected errors in metrics_status like other mirror CLI
handlers, so the mgr module does not crash on failure. Return
-errno.EINVAL from the generic error path instead of the exception
message as exit code.
Kotresh HR [Sun, 21 Jun 2026 17:28:38 +0000 (22:58 +0530)]
mgr/mirroring: detect stale snapshot mirror sync metrics in omap
Persisted metrics in the cephfs_mirror omap can outlive the writing
daemon when cephfs-mirror stops or a directory is reshuffled to another
instance; mgr would keep reporting stale progress until the owning
daemon writes again.
Extend format_and_order_sync_stat_for_display() to compare persisted
_instance_id against InstanceWatcher live instances (via
FSPolicy.get_live_instance_ids()) and the directory's tracked instance
(via Policy.get_tracked_instance_id()). Mark metrics stale when the
persisted writer is no longer live (any state), or when it does not
match the tracked instance while persisted state is not "idle". Show
state "stale" with current_syncing_snap omitted.
Pass policy and live instance ids through load_sync_stat_metrics() and
fetch_sync_stat_metrics(), and into sync_stat_complete_cache and
sync_stat_partial_cache loaders. Cache hits serve already-formatted
(stale-marked) entries until TTL expiry without re-checking instance
liveness.
Kotresh HR [Tue, 16 Jun 2026 10:42:41 +0000 (16:12 +0530)]
mgr/mirroring: cache fs snapshot mirror status omap metrics
Add a short-lived in-memory cache for metrics returned by
"ceph fs snapshot mirror status", which reads persisted sync stats
from the cephfs_mirror object omap. Omap walks are relatively
expensive; caching reduces repeated reads when the CLI or multiple
clients poll at short intervals.
Two TTL-bucketed LRU caches (CACHE_TTL_SECS) are used instead of one
unified cache. The complete cache always holds every mirrored
directory and peer for a filesystem; the partial cache holds one
directory. A single directory-granularity cache cannot prove it has
all directories, so full-scan queries rely on the complete cache
contract.
Cache implementation (lru_cache_timeout decorator backed by
_TimedLRUCache, not functools.lru_cache):
- lru_cache has no TTL and no peek-on-hit API. Single-directory
queries must read the complete cache without loading on miss;
lru_cache.cache is also unavailable on Python 3.14+.
- complete (sync_stat_complete_cache): full omap prefix scan via
load_sync_stat_metrics. Cache key: (time_token, filesystem).
COMPLETE_CACHE_MAX limits filesystem entries per TTL window.
Bind cache_peek/cache_info/cache_clear via a CachedMethod descriptor
so TTL lookups receive (self, ...); otherwise single-dir status fails
with 'str' object has no attribute 'mgr'.
Serve logic (under the existing lock):
- status <fs> [--peer_uuid=<uuid>]: load complete cache on miss;
filter by peer when requested.
- status <fs> <dir>: peek complete cache (no load on miss); if the
entry contains the directory and peers, serve from complete.
Otherwise load partial cache (omap on miss).
Kotresh HR [Mon, 1 Jun 2026 09:29:31 +0000 (14:59 +0530)]
mgr/mirroring: Reorder and format metrics output
Reorder and format status output to match the output of asok
interface peer_status command. Return metrics under
metrics/<dir>/peer/<uuid> to match the asok peer_status layout,
including {"metrics": {}} when no peers are configured.
mgr/mirroring: Add new interface to expose mirroring metrics
Add the following new interface to expose mirroring metrics
ceph fs snapshot mirror status <fsname> [<mirrored_dir_path>] [--peer_uuid=<peer_uuid>]
The cmd loads the persisted directory sync metrics from the
cephfs_mirror object's omap. Metrics are grouped by mirrored
directory and peer.
When --peer_uuid is specified, only metrics for that peer are
returned. peer_uuid is a named CLI argument (_end_positional_) so
peer-only filtering does not require a mirrored directory path.
Kotresh HR [Sat, 20 Jun 2026 07:58:56 +0000 (13:28 +0530)]
tools/cephfs_mirror: Remove persisted dir stats
When a directory is removed from mirroring, the persisted directory
stats need to be removed. This patch handles the cleanup.
Omap keys must not be removed when mirrored directories are reshuffled
across cephfs-mirror daemons. The mgr release notify now carries a
purging flag (set only during permanent removal, not reshuffle), and
the daemon removes persisted stats only when purging is true. On
reshuffle with an in-progress sync, clear live current_syncing_snap
state and persist idle metrics so the acquiring daemon does not inherit
stale syncing omap entries.
Kotresh HR [Sat, 20 Jun 2026 06:21:42 +0000 (11:51 +0530)]
tools/cephfs_mirror: Load persisted mirror metrics from omap
Load last_synced_snap metadata from the cephfs_mirror object omap on
PeerReplayer initialization and when a mirrored directory is added.
Live current_syncing_snap metrics are not restored; they are rebuilt
when synchronization starts.
When the daemon restarts after a snapshot was synced on the remote but
metrics were not yet written to omap, loaded metadata may belong to an
older snapshot. Add reconcile_last_synced_snap() to compare against
the remote snap map, clear stale last-sync fields, and update
last_synced_snap id/name in memory.
Treat snaps_synced, snaps_deleted, and snaps_renamed as per-session
counters. Do not load them from omap; they start at zero for each
daemon session and are still reported via the admin socket. Persist
omap metrics unconditionally after reconcile so the mgr picks up the
new instance id and cleared session counters on restart.
Kotresh HR [Sat, 20 Jun 2026 05:33:49 +0000 (11:03 +0530)]
tools/cephfs_mirror: Persist metrics to omap
Persist snapshot mirroring metrics to the cephfs_mirror object omap
for the mgr/mirroring module to support status command.
The tick thread only keeps omap up to date for in-progress syncs on
registered directories. Persist explicitly when stats change so omap
is updated before a directory unregisters:
- snap delete/rename propagation (inc_deleted_snap / inc_renamed_snap)
- after each successful snap sync (following set_last_synced_stat)
- after sync_snaps failure (following _inc_failed_count)
- after sync_perms failure (following _inc_failed_count)
Without the explicit call sites, omap can keep stale live or idle state
after sync completes or fails because directories leave m_registered
before the next tick.
Kotresh HR [Sun, 17 May 2026 21:01:58 +0000 (02:31 +0530)]
tools/cephfs_mirror: Adds capability to persist metrics
Adds the capability to persist mirroring metrics.
The metrics are persisted in the omap of the cephfs-mirror
object. Metrics are persisted asynchronously.
Each mirrored directory path stores the corresponding
metrics as the value of a unique omap key representing
the mirrored directory. The omap key is as below.
projects / ceph.git / log