Patrick Donnelly [Mon, 20 Jul 2026 18:28:42 +0000 (14:28 -0400)]
Merge PR #69553 into umbrella
* refs/pull/69553/head:
doc: add PendingReleaseNotes entry for rgw multisite DNS endpoint resolution
rgw/rest: add TODO for concurrent endpoint DNS resolution
rgw/multisite: fix endpoint unreachable detection in RGWRESTConn sync paths
rgw: store RGWEndpoint URL as boost::urls::url
doc/radosgw: expose rgw_rest_conn_connect_to_resolved_ips and rgw_rest_conn_ip_fail_timeout_secs
rgw: rename round-robin counters for brevity
rgw/zone: increase visibility into zone connections via admin socket
rgw: make CONN_STATUS_EXPIRE_SECS a cfg option
rgw/rest: track connection failures per-IP instead of per-endpoint
rgw/rest: remove unused headers
rgw/rest: consolidate endpoint_urls and resolved_endpoints into single vector
rgw: add operator<< for RGWEndpoint and simplify logging
rgw: track original URL within RGWEndpoint instead of separate member (refactor)
rgw: fix incomplete RGWRESTConn move constructor/assignment
rgw/rest: consolidate endpoint status tracking into ResolvedEndpoint
rgw/http: apply RGWEndpoint connect_to via libcurl CURLOPT_CONNECT_TO
rgw/rest: round-robin resolved endpoint IPs into curl CONNECT_TO mapping
rgw/rest: resolve multisite endpoints to all A/AAAA records (optional)
rgw: add rgw_resolve_endpoints_into_all_addresses config option
rgw/http: introduce RGWEndpoint to carry url + connect_to (refactor)
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Patrick Donnelly [Mon, 20 Jul 2026 18:27:59 +0000 (14:27 -0400)]
Merge PR #69741 into umbrella
* refs/pull/69741/head:
tests/neorados: fix ceph_test_neorados_completions being not installed
common/async: async_cond::notify/cancel must post to handler's associated executor
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Patrick Donnelly [Mon, 20 Jul 2026 18:26:49 +0000 (14:26 -0400)]
Merge PR #69723 into umbrella
* refs/pull/69723/head:
mgr/dashboard: Add "gw refresh_network" cmd
mgr/dashboard: bump nvmeof submodule to 1.8.2
mgr/dashboard: align nvmeof cli with missing parameters and functions from the old nvmeof cli
monitoring: fix NVMeoFMultipleNamespacesOfRBDImage
Patrick Donnelly [Mon, 20 Jul 2026 18:11:35 +0000 (14:11 -0400)]
Merge PR #70267 into umbrella
* refs/pull/70267/head:
21.1.0
debian/rules: Also exclude librgw and radosgw from dwz
debian/rules: exclude ceph-osd-crimson from dwz compression
script/buildcontainer-setup: fix package install on debian trixie
David Galloway [Fri, 17 Jul 2026 18:34:44 +0000 (14:34 -0400)]
container: default FROM_IMAGE to Rocky Linux 10
Since tentacle, the preferred base image for ceph containers has been
Rocky Linux 10, and the CI tag-naming logic in build.sh already assumes
rockylinux-10 is the default fromtag for every branch except reef and
squid. The actual build default was never flipped, though: anything
that ran build.sh without FROM_IMAGE set (e.g. the release container
job in ceph-build) still got a CentOS Stream 9 base.
Flip the Containerfile ARG and the build.sh fallback to
docker.io/rockylinux/rockylinux:10 so umbrella and later build from
Rocky 10 by default. Builds that want a different base can still pass
FROM_IMAGE explicitly, as the CI pipeline does.
Kefu Chai [Fri, 23 Jan 2026 07:04:47 +0000 (15:04 +0800)]
debian/rules: exclude ceph-osd-crimson from dwz compression
When building with DWZ enabled, the debian packaging fails with:
```
dh_dwz: error: Aborting due to earlier error
```
Running the dwz command manually reveals the root cause:
```
$ dwz -mdebian/ceph-osd-crimson/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug \
-M/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug -- \
debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson \
debian/ceph-osd-crimson/usr/bin/crimson-store-nbd
dwz: debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson: Too many DIEs, not optimizing
dwz: Too few files for multifile optimization
```
The dwz tool has a limit on the number of DWARF DIEs (Debug Information
Entries) it can process. The ceph-osd-crimson binary, being a large C++
executable with extensive template usage, exceeds this limit, causing
dwz to exit with status 1 and fail the build.
This change excludes only ceph-osd-crimson from dwz processing using
the -X flag, allowing other binaries in the package to still benefit
from DWARF compression while avoiding the build failure.
Please note, we always disable DWZ in ceph-build's ceph-dev-pipeline.
Kefu Chai [Fri, 23 Jan 2026 07:04:47 +0000 (15:04 +0800)]
debian/rules: exclude ceph-osd-crimson from dwz compression
When building with DWZ enabled, the debian packaging fails with:
```
dh_dwz: error: Aborting due to earlier error
```
Running the dwz command manually reveals the root cause:
```
$ dwz -mdebian/ceph-osd-crimson/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug \
-M/usr/lib/debug/.dwz/x86_64-linux-gnu/ceph-osd-crimson.debug -- \
debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson \
debian/ceph-osd-crimson/usr/bin/crimson-store-nbd
dwz: debian/ceph-osd-crimson/usr/bin/ceph-osd-crimson: Too many DIEs, not optimizing
dwz: Too few files for multifile optimization
```
The dwz tool has a limit on the number of DWARF DIEs (Debug Information
Entries) it can process. The ceph-osd-crimson binary, being a large C++
executable with extensive template usage, exceeds this limit, causing
dwz to exit with status 1 and fail the build.
This change excludes only ceph-osd-crimson from dwz processing using
the -X flag, allowing other binaries in the package to still benefit
from DWARF compression while avoiding the build failure.
Please note, we always disable DWZ in ceph-build's ceph-dev-pipeline.
David Galloway [Fri, 10 Jul 2026 20:12:01 +0000 (16:12 -0400)]
script/buildcontainer-setup: fix package install on debian trixie
Debian removed software-properties-common from the archive in trixie,
so the flat apt-get install list fails there. The package was only
needed to provide add-apt-repository for llvm.sh, which installs
clang-19 from apt.llvm.org. Trixie ships clang-19 natively, so install
it from the distro instead; run-make.sh's prepare() then finds clang-19
and skips llvm.sh entirely. Ubuntu and older Debian releases still have
software-properties-common and keep the previous behavior.
David Galloway [Fri, 10 Jul 2026 20:12:01 +0000 (16:12 -0400)]
script/buildcontainer-setup: fix package install on debian trixie
Debian removed software-properties-common from the archive in trixie,
so the flat apt-get install list fails there. The package was only
needed to provide add-apt-repository for llvm.sh, which installs
clang-19 from apt.llvm.org. Trixie ships clang-19 natively, so install
it from the distro instead; run-make.sh's prepare() then finds clang-19
and skips llvm.sh entirely. Ubuntu and older Debian releases still have
software-properties-common and keep the previous behavior.
Fixes: https://tracker.ceph.com/issues/78111 Signed-off-by: David Galloway <david.galloway@ibm.com>
(cherry picked from commit b5697987ae86f51d7f16a1409ce9f2b542f3de52)
Super User [Wed, 8 Jul 2026 12:05:16 +0000 (17:35 +0530)]
doc/rbd: clarify mirror resync snapshot behavior
Add documentation explaining that rbd mirror resync only copies
image state and data up to the last mirror-snapshot for
snapshot-based mirroring. Document the requirement to create
a new mirror-snapshot on current primary so that new changes
(including resize operation) gets reflected on secondary after
resync.
Karthik U S [Sat, 16 May 2026 03:15:50 +0000 (08:45 +0530)]
qa/test_mirroring: Add tests for mirroring checkpoints
Adding integration tests for validating the cephfs mirroring
checkpoints feature
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit fe7fa7b0761720181cf8735ac7b5f17080c1d8a7)
Karthik U S [Thu, 4 Jun 2026 11:47:05 +0000 (17:17 +0530)]
doc: Add docs and pending release notes for mirroring checkpoints
Adding documentations and pending release notes for the mirroring
checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 12bd34d69ac7e136d14d25ea274e99727b559870)
Karthik U S [Wed, 10 Jun 2026 05:48:25 +0000 (11:18 +0530)]
mgr/mirroring,tools/cephfs_mirror: Handle checkpoint state transition
When a new checkpoint is being added or when the daemon gets restarted,
it will check whether the newly created checkpoint or any other old
checkpoints have already been mirrored onto the remote peer. If so, it
will transition to the correct state by checking for the highest snap id
present on the remote, and setting all the checkpoints which have snap id
lesser than or equal to that of the remote to COMPLETE. This is done by
sending an acquire notification from the mirroring module to the mirror
daemon, which is handled in the add_directory path, by adding the directory
to be checked for the state transition in the tick thread.
This path gets triggered:
a) when the daemon gets restarted
b) when the peer mapping changes
c) by sending the acquire notification from checkpoint add/now CLIs.
d) when mirroring module restarts
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit c64fe9636514d9e9b1c286a0b21b84569e266067)
Karthik U S [Sat, 16 May 2026 00:50:03 +0000 (06:20 +0530)]
tools/cephfs_mirror: update checkpoint status during snapshot sync
When the sync completes or fails for a checkpointed snapshot, transition
the status of that checkpoint from CREATED/FAILED to COMPLETE/FAILED in
the do_sync_snaps() along with the timestamp of the event.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 37163436cc87a2be691202bb565c7b5007df026f)
Karthik U S [Tue, 12 May 2026 23:21:53 +0000 (04:51 +0530)]
tools/cephfs_mirror: Helper functions for mirroring checkpoints
Implementation of helper functions and data structures for the
snapshot based mirroring checkpoints feature.
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 0a394ef438abd0248d078ea72e8ae5ce4f3d5e84)
Fixes: https://tracker.ceph.com/issues/73454 Signed-off-by: Karthik U S <karthik.u.s1@ibm.com>
(cherry picked from commit 959bc2eb07cacf70a4da09170e1c9bbed4b9423f)
Kotresh HR [Tue, 23 Jun 2026 15:13:18 +0000 (20:43 +0530)]
qa: Add mgr snapshot mirror status tests
Add teuthology coverage for `ceph fs snapshot mirror status`:
parity with asok peer_status, default idle metrics, stale omap
handling, error paths, filter scopes, daemon restart, and cache TTL.
Reuse existing peer_dir_status and metrics assertion helpers.
Adjust stale test wait for InstanceWatcher timeout and set a
short cache TTL in the cache test. Pass peer_uuid via --peer_uuid=
in the test helper for peer-only scope queries.
Kotresh HR [Sun, 21 Jun 2026 17:59:43 +0000 (23:29 +0530)]
doc/cephfs: Document fs snapshot mirror status mgr command
Describe the mirroring module command that reads persisted omap
metrics, including syntax, output layout, stale detection, caching,
and comparison with the admin socket peer status interface.
Document --peer_uuid as a named argument for peer-only and
directory/peer filtering.
Kotresh HR [Sun, 21 Jun 2026 17:52:29 +0000 (23:22 +0530)]
mgr/mirroring: make snapshot mirror metrics cache optional
Add snapshot_mirror_metrics_cache_enabled (default true). When disabled,
metrics_status reads omap directly and skips complete and partial caches.
When enabled, behavior is unchanged.
Kotresh HR [Sun, 21 Jun 2026 17:50:28 +0000 (23:20 +0530)]
mgr/mirroring: make snapshot mirror metrics cache TTL configurable
Add snapshot_mirror_metrics_cache_ttl as a runtime mgr/mirroring module
option (default 15 seconds) instead of a hard-coded CACHE_TTL_SECS.
Both complete and partial lru_cache_timeout wrappers read the value
when caching omap metrics so operators can tune cache freshness without
code changes.
Kotresh HR [Sun, 21 Jun 2026 17:48:15 +0000 (23:18 +0530)]
mgr/mirroring: handle missing cephfs_mirror object in metrics status
When snapshot mirroring is not enabled for a filesystem, the
cephfs_mirror RADOS object does not exist and "ceph fs snapshot mirror
status" fails reading sync stat omap. Map rados ENOENT to a clear
MirrorException so the CLI reports that snapshot mirroring must be
enabled instead of a generic omap read failure or an uncaught exception.
Also catch unexpected errors in metrics_status like other mirror CLI
handlers, so the mgr module does not crash on failure. Return
-errno.EINVAL from the generic error path instead of the exception
message as exit code.
Kotresh HR [Sun, 21 Jun 2026 17:28:38 +0000 (22:58 +0530)]
mgr/mirroring: detect stale snapshot mirror sync metrics in omap
Persisted metrics in the cephfs_mirror omap can outlive the writing
daemon when cephfs-mirror stops or a directory is reshuffled to another
instance; mgr would keep reporting stale progress until the owning
daemon writes again.
Extend format_and_order_sync_stat_for_display() to compare persisted
_instance_id against InstanceWatcher live instances (via
FSPolicy.get_live_instance_ids()) and the directory's tracked instance
(via Policy.get_tracked_instance_id()). Mark metrics stale when the
persisted writer is no longer live (any state), or when it does not
match the tracked instance while persisted state is not "idle". Show
state "stale" with current_syncing_snap omitted.
Pass policy and live instance ids through load_sync_stat_metrics() and
fetch_sync_stat_metrics(), and into sync_stat_complete_cache and
sync_stat_partial_cache loaders. Cache hits serve already-formatted
(stale-marked) entries until TTL expiry without re-checking instance
liveness.
Kotresh HR [Tue, 16 Jun 2026 10:42:41 +0000 (16:12 +0530)]
mgr/mirroring: cache fs snapshot mirror status omap metrics
Add a short-lived in-memory cache for metrics returned by
"ceph fs snapshot mirror status", which reads persisted sync stats
from the cephfs_mirror object omap. Omap walks are relatively
expensive; caching reduces repeated reads when the CLI or multiple
clients poll at short intervals.
Two TTL-bucketed LRU caches (CACHE_TTL_SECS) are used instead of one
unified cache. The complete cache always holds every mirrored
directory and peer for a filesystem; the partial cache holds one
directory. A single directory-granularity cache cannot prove it has
all directories, so full-scan queries rely on the complete cache
contract.
Cache implementation (lru_cache_timeout decorator backed by
_TimedLRUCache, not functools.lru_cache):
- lru_cache has no TTL and no peek-on-hit API. Single-directory
queries must read the complete cache without loading on miss;
lru_cache.cache is also unavailable on Python 3.14+.
- complete (sync_stat_complete_cache): full omap prefix scan via
load_sync_stat_metrics. Cache key: (time_token, filesystem).
COMPLETE_CACHE_MAX limits filesystem entries per TTL window.
Bind cache_peek/cache_info/cache_clear via a CachedMethod descriptor
so TTL lookups receive (self, ...); otherwise single-dir status fails
with 'str' object has no attribute 'mgr'.
Serve logic (under the existing lock):
- status <fs> [--peer_uuid=<uuid>]: load complete cache on miss;
filter by peer when requested.
- status <fs> <dir>: peek complete cache (no load on miss); if the
entry contains the directory and peers, serve from complete.
Otherwise load partial cache (omap on miss).
Kotresh HR [Mon, 1 Jun 2026 09:29:31 +0000 (14:59 +0530)]
mgr/mirroring: Reorder and format metrics output
Reorder and format status output to match the output of asok
interface peer_status command. Return metrics under
metrics/<dir>/peer/<uuid> to match the asok peer_status layout,
including {"metrics": {}} when no peers are configured.
mgr/mirroring: Add new interface to expose mirroring metrics
Add the following new interface to expose mirroring metrics
ceph fs snapshot mirror status <fsname> [<mirrored_dir_path>] [--peer_uuid=<peer_uuid>]
The cmd loads the persisted directory sync metrics from the
cephfs_mirror object's omap. Metrics are grouped by mirrored
directory and peer.
When --peer_uuid is specified, only metrics for that peer are
returned. peer_uuid is a named CLI argument (_end_positional_) so
peer-only filtering does not require a mirrored directory path.
Kotresh HR [Sat, 20 Jun 2026 07:58:56 +0000 (13:28 +0530)]
tools/cephfs_mirror: Remove persisted dir stats
When a directory is removed from mirroring, the persisted directory
stats need to be removed. This patch handles the cleanup.
Omap keys must not be removed when mirrored directories are reshuffled
across cephfs-mirror daemons. The mgr release notify now carries a
purging flag (set only during permanent removal, not reshuffle), and
the daemon removes persisted stats only when purging is true. On
reshuffle with an in-progress sync, clear live current_syncing_snap
state and persist idle metrics so the acquiring daemon does not inherit
stale syncing omap entries.
Kotresh HR [Sat, 20 Jun 2026 06:21:42 +0000 (11:51 +0530)]
tools/cephfs_mirror: Load persisted mirror metrics from omap
Load last_synced_snap metadata from the cephfs_mirror object omap on
PeerReplayer initialization and when a mirrored directory is added.
Live current_syncing_snap metrics are not restored; they are rebuilt
when synchronization starts.
When the daemon restarts after a snapshot was synced on the remote but
metrics were not yet written to omap, loaded metadata may belong to an
older snapshot. Add reconcile_last_synced_snap() to compare against
the remote snap map, clear stale last-sync fields, and update
last_synced_snap id/name in memory.
Treat snaps_synced, snaps_deleted, and snaps_renamed as per-session
counters. Do not load them from omap; they start at zero for each
daemon session and are still reported via the admin socket. Persist
omap metrics unconditionally after reconcile so the mgr picks up the
new instance id and cleared session counters on restart.
Kotresh HR [Sat, 20 Jun 2026 05:33:49 +0000 (11:03 +0530)]
tools/cephfs_mirror: Persist metrics to omap
Persist snapshot mirroring metrics to the cephfs_mirror object omap
for the mgr/mirroring module to support status command.
The tick thread only keeps omap up to date for in-progress syncs on
registered directories. Persist explicitly when stats change so omap
is updated before a directory unregisters:
- snap delete/rename propagation (inc_deleted_snap / inc_renamed_snap)
- after each successful snap sync (following set_last_synced_stat)
- after sync_snaps failure (following _inc_failed_count)
- after sync_perms failure (following _inc_failed_count)
Without the explicit call sites, omap can keep stale live or idle state
after sync completes or fails because directories leave m_registered
before the next tick.
Kotresh HR [Sun, 17 May 2026 21:01:58 +0000 (02:31 +0530)]
tools/cephfs_mirror: Adds capability to persist metrics
Adds the capability to persist mirroring metrics.
The metrics are persisted in the omap of the cephfs-mirror
object. Metrics are persisted asynchronously.
Each mirrored directory path stores the corresponding
metrics as the value of a unique omap key representing
the mirrored directory. The omap key is as below.
Kotresh HR [Tue, 16 Jun 2026 17:31:13 +0000 (23:01 +0530)]
qa: Add tests for cephfs_mirror_directory perf counters
Verify counter dump registration, current-sync gauges while syncing
(full/delta), last-sync and summary counters after idle sync, and
extend mirror stats and remote-snap failure tests for per-directory
snaps_* and dir_state.
Describe per-directory labeled perf counters, labels, update behavior,
mapping to peer status, and counter reference tables. Document the
per-peer tick thread and cephfs_mirror_tick_interval, which refreshes
current-sync gauges on each tick. Add a PendingReleaseNotes entry for
the new cephfs_mirror_directory perf counter group.
Kotresh HR [Tue, 16 Jun 2026 16:57:12 +0000 (22:27 +0530)]
cephfs_mirror: update per-directory last sync and summary perf counters
Wire the remaining cephfs_mirror_directory labeled counters that were
registered in the prior commit but not yet refreshed. Live
current_syncing_snap gauges continue to be updated from the per-peer tick
thread; this commit updates last_synced_snap and per-directory snap
summary counters at the points where SnapSyncStat is actually modified.
Depends-on the cephfs_mirror_directory PerfCounters schema (dir_state,
current_*, last_*, snaps_*) added when each mirrored directory is
registered.
These counters appear under "cephfs_mirror_directory" in "counter dump" with
the same labels as current-sync metrics (source_fscid, source_filesystem,
peer_uuid, peer_cluster_name, peer_cluster_filesystem, directory). ceph-exporter
exposes them as e.g. ceph_cephfs_mirror_directory_last_sync_bytes and
ceph_cephfs_mirror_directory_snaps_synced.
Unlike cephfs_mirror_peers, values are per (peer_uuid, directory) rather than
aggregated across all directories on the peer. Peer-level counters are
unchanged.
Kotresh HR [Tue, 16 Jun 2026 17:22:21 +0000 (22:52 +0530)]
tools/cephfs_mirror: Expose per-directory snap metrics via perf counters
Introduce a new labeled perf counter group, cephfs_mirror_directory, so
per-directory snapshot mirror progress can be scraped via "counter dump" and
exported to Prometheus by ceph-exporter (e.g.
ceph_cephfs_mirror_directory_current_sync_bytes).
Design
------
* One PerfCounters instance per mirrored directory on a peer, keyed in
m_directory_perf_counters and registered on the daemon-wide
PerfCountersCollection.
* Labels on each instance (flat counter dump array entries):
- source_fscid, source_filesystem
- peer_uuid, peer_cluster_name, peer_cluster_filesystem
- directory (dir_root, e.g. "/parent/d1")
The peer_uuid label disambiguates the same directory path mirrored to
different peers.
* Counters are created in init() and add_directory(), removed in
remove_directory() and the PeerReplayer destructor.
* Priority follows cephfs_mirror_perf_stats_prio (same as
cephfs_mirror_peers).
Update path
-----------
Live / current_syncing_snap gauges are refreshed from
update_directory_current_sync_perf_counters(), called by
refresh_directory_current_sync_perf_counters() from the per-peer tick
thread (run_tick()). Each cephfs_mirror_tick_interval seconds (default 5)
the tick thread updates counters for each registered (actively syncing)
directory.
All of the following are added to the builder in this commit. Only the
"current sync" and dir_state fields listed under "Updated in this commit"
are written here; last_synced_snap and per-directory snap summary counters
are registered for a follow-up commit that updates them when stats change.
Current syncing snapshot (peer_status "current_syncing_snap")
[Updated in this commit]
current_snap_id - snapshot id being synchronized
current_sync_mode - 0 = full, 1 = delta (snapdiff)
current_read_bps - bytes/sec read (raw, not formatted)
current_write_bps - bytes/sec written
crawl_state - 0 = N/A, 1 = in-progress, 2 = completed
crawl_duration_seconds - crawl duration; in-progress uses now - start
datasync_wait_state - 0 = none, 1 = waiting, 2 = complete
datasync_wait_duration_seconds
current_sync_bytes - bytes synced so far for this snap
current_total_bytes - total bytes for this snap
current_sync_bytes_percent - basis points (1745 = 17.45%)
current_sync_files
current_total_files
current_sync_files_percent - basis points
current_eta_valid - 0 = calculating, 1 = ETA available
current_eta_seconds - ETA in seconds when valid
Per-directory snapshot summary (peer_status snaps_*)
[Registered only; not updated in this commit]
snaps_synced, snaps_deleted, snaps_renamed
Last synced snapshot (peer_status "last_synced_snap")
[Registered only; not updated in this commit]
last_snap_id
last_crawl_duration_seconds
last_datasync_wait_duration_seconds
last_sync_duration_seconds
last_sync_timestamp - utime_t / seconds since epoch
last_sync_bytes
last_sync_files
When idle or failed, current_* counters are zeroed and dir_state reflects
0 or 2 respectively.
Kotresh HR [Tue, 16 Jun 2026 16:38:15 +0000 (22:08 +0530)]
tools/cephfs_mirror: Add per-peer tick thread with configurable interval
Introduce a per-peer tick thread controlled by cephfs_mirror_tick_interval
(default 5 seconds). The interval is re-read each iteration so configuration
changes take effect without restarting the daemon. The thread provides a
generic hook for future periodic mirroring work.
Kotresh HR [Mon, 25 May 2026 18:22:29 +0000 (23:52 +0530)]
doc: Update the mirroring doc with new metrics fields
Update the mirroring documentation and also the
release notes with new metrics introduced and it's
availability via 'fs mirror peer status' asok
interface.
Kotresh HR [Fri, 5 Jun 2026 14:23:14 +0000 (19:53 +0530)]
tools/cephfs_mirror: Nest peer_status metrics by dir path and peer uuid
Restructure peer_status output so mirrored directory paths can be
shared by multiple peers without key collisions. Metrics are grouped
as metrics/<dir_path>/peer/<peer_uuid>/ instead of flat dir keys.
Kotresh HR [Sat, 28 Mar 2026 11:23:33 +0000 (16:53 +0530)]
tools/cephfs_mirror: Add eta metrics
Add estimate time of completion for the current
syncing snapshot. The calculation takes into
account the average read/write throughput from
the start of snapshot sync and not the current
read/write throughput. So the ETA is affected
accordingly.
Kotresh HR [Sat, 28 Mar 2026 10:57:02 +0000 (16:27 +0530)]
tools/cephfs_mirror: Add read/write throughput
The read throughput added measures the bytes
read per second from the source ceph filesystem.
Similarly, the write throughput added measures
the bytes written per second to the remote ceph
filesystem. It's derived from the time spent
in preadv and pwritev calls.
sync-mode:
---------
The 'sync-mode: full/delta' is added to peer status.
The 'delta' means, blockdiff along with snapdiff is
being used to sync the files where as 'full' means
full directory is crawled and each file is synced
entirely.
crawl:
-----
The state can be in-progress/completed. This
identifies whether the crawler thread is done
queuing the files for data sync threads.
The time taken for the duration is also shown.
If the crawl is in-progress, the duration
would show the time taken till then from the
start of the crawl. If the crawl state is
completed, then duration indicates total
time taken for the crawl.
The crawl duration is shown in "d h m s" format.
The existing 'sync_duration' in last_synced_snap
is also formatted
The values are as below. When crawl state is
completed, the 'total_files' metric doesn't
grow anymore.
crawl_duration:
--------------
The crawl_duration of last snapshot is saved in last_synced_snap
section as well.
Kotresh HR [Mon, 16 Feb 2026 10:59:31 +0000 (16:29 +0530)]
tools/cephfs_mirror: Add inprogress bytes and files metric
Add following mirroring progress metrics to current_syncing_snap
as below
bytes:
sync_bytes - bytes synced till now
total_bytes - total bytes to be synced
sync_percent - Percentage of bytes synced till now
files:
total_files - Total files to be synced
sync_files - files synced till now
sync_percent - Percentage of files synced till now
sync_files and sync_bytes are also stored in last_synced_snap section
after the snapshot is synced.
Casey Bodley [Tue, 9 Jun 2026 18:04:53 +0000 (14:04 -0400)]
qa/dnsmasq: use managed dnsmasq instead of editing resolv.conf
this task was manually editing /etc/resolv.conf to configure dns names.
on ubuntu 24, these changes were getting clobbered by NetworkManager or
systemd service restarts
instead, manage dnsmasq depending on host distro. rpm-based distros use
NetworkManager's dnsmasq plugin, while deb-based distros use
systemd-resolved to point at the local dnsmasq
osd/PeeringState: add perf counters for PG rebuild times
Track per-OSD PG rebuild duration in prepare_stats_for_publish()
(primary OSD only). Three new counters are added to the
recoverystate_perf collection:
- pg_rebuild_duration: LONGRUNAVG time counter (sum+count pair);
This counter internally maintains 'avgcount' that tracks the
cumulative number of rebuild events.
- pg_rebuild_max_secs: maximum rebuild duration observed in seconds
- pg_rebuild_min_secs: minimum rebuild duration observed in seconds
The logic uses a per-PG in-memory latch (rebuild_start_time) to
capture the redundancy failure entry point. When a PG is first observed
to be in a vulnerable state (PG_STATE_DEGRADED,
PG_STATE_UNDERSIZED, or num_objects_misplaced/degraded > 0), it
latches info.stats.last_change as the start time, provided
last_change > last_clean, which ensures only genuine new failures
after the prior clean interval are tracked.
On recovery, the rebuild duration is computed as (now - rebuild_start_time)
and is only recorded if delta num_objects_recovered > 0 or the PG had
confirmed redundancy loss at latch time, filtering out spurious state
transitions. The latch is cleared after each recorded event.
The latch is also cleared in clear_primary_state() so that an interval
change or role transition (primary -> replica) does not carry a stale
start time or baseline recovered count into a future interval.
The new last_degraded is intentionally not used here to retain compatibility
with older Ceph branches where the field doesn't exist and requires encoding
changes. A future simplification can replace the latch with a direct
(last_clean - last_degraded) calculation once last_degraded is consistently
available.
This interim solution is a close approximation of the PG rebuild time.
These counters are scraped per-OSD by ceph-exporter and exposed to
Prometheus, enabling durability score calculations over user-defined
time windows.
Other Changes:
1. Add unit tests to TestPeeringState.cc that exercise the latch logic.
2. Add a standalone integration test to verify that the rebuild perf counters
are incremented on the primary OSD after a recovery event.
The mgr dispatch throttle (ms_dispatch_throttle_bytes, default 100MB)
was far smaller than the OSD throttle (mgr_osd_bytes, 512MB),
making the dispatch queue the effective bottleneck on large clusters.
When the dispatch queue fills, OSD connections stall waiting for
throttle space. These connections time out and reconnect, causing
connection churn and heap growth from repeated connection setup/teardown.
The MGR also falls behind on PG stats processing, leading to unknown
and inactive PGs.
Add mgr_dispatch_throttle_bytes (default 1GB) and a
Messenger::set_dispatch_throttle_size() API to configure it at init time.
The new default is sized to accommodate the sum of all per-type policy
throttles, ensuring the dispatch queue is never the bottleneck.
Matthew N Heler [Fri, 19 Jun 2026 23:10:03 +0000 (18:10 -0500)]
mgr: filter root logger fallback
Make the mgr root logger fallback follow debug_mgr and preserve the
original logger name, so third-party DEBUG logs are not emitted by
default or mislabeled as mgr logs.
ceph-volume: skip internal raid mirror LVs in inventory
ceph-volume inventory started including all LVM mapper devices after c06bee965f1. On hosts with raid mirrored system volumes, that pulls in
hidden legs like var_rmeta_0 which have no /dev/vg/lv node and makes
cephadm's ceph-volume inventory call fail.
Skip those internal LVs in get_devices() and avoid rewriting the device
path to a missing lv_path in Device._parse().
Nitzan Mordechai [Tue, 23 Jun 2026 10:23:54 +0000 (10:23 +0000)]
mon/config: trim whitespace in config target
When set\get\rm config values with leading or trailing spaces,
those spaces need to be removed before we are trying to get\set\rm
those keys.
add trim code before each approch and also for exist keys in config.
projects / ceph.git / log