osd/PeeringState: add perf counters for PG rebuild times
Track per-OSD PG rebuild duration in prepare_stats_for_publish()
(primary OSD only). Three new counters are added to the
recoverystate_perf collection:
- pg_rebuild_duration: LONGRUNAVG time counter (sum+count pair);
This counter internally maintains 'avgcount' that tracks the
cumulative number of rebuild events.
- pg_rebuild_max_secs: maximum rebuild duration observed in seconds
- pg_rebuild_min_secs: minimum rebuild duration observed in seconds
The logic uses a per-PG in-memory latch (rebuild_start_time) to
capture the redundancy failure entry point. When a PG is first observed
to be in a vulnerable state (PG_STATE_DEGRADED,
PG_STATE_UNDERSIZED, or num_objects_misplaced/degraded > 0), it
latches info.stats.last_change as the start time, provided
last_change > last_clean, which ensures only genuine new failures
after the prior clean interval are tracked.
On recovery, the rebuild duration is computed as (now - rebuild_start_time)
and is only recorded if delta num_objects_recovered > 0 or the PG had
confirmed redundancy loss at latch time, filtering out spurious state
transitions. The latch is cleared after each recorded event.
The latch is also cleared in clear_primary_state() so that an interval
change or role transition (primary -> replica) does not carry a stale
start time or baseline recovered count into a future interval.
The new last_degraded is intentionally not used here to retain compatibility
with older Ceph branches where the field doesn't exist and requires encoding
changes. A future simplification can replace the latch with a direct
(last_clean - last_degraded) calculation once last_degraded is consistently
available.
This interim solution is a close approximation of the PG rebuild time.
These counters are scraped per-OSD by ceph-exporter and exposed to
Prometheus, enabling durability score calculations over user-defined
time windows.
Other Changes:
1. Add unit tests to TestPeeringState.cc that exercise the latch logic.
2. Add a standalone integration test to verify that the rebuild perf counters
are incremented on the primary OSD after a recovery event.
projects / ceph.git / commit