]> git-server-git.apps.pok.os.sepia.ceph.com Git - ceph.git/commit
osd/PeeringState: add perf counters for PG rebuild times 69962/head
authorSridhar Seshasayee <sridhar.seshasayee@ibm.com>
Wed, 6 May 2026 15:11:33 +0000 (20:41 +0530)
committerSridhar Seshasayee <sridhar.seshasayee@ibm.com>
Mon, 6 Jul 2026 14:10:55 +0000 (19:40 +0530)
commitfa412570af0f91620dea474495fe01d7e4efb881
treeb8e4caf15e2ed8bd7f25be1aa81a1601e3f2849f
parent7c8dae21fa0400abee8da5011a76f3b747dec84e
osd/PeeringState: add perf counters for PG rebuild times

Track per-OSD PG rebuild duration in prepare_stats_for_publish()
(primary OSD only). Three new counters are added to the
recoverystate_perf collection:

 - pg_rebuild_duration: LONGRUNAVG time counter (sum+count pair);
   This counter internally maintains 'avgcount' that tracks the
   cumulative number of rebuild events.
 - pg_rebuild_max_secs: maximum rebuild duration observed in seconds
 - pg_rebuild_min_secs: minimum rebuild duration observed in seconds

The logic uses a per-PG in-memory latch (rebuild_start_time) to
capture the redundancy failure entry point. When a PG is first observed
to be in a vulnerable state (PG_STATE_DEGRADED,
PG_STATE_UNDERSIZED, or num_objects_misplaced/degraded > 0), it
latches info.stats.last_change as the start time, provided
last_change > last_clean, which ensures only genuine new failures
after the prior clean interval are tracked.

On recovery, the rebuild duration is computed as (now - rebuild_start_time)
and is only recorded if delta num_objects_recovered > 0 or the PG had
confirmed redundancy loss at latch time, filtering out spurious state
transitions. The latch is cleared after each recorded event.

The latch is also cleared in clear_primary_state() so that an interval
change or role transition (primary -> replica) does not carry a stale
start time or baseline recovered count into a future interval.

The new last_degraded is intentionally not used here to retain compatibility
with older Ceph branches where the field doesn't exist and requires encoding
changes. A future simplification can replace the latch with a direct
(last_clean - last_degraded) calculation once last_degraded is consistently
available.

This interim solution is a close approximation of the PG rebuild time.

These counters are scraped per-OSD by ceph-exporter and exposed to
Prometheus, enabling durability score calculations over user-defined
time windows.

Other Changes:
1. Add unit tests to TestPeeringState.cc that exercise the latch logic.
2. Add a standalone integration test to verify that the rebuild perf counters
   are incremented on the primary OSD after a recovery event.

Fixes: https://tracker.ceph.com/issues/77493
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
(cherry picked from commit 4d29f4e4021081f19a304dea6938b6f4350eb748)
qa/standalone/osd/osd-recovery-stats.sh
src/osd/PeeringState.cc
src/osd/PeeringState.h
src/osd/osd_perf_counters.cc
src/osd/osd_perf_counters.h
src/test/osd/TestPeeringState.cc