git-server-git.apps.pok.os.sepia.ceph.com Git

]> git-server-git.apps.pok.os.sepia.ceph.com Git - ceph.git/log

Shai Fultheim [Sun, 17 May 2026 09:40:44 +0000 (12:40 +0300)]

crimson/os/seastore: auto-tune cleaner gc segment pick under random-write

SegmentCleaner uses one of three configurable gc formulas to select
the next segment to reclaim: GREEDY (lowest util wins), COST_BENEFIT
((1-u)*age/(2u)), or BENEFIT (an age-weighted quadratic in util).
COST_BENEFIT is the default and the right choice for journaling /
LIFO workloads, where old segments accumulate more dead bytes than
young ones — age predicts deadness, so an old high-util segment is
worth reclaiming because its util will keep rising as long as we wait.

That assumption breaks under random-write at high cluster fill. Dead
bytes spread uniformly across segments regardless of age, so age stops
predicting future deadness, and (1-u)/(2u) becomes the only term that
distinguishes candidates. With every segment in the 0.7-0.94 util
band, (1-u)/(2u) ranges from 0.227 to 0.032 — a 7x spread the formula
can easily lose to a 7x age difference. Result: a 0.94-util old
segment scores higher than a 0.68-util young one, even though
reclaiming the 0.68 segment would free 5x more space (32% of a
64 MB segment vs 6%).

Observed in qa/standalone/crimson randwrite at ~70% full: with the
unmodified formula, cleaner picks settled on 0.92-0.94 util segments
freeing ~4 MB net each; net free rate collapsed to single-digit KB/s
even though the cleaner was running cycles at ~30 µs each. fio's
stall watchdog killed the bench after 535 GB user written (target
1280 GB). Switching gc_formula = greedy by hand let the bench
complete the target.

This patch detects the mis-selection at runtime and overrides the
formula's pick with the greedy choice only when the difference is
significant. In get_next_reclaim_segment() we already iterate all
closed reclaimable segments to find the formula's max-score candidate;
in the same pass we now also track the lowest-util candidate (what
GREEDY would have picked). After the loop, if greedy's free-fraction
(1 - greedy_util) is at least seastore_segment_cleaner_gc_autotune_ratio
times the formula's pick's free-fraction (default 2.0), we swap to
greedy. Since all segments share the same size, comparing free-
fractions is equivalent to comparing freed bytes; the fraction form
avoids an unnecessary multiplication.

The full design rationale (regime-by-regime behaviour, safety guard
against picked_free near zero, score-recompute on override, threshold
calibration) lives in doc/dev/crimson/seastore.rst under the new
"Cleaner GC autotune" section. The code references it from short
inline comments.

Configurable knobs:

  * seastore_segment_cleaner_gc_autotune (bool, default true) —
    operators can disable the override entirely to honor the
    configured formula unconditionally. Ignored when gc_formula =
    greedy.

  * seastore_segment_cleaner_gc_autotune_ratio (float, default 2.0,
    min 1.0) — operators can tune the override threshold. Higher is
    more conservative (preserves age weighting more aggressively);
    lower is more aggressive (behaviour converges toward pure greedy).

The override predicate is factored into a static helper
`SegmentCleaner::should_override_to_greedy(picked_free, greedy_free,
ratio)` so the call site stays readable and the predicate is
independently testable.

With this change the qa/standalone/crimson randwrite bench at 70%
fill completes the target run rather than stalling at the 500-600 GB
mark, with the override firing reliably under high uniform alive_
ratio and not firing under low or non-uniform alive_ratio. Override
behaviour can be observed with debug_seastore_cleaner=20.

Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>