crimson/os/seastore: yield to user IO between cleaner cycles

After the deadlock fix in the preceding commit ("fix IO-block deadlock
when cleaner is sleeping"), the cleaner stays awake while user IO is
blocked, but a second symptom appears at high alive_ratio (~0.79): the
cleaner's segment-allocate-and-fill loop runs tightly enough that the
user-IO continuation scheduled by maybe_wake_blocked_io() never gets a
chance to retry try_reserve_io() before the cleaner consumes the
projected_avail headroom again on its next iteration. User IO wakes,
sees projected_avail still below hard_limit, re-blocks immediately.

In the qa/standalone/crimson randwrite bench this manifests as: cluster
makes 500-700 GB of progress, then user_written counter freezes for
~75 seconds (watchdog window) while the cleaner is fully busy.

In BackgroundProcess::run(), after each do_background_cycle, if user IO
is currently blocked, yield to the reactor. That gives the woken
user-IO continuation a chance to slot in and complete a reservation
before the cleaner starts its next reservation-consuming cycle.

With this change, the same bench runs 19 minutes (vs 11-16 min) and
writes 785 GB user (vs 506-692 GB) before the next cluster limit hits,
which is the inherent throughput cap at alive_ratio 0.79 where each
reclaim only frees ~21% of segment size — not a coordination bug.

Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>

diff --git a/src/crimson/os/seastore/extent_placement_manager.cc b/src/crimson/os/seastore/extent_placement_manager.cc
index c579b5fd14fedae55f142a87cf24bf636a3ee1db..928b8d37966cf65c6e1628b5ddcb6bf8ad40b9d9 100644 (file)
--- a/src/crimson/os/seastore/extent_placement_manager.cc
+++ b/src/crimson/os/seastore/extent_placement_manager.cc
@@ -807,6 +807,11 @@ ExtentPlacementManager::BackgroundProcess::maybe_wake_blocked_io()
     DEBUG("");
     blocking_io->set_value();
     blocking_io = std::nullopt;
+    // Remember that we just woke a blocked IO; run() yields once on
+    // this edge so the woken continuation has a chance to retry the
+    // reservation before the cleaner spins another cycle and consumes
+    // the projected_avail headroom we just freed.
+    pending_user_io_wake = true;
   }
 }
 
@@ -818,6 +823,12 @@ ExtentPlacementManager::BackgroundProcess::run()
     if (background_should_run()) {
       log_state("run(background)");
       co_await do_background_cycle();
+      // Edge-triggered: yield only when a blocked IO was actually woken, so the
+      // resumed continuation retries try_reserve_io() before the next cycle runs.
+      if (pending_user_io_wake) {
+        pending_user_io_wake = false;
+        co_await seastar::yield();
+      }
     } else {
       log_state("run(block)");
       assert(!blocking_background);

diff --git a/src/crimson/os/seastore/extent_placement_manager.h b/src/crimson/os/seastore/extent_placement_manager.h
index 123826f16be77d5f7f28200e5b64259934b28c2e..df32bcd7ba182f2571f6af153b6241030b7074ab 100644 (file)
--- a/src/crimson/os/seastore/extent_placement_manager.h
+++ b/src/crimson/os/seastore/extent_placement_manager.h
@@ -1124,6 +1124,11 @@ private:
     std::optional<seastar::future<>> process_join;
     std::optional<seastar::promise<>> blocking_background;
     std::optional<seastar::promise<>> blocking_io;
+    // Set by maybe_wake_blocked_io() whenever it actually unblocks a
+    // user IO; consumed by run() to yield exactly once on that edge,
+    // giving the woken continuation a chance to retry the reservation
+    // before the next background cycle.
+    bool pending_user_io_wake = false;
     bool is_running_until_halt = false;
     state_t state = state_t::STOP;
     eviction_state_t eviction_state;