crimson/os/seastore: fix IO-block deadlock when cleaner is sleeping

Two coordinated changes that together close a stall observed at high
alive_ratio in the qa/standalone/crimson randwrite bench (one OSD
frozen for 70+ minutes, alive_ratio ~0.79, projected_avail_ratio ~0.10,
slow_ops accumulating indefinitely).

1. SegmentCleaner::should_clean_space() used segments.get_available_ratio()
   (actual ratio) while should_block_io_on_clean() used
   get_projected_available_ratio() (actual minus in-flight reservations).
   When the actual ratio sat just above available_ratio_hard_limit but
   the projected ratio dipped below it, IO would block while the cleaner
   slept. Make should_clean_space() also trip on the projected ratio.

2. BackgroundProcess::reserve_projected_usage() did not wake the
   background process when an IO blocked. With the cleaner asleep and
   all IO blocked, nothing called maybe_wake_blocked_io() (no
   release_projected_usage runs without completing IO; no segment
   release runs without the cleaner). Kick do_wake_background() at the
   point of blocking, so the cleaner re-evaluates and runs.

Signed-off-by: Shai Fultheim <shai.fultheim@gmail.com>

diff --git a/src/crimson/os/seastore/async_cleaner.h b/src/crimson/os/seastore/async_cleaner.h
index a0ca2d1a056f2693b9fbc38d25dd495da3d2dcde..cd3ee9e1a3aa160bf0ae832cd9e6236f144775fd 100644 (file)
--- a/src/crimson/os/seastore/async_cleaner.h
+++ b/src/crimson/os/seastore/async_cleaner.h
@@ -1433,9 +1433,12 @@ public:
       return false;
     }
     auto aratio = segments.get_available_ratio();
+    auto projected_aratio = get_projected_available_ratio();
     auto rratio = get_reclaim_ratio();
+    // should_block_io_on_clean() uses projected ratio; mirror that here so the
+    // cleaner wakes whenever IO would block, not only when actual space is low.
     return (
-      (aratio < config.available_ratio_hard_limit) ||
+      (projected_aratio < config.available_ratio_hard_limit) ||
       ((aratio < config.available_ratio_gc_max) &&
        (rratio > config.reclaim_ratio_gc_threshold))
     );

diff --git a/src/crimson/os/seastore/extent_placement_manager.cc b/src/crimson/os/seastore/extent_placement_manager.cc
index bfb0a3232a2eb504e94f80036082f8ddf628caaa..c579b5fd14fedae55f142a87cf24bf636a3ee1db 100644 (file)
--- a/src/crimson/os/seastore/extent_placement_manager.cc
+++ b/src/crimson/os/seastore/extent_placement_manager.cc
@@ -755,8 +755,14 @@ ExtentPlacementManager::BackgroundProcess::reserve_projected_usage(
     ++stats.io_blocked_count;
     stats.io_blocked_sum += stats.io_blocking_num;
 
-    blocking_io = seastar::promise<>();
     auto begin_time = seastar::lowres_system_clock::now();
+    // IO blocked -> needs cleaner -> cleaner sleeping -> nothing runs -> deadlock.
+    // Kick the background so it can free space and call maybe_wake_blocked_io().
+    auto arm_blocking_io_and_wake = [this] {
+      blocking_io = seastar::promise<>();
+      do_wake_background();
+    };
+    arm_blocking_io_and_wake();
     // we just blocked this IO, now wait until
     // maybe_wake_blocked_io will set value to blocking_io
     do {
@@ -784,7 +790,7 @@ ExtentPlacementManager::BackgroundProcess::reserve_projected_usage(
           if (!res.cleaner_result.is_successful()) {
           ++stats.io_retried_blocked_count_clean;
         }
-        blocking_io = seastar::promise<>();
+        arm_blocking_io_and_wake();
       }
     } while (blocking_io);
   }