crimson/os/seastore/journal: dispatch records greedily while io slots are free

`RecordSubmitter` dispatches a small record (or the pending batch) only
once the journal is fully drained (`state == IDLE`): `submit()` takes
the direct-write path only at IDLE, and `decrement_io_with_flush()`
flushes the pending batch only after *all* outstanding io has
completed.  The remaining dispatch triggers never fire for 4K-class
records: `preferred_fullness` measures the padding ratio of a single
block, and small records do not reach the batch capacity.

At low-to-mid client queue depth this degenerates into strict
alternation: one journal io in flight, each batch carrying only the
records that arrived during the previous device write.  Measured with
4K randwrite QD20 against a single crimson OSD (circular-bounded
journal): io_depth 1.07, 1.9 records/io, ~2.2ms `submit_record`
latency -- 1.5x the raw device write+flush latency, since each record
first waits ~0.5x for the in-flight io to drain.  `io_depth_limit`
(default 5) is never approached.

Dispatch whenever the device has a free io slot instead:

- `submit()`: take the direct-write path whenever `state != FULL`;
  records are written greedily while slots are free, and batching
  kicks in once `io_depth_limit` is reached.
- `decrement_io_with_flush()`: flush the pending batch into the freed
  slot instead of waiting for IDLE.

Ordering is unaffected: a batch's `write_base` is assigned at dispatch
from `journal_allocator.get_written_to()`, so the on-disk layout
follows submission order; completions are reported in order via the
`device_submission` exit turnstile and the exclusive `finalize` phase,
so a record is never acked before all earlier records are durable.
Replay stops at the first invalid record, and anything beyond such a
hole was never acked.  The same concurrency already occurs through the
`SUBMIT_FULL` path under high load; this extends it to the
low-fullness regime.

## Benchmark

Ceph 21.3.0 dev (RelWithDebInfo), single crimson OSD, 4 reactors
pinned to 4 cores, fio rbd engine on 4 disjoint cores, size-1 pool,
24 GiB prefilled image, consumer NVMe without PLP (~0.7ms fdatasync).

4K randwrite QD20, 180s, `RANDOM_BLOCK_SSD` (circular-bounded journal):

| metric                  | before | after         |
|-------------------------|-------:|--------------:|
| IOPS                    |  5,800 | 10,918 (+88%) |
| avg client latency      |  3.4ms | 1.83ms        |
| avg journal io_depth    |   1.07 | 3.18          |
| records per io          |    1.9 | 1.01          |
| `submit_record` latency |  2.2ms | 1.05ms        |

4K randwrite, 45s per queue depth, `RANDOM_BLOCK_SSD`:

| iodepth |   1 |     4 |    16 |      64 |
|---------|----:|------:|------:|--------:|
| before  | 452 | 1,270 | 3,470 | ~10,300 |
| after   | 536 | 1,908 | 9,337 |  15,333 |

4K randwrite QD20, 180s, `SEGMENTED` (segmented journal and ool
writer, both built on `RecordSubmitter`):

| metric               | before | after         |
|----------------------|-------:|--------------:|
| IOPS                 |  2,728 | 6,123 (+124%) |
| avg client latency   |  7.3ms | 3.26ms        |
| avg journal io_depth |     ~1 | 2.26          |

A classic BlueStore OSD on the same 4-core/QD20 harness does 10,243
IOPS; crimson previously trailed it by 1.8x and now matches it.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

diff --git a/src/crimson/os/seastore/journal/record_submitter.cc b/src/crimson/os/seastore/journal/record_submitter.cc
index b6d22de97e688cd05494b5c7dcfd56259e189656..7fdac752b454eac4d368afe530e4d422883b0b2e 100644 (file)
--- a/src/crimson/os/seastore/journal/record_submitter.cc
+++ b/src/crimson/os/seastore/journal/record_submitter.cc
@@ -308,7 +308,9 @@ RecordSubmitter::submit(
   auto eval = p_current_batch->evaluate_submit(
       record.size, journal_allocator.get_block_size());
   bool needs_flush = (
-      state == state_t::IDLE ||
+      // dispatch greedily while the device has free io slots; batching
+      // kicks in only once io_depth_limit is reached
+      state != state_t::FULL ||
       eval.submit_size.get_fullness() > preferred_fullness ||
       // RecordBatch::needs_flush()
       eval.is_full ||
@@ -507,12 +509,11 @@ void RecordSubmitter::decrement_io_with_flush()
     ceph_assert(!wait_unfull_flush_promise.has_value());
   }
 
+  // dispatch the pending batch into the freed io slot instead of
+  // waiting for the journal to fully drain (state == IDLE)
   auto needs_flush = (
-      !p_current_batch->is_empty() && (
-        state == state_t::IDLE ||
-        p_current_batch->get_submit_size().get_fullness() > preferred_fullness ||
-        p_current_batch->needs_flush()
-      ));
+      !p_current_batch->is_empty() &&
+      state != state_t::FULL);
   if (needs_flush) {
     DEBUG("{} flush", get_name());
     flush_current_batch();