os/bluestore/BlueRocksEnv: Avoid flushing too much data at once.

Although, in _flush func we already check length. If length of dirty
is less then bluefs_min_flush_size, we will skip this flush.
But in fact, we found rocksdb can call many times Append() and then
call Flush(). This make flush_data is much larger than
bluefs_min_flush_size.

From my test, w/o this patch, it can reduce 99.99% latency(from
145.753ms to 20.474ms) for 4k randwrite with bluefs_buffered_io=true.

Because Bluefs::flush acquire lock. So we add new api try_flush
to avoid lock contention.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit 3a1d4ff6dcdcd897d31a85ef82c726599624b66d)

diff --git a/src/os/bluestore/BlueFS.h b/src/os/bluestore/BlueFS.h
index 86748e4afb0964c59d61c2b512484171969c926c..2b7ccfafce20608956bf739a34f88be79eeef698 100644 (file)
--- a/src/os/bluestore/BlueFS.h
+++ b/src/os/bluestore/BlueFS.h
@@ -557,6 +557,12 @@ public:
     int r = _flush(h, force, l);
     ceph_assert(r == 0);
   }
+  void try_flush(FileWriter *h) {
+    h->buffer_appender.flush();
+    if (h->buffer.length() >= cct->_conf->bluefs_min_flush_size) {
+      flush(h, true);
+    }
+  }
   void flush_range(FileWriter *h, uint64_t offset, uint64_t length) {
     std::lock_guard l(lock);
     _flush_range(h, offset, length);

diff --git a/src/os/bluestore/BlueRocksEnv.cc b/src/os/bluestore/BlueRocksEnv.cc
index 51614c09d2cd61a736b8e4eb4585e6f44543f86c..b4703ae2c2a9af2e55e853aaa9ec044b72dbf355 100644 (file)
--- a/src/os/bluestore/BlueRocksEnv.cc
+++ b/src/os/bluestore/BlueRocksEnv.cc
@@ -172,6 +172,9 @@ class BlueRocksWritableFile : public rocksdb::WritableFile {
 
   rocksdb::Status Append(const rocksdb::Slice& data) override {
     h->append(data.data(), data.size());
+    // Avoid calling many time Append() and then calling Flush().
+    // Especially for buffer mode, flush much data will cause jitter.
+    fs->try_flush(h);
     return rocksdb::Status::OK();
   }