os/bluestore/BlueRocksEnv: Avoid flushing too much data at once.

Although, in _flush func we already check length. If length of dirty
is less then bluefs_min_flush_size, we will skip this flush.
But in fact, we found rocksdb can call many times Append() and then
call Flush(). This make flush_data is much larger than
bluefs_min_flush_size.

From my test, w/o this patch, it can reduce 99.99% latency(from
145.753ms to 20.474ms) for 4k randwrite with bluefs_buffered_io=true.

Because Bluefs::flush acquire lock. So we add new api try_flush
to avoid lock contention.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit 3a1d4ff6dcdcd897d31a85ef82c726599624b66d)

diff --git a/src/os/bluestore/BlueFS.h b/src/os/bluestore/BlueFS.h
index 14fca0fb5b3c8d73b38abc50b5445af3604871ad..90db5b743dc340c98a4a3569ba3cfa92e9b55e8a 100644 (file)
--- a/src/os/bluestore/BlueFS.h
+++ b/src/os/bluestore/BlueFS.h
@@ -560,6 +560,12 @@ public:
     int r = _flush(h, force, l);
     ceph_assert(r == 0);
   }
+  void try_flush(FileWriter *h) {
+    h->buffer_appender.flush();
+    if (h->buffer.length() >= cct->_conf->bluefs_min_flush_size) {
+      flush(h, true);
+    }
+  }
   void flush_range(FileWriter *h, uint64_t offset, uint64_t length) {
     std::lock_guard l(lock);
     _flush_range(h, offset, length);

diff --git a/src/os/bluestore/BlueRocksEnv.cc b/src/os/bluestore/BlueRocksEnv.cc
index b48e0678daa73dc1ceb7e3404222cadbf10102a4..5464a51061abc22ab470cba72505ad437891de4a 100644 (file)
--- a/src/os/bluestore/BlueRocksEnv.cc
+++ b/src/os/bluestore/BlueRocksEnv.cc
@@ -172,6 +172,9 @@ class BlueRocksWritableFile : public rocksdb::WritableFile {
 
   rocksdb::Status Append(const rocksdb::Slice& data) override {
     h->append(data.data(), data.size());
+    // Avoid calling many time Append() and then calling Flush().
+    // Especially for buffer mode, flush much data will cause jitter.
+    fs->try_flush(h);
     return rocksdb::Status::OK();
   }