rgw: Fix ops logs sometimes having several entries per line.

Although not explicitly documented, the RGW ops log is generally
formatted with one entry per line. This makes it work well with log
shipping/ingestion services (many of which default to treating each line
as a separate entry) and in particular works well with log servers that
index based on the available JSON fields.

The current implementation separates the log call from the resulting
disk IO: Logs are written to a buffer and a separate thread flushes them
to disk (possibly in batches). The current code appends a newline only
at the end of the batch being flushed to disk and in many cases when
under load this means that several log entries are concatenated onto a
single line, which complicates attempts to process those logs.

This PR separates the addition of a new line from the flush to disk,
appending a newline after every log entry but still only flushing at the
end of the batch to avoid additional IO overhead.

Fixes: https://tracker.ceph.com/issues/76566
Signed-off-by: Jacques Heunis <jheunis@bloomberg.net>

diff --git a/src/rgw/rgw_log.cc b/src/rgw/rgw_log.cc
index 27580cce2678f3d1ca2561b44d3a1e0f9bbd9439..ea849ea2e6eb897ab6714835f80a9c2fffc06e53 100644 (file)
--- a/src/rgw/rgw_log.cc
+++ b/src/rgw/rgw_log.cc
@@ -420,12 +420,13 @@ void OpsLogFile::flush()
         std::this_thread::sleep_for(std::chrono::seconds(sleep_time_secs));
         try_num++;
       } else {
+        file << '\n';
         break;
       }
     }
   }
   flush_buffer.clear();
-  file << std::endl;
+  file.flush();
 }
 
 void* OpsLogFile::entry() {