doc: update troubleshooting and mds-config-ref docs for In-memory Log Dump

* Added 'mds_extraordinary_events_dump_interval' to  mds-config-ref.rst.
* Updated troubleshooting.rst with 'In-memory Log Dump' section.

Fixes: https://tracker.ceph.com/issues/40633
Signed-off-by: Jos Collin <jcollin@redhat.com>

diff --git a/doc/cephfs/mds-config-ref.rst b/doc/cephfs/mds-config-ref.rst
index 117d205726426c0e0f4a8824b85b5847e09f7299..a5e0bba91775766679568e2265e0eb34d2a6bdb9 100644 (file)
--- a/doc/cephfs/mds-config-ref.rst
+++ b/doc/cephfs/mds-config-ref.rst
@@ -62,3 +62,4 @@
 .. confval:: mds_skip_ino
 .. confval:: mds_min_caps_per_client
 .. confval:: mds_symlink_recovery
+.. confval:: mds_extraordinary_events_dump_interval

diff --git a/doc/cephfs/troubleshooting.rst b/doc/cephfs/troubleshooting.rst
index ea33019f04367a6402d2358eabac1092b5b75c5d..59927215c4ccb133c27c5e1bc620d59fcdcf8a17 100644 (file)
--- a/doc/cephfs/troubleshooting.rst
+++ b/doc/cephfs/troubleshooting.rst
@@ -188,6 +188,40 @@ You can enable dynamic debug against the CephFS module.
 
 Please see: https://github.com/ceph/ceph/blob/master/src/script/kcon_all.sh
 
+In-memory Log Dump
+==================
+
+In-memory logs can be dumped by setting ``mds_extraordinary_events_dump_interval``
+during a lower level debugging (log level < 10). ``mds_extraordinary_events_dump_interval``
+is the interval in seconds for dumping the recent in-memory logs when there is an Extra-Ordinary event.
+
+The Extra-Ordinary events are classified as:
+
+* Client Eviction
+* Missed Beacon ACK from the monitors
+* Missed Internal Heartbeats
+
+In-memory Log Dump is disabled by default to prevent log file bloat in a production environment.
+The below commands consecutively enables it::
+
+  $ ceph config set mds debug_mds <log_level>/<gather_level>
+  $ ceph config set mds mds_extraordinary_events_dump_interval <seconds>
+
+The ``log_level`` should be < 10 and ``gather_level`` should be >= 10 to enable in-memory log dump.
+When it is enabled, the MDS checks for the extra-ordinary events every
+``mds_extraordinary_events_dump_interval`` seconds and if any of them occurs, MDS dumps the
+in-memory logs containing the relevant event details in ceph-mds log.
+
+.. note:: For higher log levels (log_level >= 10) there is no reason to dump the In-memory Logs and a
+          lower gather level (gather_level < 10) is insufficient to gather In-memory Logs. Thus a
+          log level >=10 or a gather level < 10 in debug_mds would prevent enabling the In-memory Log Dump.
+          In such cases, when there is a failure it's required to reset the value of
+          mds_extraordinary_events_dump_interval to 0 before enabling using the above commands.
+
+The In-memory Log Dump can be disabled using::
+
+  $ ceph config set mds mds_extraordinary_events_dump_interval 0
+
 Reporting Issues
 ================