From: Jos Collin <jcollin@redhat.com>
Date: Wed, 1 Jun 2022 12:06:42 +0000 (+0530)
Subject: doc: update troubleshooting and mds-config-ref docs for In-memory Log Dump
X-Git-Tag: v18.0.0~281^2
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=b2d82f338acb600ecf318c05ff1bae3fbcfb6093;p=ceph.git

doc: update troubleshooting and mds-config-ref docs for In-memory Log Dump

* Added 'mds_extraordinary_events_dump_interval' to  mds-config-ref.rst.
* Updated troubleshooting.rst with 'In-memory Log Dump' section.

Fixes: https://tracker.ceph.com/issues/40633
Signed-off-by: Jos Collin <jcollin@redhat.com>
---

diff --git a/doc/cephfs/mds-config-ref.rst b/doc/cephfs/mds-config-ref.rst
index 117d20572642..a5e0bba91775 100644
--- a/doc/cephfs/mds-config-ref.rst
+++ b/doc/cephfs/mds-config-ref.rst
@@ -62,3 +62,4 @@
 .. confval:: mds_skip_ino
 .. confval:: mds_min_caps_per_client
 .. confval:: mds_symlink_recovery
+.. confval:: mds_extraordinary_events_dump_interval
diff --git a/doc/cephfs/troubleshooting.rst b/doc/cephfs/troubleshooting.rst
index ea33019f0436..59927215c4cc 100644
--- a/doc/cephfs/troubleshooting.rst
+++ b/doc/cephfs/troubleshooting.rst
@@ -188,6 +188,40 @@ You can enable dynamic debug against the CephFS module.
 
 Please see: https://github.com/ceph/ceph/blob/master/src/script/kcon_all.sh
 
+In-memory Log Dump
+==================
+
+In-memory logs can be dumped by setting ``mds_extraordinary_events_dump_interval``
+during a lower level debugging (log level < 10). ``mds_extraordinary_events_dump_interval``
+is the interval in seconds for dumping the recent in-memory logs when there is an Extra-Ordinary event.
+
+The Extra-Ordinary events are classified as:
+
+* Client Eviction
+* Missed Beacon ACK from the monitors
+* Missed Internal Heartbeats
+
+In-memory Log Dump is disabled by default to prevent log file bloat in a production environment.
+The below commands consecutively enables it::
+
+  $ ceph config set mds debug_mds <log_level>/<gather_level>
+  $ ceph config set mds mds_extraordinary_events_dump_interval <seconds>
+
+The ``log_level`` should be < 10 and ``gather_level`` should be >= 10 to enable in-memory log dump.
+When it is enabled, the MDS checks for the extra-ordinary events every
+``mds_extraordinary_events_dump_interval`` seconds and if any of them occurs, MDS dumps the
+in-memory logs containing the relevant event details in ceph-mds log.
+
+.. note:: For higher log levels (log_level >= 10) there is no reason to dump the In-memory Logs and a
+          lower gather level (gather_level < 10) is insufficient to gather In-memory Logs. Thus a
+          log level >=10 or a gather level < 10 in debug_mds would prevent enabling the In-memory Log Dump.
+          In such cases, when there is a failure it's required to reset the value of
+          mds_extraordinary_events_dump_interval to 0 before enabling using the above commands.
+
+The In-memory Log Dump can be disabled using::
+
+  $ ceph config set mds mds_extraordinary_events_dump_interval 0
+
 Reporting Issues
 ================