From: Patrick Donnelly <pdonnell@redhat.com>
Date: Tue, 12 Nov 2019 22:50:52 +0000 (-0800)
Subject: global: disable THP for Ceph daemons
X-Git-Tag: v15.1.0~901^2
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=1713f73fedd9ea0b620c9cb27b7b606f5e00a4ea;p=ceph-ci.git

global: disable THP for Ceph daemons

Ceph is known to suffer from memory fragmentation due to transparent
huge pages (THP). This is indicated by RSS usage above configured memory
targets and is only observable when the distribution default for THP is
"always", which is the default in the upstream kernel if
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is unset.

For now, enabling THP is currently discouraged until selective use of
THP by Ceph is implemented via madvise. We will need to consider both
defaults for THP so madvise calls to both enable and disable THP will
need implemented.

All credit to Mark Nelson for doing the legwork identifying this issue
and potential solutions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Mark Nelson <mnelson@redhat.com>
---

diff --git a/src/common/options.cc b/src/common/options.cc
index f20568501cf..051f5e47e3f 100644
--- a/src/common/options.cc
+++ b/src/common/options.cc
@@ -786,6 +786,12 @@ std::vector<Option> get_global_options() {
     .set_flag(Option::FLAG_NO_MON_UPDATE)
     .set_description(""),
 
+    Option("thp", Option::TYPE_BOOL, Option::LEVEL_DEV)
+    .set_default(false)
+    .set_flag(Option::FLAG_STARTUP)
+    .set_description("enable transparent huge page (THP) support")
+    .set_long_description("Ceph is known to suffer from memory fragmentation due to THP use. This is indicated by RSS usage above configured memory targets. Enabling THP is currently discouraged until selective use of THP by Ceph is implemented."),
+
     Option("key", Option::TYPE_STR, Option::LEVEL_ADVANCED)
     .set_default("")
     .set_description("Authentication key")
diff --git a/src/global/global_init.cc b/src/global/global_init.cc
index 6962f42a543..dff97e5c7ee 100644
--- a/src/global/global_init.cc
+++ b/src/global/global_init.cc
@@ -312,6 +312,11 @@ global_init(const std::map<std::string,std::string> *defaults,
   if (prctl(PR_SET_DUMPABLE, 1) == -1) {
     cerr << "warning: unable to set dumpable flag: " << cpp_strerror(errno) << std::endl;
   }
+#  if defined(PR_SET_THP_DISABLE)
+  if (!g_conf().get_val<bool>("thp") && prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0) == -1) {
+    cerr << "warning: unable to disable THP: " << cpp_strerror(errno) << std::endl;
+  }
+#  endif
 #endif
 
   //