global: disable THP for Ceph daemons

Ceph is known to suffer from memory fragmentation due to transparent
huge pages (THP). This is indicated by RSS usage above configured memory
targets and is only observable when the distribution default for THP is
"always", which is the default in the upstream kernel if
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is unset.

For now, enabling THP is currently discouraged until selective use of
THP by Ceph is implemented via madvise. We will need to consider both
defaults for THP so madvise calls to both enable and disable THP will
need implemented.

All credit to Mark Nelson for doing the legwork identifying this issue
and potential solutions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Mark Nelson <mnelson@redhat.com>

diff --git a/src/common/options.cc b/src/common/options.cc
index f20568501cf2ded256a1f07a5abdfcbf6f018346..051f5e47e3fd396fe1cc1ab14c5398b83e29fa56 100644 (file)
--- a/src/common/options.cc
+++ b/src/common/options.cc
@@ -786,6 +786,12 @@ std::vector<Option> get_global_options() {
     .set_flag(Option::FLAG_NO_MON_UPDATE)
     .set_description(""),
 
+    Option("thp", Option::TYPE_BOOL, Option::LEVEL_DEV)
+    .set_default(false)
+    .set_flag(Option::FLAG_STARTUP)
+    .set_description("enable transparent huge page (THP) support")
+    .set_long_description("Ceph is known to suffer from memory fragmentation due to THP use. This is indicated by RSS usage above configured memory targets. Enabling THP is currently discouraged until selective use of THP by Ceph is implemented."),
+
     Option("key", Option::TYPE_STR, Option::LEVEL_ADVANCED)
     .set_default("")
     .set_description("Authentication key")

diff --git a/src/global/global_init.cc b/src/global/global_init.cc
index 6962f42a543ed34d3554c4e41ccd0abec11f2eac..dff97e5c7ee04c4385d8ab3d1f4c51e8f6534eab 100644 (file)
--- a/src/global/global_init.cc
+++ b/src/global/global_init.cc
@@ -312,6 +312,11 @@ global_init(const std::map<std::string,std::string> *defaults,
   if (prctl(PR_SET_DUMPABLE, 1) == -1) {
     cerr << "warning: unable to set dumpable flag: " << cpp_strerror(errno) << std::endl;
   }
+#  if defined(PR_SET_THP_DISABLE)
+  if (!g_conf().get_val<bool>("thp") && prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0) == -1) {
+    cerr << "warning: unable to disable THP: " << cpp_strerror(errno) << std::endl;
+  }
+#  endif
 #endif
 
   //