osd: Override recovery/backfill/sleep options with mclock scheduler.

Make the osd_*_sleep options modifiable during runtime and add them
to the set of tracked conf keys. This is to ensure that the sleep
options can be disabled/overridden during OSD bring-up if mclock
scheduler is employed.

Introduce OSD::maybe_override_options_for_qos():
  This method does the following if the mclock scheduler is enabled:
  - overrides the "recovery_max_active" to a high limit of 1000,
  - overrides "osd_max_backfills" option to a high limit of 1000 and
    sets the corresponding Async local and remote reserver objects also
    to the same value (1000),
  - disables osd_*_sleep options so that appropriate QoS may be
    provided with the mclock scheduler.

The above method is called in the following scenarios:
- After all the op shards are brought up during OSD initialization.
- In OSD::handle_conf_change() to override any settings related to
  QoS that the user intended to change.

Modify the mclock config reference to accurately reflect what options
can be changed when using mclock's "custom" profile and clean up
some whitespaces.

Fixes: https://tracker.ceph.com/issues/50501
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 8876d27f389ce034732cc7b34869e970a3dc138f)

Conflicts:
    doc/rados/configuration/mclock-config-ref.rst
    src/common/options/global.yaml.in
- The new "confval" directive in master for documentation is not yet
  available in pacific. Therefore, this is removed from the
  mclock-config-ref.rst document.
- The switch to yaml for global config options is not backported yet.
  Add "runtime" flag for existing options in options.cc.

diff --git a/doc/rados/configuration/mclock-config-ref.rst b/doc/rados/configuration/mclock-config-ref.rst
index 085efb16cf5c7ee87bb505940791ea777f319438..0f773698f38242afa8c526ac0c9d82f83b4fa873 100644 (file)
--- a/doc/rados/configuration/mclock-config-ref.rst
+++ b/doc/rados/configuration/mclock-config-ref.rst
@@ -5,7 +5,7 @@
 .. index:: mclock; configuration
 
 Mclock profiles mask the low level details from users, making it
-easier for them to configure mclock.  
+easier for them to configure mclock.
 
 To use mclock, you must provide the following input parameters:
 
@@ -18,7 +18,7 @@ lower-level mclock and Ceph parameters. The parameters applied by the mclock
 profile make it possible to tune the QoS between client I/O, recovery/backfill
 operations, and other background operations (for example, scrub, snap trim, and
 PG deletion). These background activities are considered best-effort internal
-clients of Ceph. 
+clients of Ceph.
 
 
 .. index:: mclock; profile definition
@@ -39,7 +39,7 @@ some Ceph-configuration parameters are transparently applied.
 
 The low-level mclock resource control parameters are the *reservation*,
 *limit*, and *weight* that provide control of the resource shares, as
-described in the `OSD Config Reference`_.
+described in the :ref:`dmclock-qos` section.
 
 
 .. index:: mclock; profile types
@@ -64,14 +64,14 @@ mclock profiles can be broadly classified into two types,
     This profile allocates equal reservation to client ops and background
     recovery ops.
 
-- **Custom**: This profile gives users complete control over all mclock and
-  Ceph configuration parameters. Using this profile is not recommended without
+- **Custom**: This profile gives users complete control over all the mclock
+  configuration parameters. Using this profile is not recommended without
   a deep understanding of mclock and related Ceph-configuration options.
 
 .. note:: Across the built-in profiles, internal clients of mclock (for example
-          "scrub", "snap trim", and "pg deletion") are given slightly lower 
-          reservations, but higher weight and no limit. This ensures that 
-          these operations are able to complete quickly if there are no other 
+          "scrub", "snap trim", and "pg deletion") are given slightly lower
+          reservations, but higher weight and no limit. This ensures that
+          these operations are able to complete quickly if there are no other
           competing services.
 
 
@@ -111,8 +111,8 @@ there might be instances that necessitate giving higher allocations to either
 client ops or recovery ops. In order to deal with such a situation, you can
 enable one of the alternate built-in profiles mentioned above.
 
-If a built-in profile is active, the following Ceph config sleep options will
-be disabled,
+If any mClock profile (including "custom") is active, the following Ceph config
+sleep options will be disabled,
 
 - ``osd_recovery_sleep``
 - ``osd_recovery_sleep_hdd``
@@ -154,7 +154,7 @@ Any existing benchmarking tool can be used for this purpose. The following
 steps use the *Ceph Benchmarking Tool* (cbt_). Regardless of the tool
 used, the steps described below remain the same.
 
-As already described in the `OSD Config Reference`_ section, the number of
+As already described in the :ref:`dmclock-qos` section, the number of
 shards and the bluestore's throttle parameters have an impact on the mclock op
 queues. Therefore, it is critical to set these values carefully in order to
 maximize the impact of the mclock scheduler.
@@ -366,6 +366,3 @@ mClock Config Options
 :Type: Float
 :Default: ``0.011``
 
-
-
-.. _OSD Config Reference: ../osd-config-ref#dmclock-qos

diff --git a/src/common/options.cc b/src/common/options.cc
index 5e640acd67bf631ca49489f9e455d8f46af9ccd1..210cdf6eeb8643c23c65d7b4b1b5e6418bdb3384 100644 (file)
--- a/src/common/options.cc
+++ b/src/common/options.cc
@@ -3204,18 +3204,22 @@ std::vector<Option> get_global_options() {
 
     Option("osd_snap_trim_sleep", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(0)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next snap trim (overrides values below)"),
 
     Option("osd_snap_trim_sleep_hdd", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(5)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next snap trim for HDDs"),
 
     Option("osd_snap_trim_sleep_ssd", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(0)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next snap trim for SSDs"),
 
     Option("osd_snap_trim_sleep_hybrid", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(2)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next snap trim when data is on HDD and journal is on SSD"),
 
     Option("osd_scrub_invalid_stats", Option::TYPE_BOOL, Option::LEVEL_ADVANCED)
@@ -3437,6 +3441,7 @@ std::vector<Option> get_global_options() {
 
     Option("osd_scrub_sleep", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(0)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Duration to inject a delay during scrubbing"),
 
     Option("osd_scrub_extended_sleep", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
@@ -3745,18 +3750,22 @@ std::vector<Option> get_global_options() {
 
     Option("osd_delete_sleep", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(0)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next removal transaction (overrides values below)"),
 
     Option("osd_delete_sleep_hdd", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(5)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next removal transaction for HDDs"),
 
     Option("osd_delete_sleep_ssd", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(1)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next removal transaction for SSDs"),
 
     Option("osd_delete_sleep_hybrid", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
     .set_default(1)
+    .set_flag(Option::FLAG_RUNTIME)
     .set_description("Time in seconds to sleep before next removal transaction when data is on HDD and journal is on SSD"),
 
     Option("osd_failsafe_full_ratio", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)

diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc
index 121190f409549a87b89ee2b67c3ca0c405cae59c..057c405a1376ef784f0183106bf2ba1df06a5869 100644 (file)
--- a/src/osd/OSD.cc
+++ b/src/osd/OSD.cc
@@ -2323,6 +2323,9 @@ OSD::OSD(CephContext *cct_, ObjectStore *store_,
       this);
     shards.push_back(one_shard);
   }
+
+  // override some config options if mclock is enabled on all the shards
+  maybe_override_options_for_qos();
 }
 
 OSD::~OSD()
@@ -9920,6 +9923,15 @@ const char** OSD::get_tracked_conf_keys() const
     "osd_recovery_sleep_hdd",
     "osd_recovery_sleep_ssd",
     "osd_recovery_sleep_hybrid",
+    "osd_delete_sleep",
+    "osd_delete_sleep_hdd",
+    "osd_delete_sleep_ssd",
+    "osd_delete_sleep_hybrid",
+    "osd_snap_trim_sleep",
+    "osd_snap_trim_sleep_hdd",
+    "osd_snap_trim_sleep_ssd",
+    "osd_snap_trim_sleep_hybrid"
+    "osd_scrub_sleep",
     "osd_recovery_max_active",
     "osd_recovery_max_active_hdd",
     "osd_recovery_max_active_ssd",
@@ -9969,46 +9981,9 @@ void OSD::handle_conf_change(const ConfigProxy& conf,
       changed.count("osd_recovery_max_active") ||
       changed.count("osd_recovery_max_active_hdd") ||
       changed.count("osd_recovery_max_active_ssd")) {
-    if (cct->_conf.get_val<std::string>("osd_op_queue") == "mclock_scheduler" &&
-        cct->_conf.get_val<std::string>("osd_mclock_profile") != "custom") {
-      // Set ceph config option to meet QoS goals
-      // Set high value for recovery max active
-      uint32_t recovery_max_active = 1000;
-      if (cct->_conf->osd_recovery_max_active) {
-        cct->_conf.set_val(
-            "osd_recovery_max_active", std::to_string(recovery_max_active));
-      }
-      if (store_is_rotational) {
-        cct->_conf.set_val(
-            "osd_recovery_max_active_hdd", std::to_string(recovery_max_active));
-      } else {
-        cct->_conf.set_val(
-            "osd_recovery_max_active_ssd", std::to_string(recovery_max_active));
-      }
-      // Set high value for osd_max_backfill
-      cct->_conf.set_val("osd_max_backfills", std::to_string(1000));
-
-      // Disable recovery sleep
-      cct->_conf.set_val("osd_recovery_sleep", std::to_string(0));
-      cct->_conf.set_val("osd_recovery_sleep_hdd", std::to_string(0));
-      cct->_conf.set_val("osd_recovery_sleep_ssd", std::to_string(0));
-      cct->_conf.set_val("osd_recovery_sleep_hybrid", std::to_string(0));
-
-      // Disable delete sleep
-      cct->_conf.set_val("osd_delete_sleep", std::to_string(0));
-      cct->_conf.set_val("osd_delete_sleep_hdd", std::to_string(0));
-      cct->_conf.set_val("osd_delete_sleep_ssd", std::to_string(0));
-      cct->_conf.set_val("osd_delete_sleep_hybrid", std::to_string(0));
-
-      // Disable snap trim sleep
-      cct->_conf.set_val("osd_snap_trim_sleep", std::to_string(0));
-      cct->_conf.set_val("osd_snap_trim_sleep_hdd", std::to_string(0));
-      cct->_conf.set_val("osd_snap_trim_sleep_ssd", std::to_string(0));
-      cct->_conf.set_val("osd_snap_trim_sleep_hybrid", std::to_string(0));
-
-      // Disable scrub sleep
-      cct->_conf.set_val("osd_scrub_sleep", std::to_string(0));
-    } else {
+    if (!maybe_override_options_for_qos() &&
+        changed.count("osd_max_backfills")) {
+      // Scheduler is not "mclock". Fallback to earlier behavior
       service.local_reserver.set_max(cct->_conf->osd_max_backfills);
       service.remote_reserver.set_max(cct->_conf->osd_max_backfills);
     }
@@ -10102,6 +10077,54 @@ void OSD::handle_conf_change(const ConfigProxy& conf,
   }
 }
 
+bool OSD::maybe_override_options_for_qos()
+{
+  // If the scheduler enabled is mclock, override the recovery, backfill
+  // and sleep options so that mclock can meet the QoS goals.
+  if (cct->_conf.get_val<std::string>("osd_op_queue") == "mclock_scheduler") {
+    dout(1) << __func__
+            << ": Changing recovery/backfill/sleep settings for QoS" << dendl;
+
+    // Set high value for recovery max active
+    uint32_t rec_max_active = 1000;
+    cct->_conf.set_val(
+      "osd_recovery_max_active", std::to_string(rec_max_active));
+    cct->_conf.set_val(
+      "osd_recovery_max_active_hdd", std::to_string(rec_max_active));
+    cct->_conf.set_val(
+      "osd_recovery_max_active_ssd", std::to_string(rec_max_active));
+
+    // Set high value for osd_max_backfill
+    uint32_t max_backfills = 1000;
+    cct->_conf.set_val("osd_max_backfills", std::to_string(max_backfills));
+    service.local_reserver.set_max(max_backfills);
+    service.remote_reserver.set_max(max_backfills);
+
+    // Disable recovery sleep
+    cct->_conf.set_val("osd_recovery_sleep", std::to_string(0));
+    cct->_conf.set_val("osd_recovery_sleep_hdd", std::to_string(0));
+    cct->_conf.set_val("osd_recovery_sleep_ssd", std::to_string(0));
+    cct->_conf.set_val("osd_recovery_sleep_hybrid", std::to_string(0));
+
+    // Disable delete sleep
+    cct->_conf.set_val("osd_delete_sleep", std::to_string(0));
+    cct->_conf.set_val("osd_delete_sleep_hdd", std::to_string(0));
+    cct->_conf.set_val("osd_delete_sleep_ssd", std::to_string(0));
+    cct->_conf.set_val("osd_delete_sleep_hybrid", std::to_string(0));
+
+    // Disable snap trim sleep
+    cct->_conf.set_val("osd_snap_trim_sleep", std::to_string(0));
+    cct->_conf.set_val("osd_snap_trim_sleep_hdd", std::to_string(0));
+    cct->_conf.set_val("osd_snap_trim_sleep_ssd", std::to_string(0));
+    cct->_conf.set_val("osd_snap_trim_sleep_hybrid", std::to_string(0));
+
+    // Disable scrub sleep
+    cct->_conf.set_val("osd_scrub_sleep", std::to_string(0));
+    return true;
+  }
+  return false;
+}
+
 void OSD::update_log_config()
 {
   map<string,string> log_to_monitors;

diff --git a/src/osd/OSD.h b/src/osd/OSD.h
index 90d2f800fb841a96236ca4c06d0e46493ddd70a8..be76e08058768e2c6c39a6fc9b427339c0cffe75 100644 (file)
--- a/src/osd/OSD.h
+++ b/src/osd/OSD.h
@@ -2065,6 +2065,7 @@ private:
   float get_osd_snap_trim_sleep();
 
   int get_recovery_max_active();
+  bool maybe_override_options_for_qos();
 
   void scrub_purged_snaps();
   void probe_smart(const std::string& devid, std::ostream& ss);