mgr: fix dispatch throttle bottleneck causing OSD connection timeouts

The mgr dispatch throttle (ms_dispatch_throttle_bytes, default 100MB)
was far smaller than the OSD throttle (mgr_osd_bytes, 512MB),
making the dispatch queue the effective bottleneck on large clusters.

When the dispatch queue fills, OSD connections stall waiting for
throttle space. These connections time out and reconnect, causing
connection churn and heap growth from repeated connection setup/teardown.
The MGR also falls behind on PG stats processing, leading to unknown
and inactive PGs.

Add mgr_dispatch_throttle_bytes (default 1GB) and a
Messenger::set_dispatch_throttle_size() API to configure it at init time.
The new default is sized to accommodate the sum of all per-type policy
throttles, ensuring the dispatch queue is never the bottleneck.

Fixes: https://tracker.ceph.com/issues/66310
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>

diff --git a/doc/mgr/administrator.rst b/doc/mgr/administrator.rst
index 6d60e897b58dd347e474f617e3ab318459391bc3..5b24ba60c7089a34bdbd9f384d8359d1a8c46f73 100644 (file)
--- a/doc/mgr/administrator.rst
+++ b/doc/mgr/administrator.rst
@@ -161,9 +161,9 @@ The Manager automatically adjusts :confval:`mgr_stats_period` based on message q
 depth to prevent overload during high cluster activity. This feature is enabled by
 default and can be controlled with the following settings:
 
-- :confval:`mgr_stats_period_autotune` (boolean, default: true): Enable or disable
+- :confval:`mgr_stats_period_autotune` : Enable or disable
   automatic tuning of the stats period.
-- :confval:`mgr_stats_period_autotune_queue_threshold` (integer, default: 100):
+- :confval:`mgr_stats_period_autotune_queue_threshold` :
   The message queue depth threshold that triggers an increase in the stats period.
 
 When the queue depth exceeds this threshold, the stats period is increased to

diff --git a/src/common/options/mgr.yaml.in b/src/common/options/mgr.yaml.in
index b9fa2a9b0c3a77961b8191c07dc38f426eb965d7..4c3e65229df1cb3683ba3009ddc278cc30dd9d04 100644 (file)
--- a/src/common/options/mgr.yaml.in
+++ b/src/common/options/mgr.yaml.in
@@ -56,12 +56,28 @@ options:
   long_desc: When mgr_stats_period_autotune is enabled, the Manager will increase
     the stats reporting period if the incoming message queue exceeds this threshold.
     Higher values make the system less sensitive to temporary queue spikes but may
-    allow longer periods of Manager overload.
+    allow longer periods of Manager overload. Should be well below the smallest
+    of mgr_mon_messages, mgr_mds_messages, and mgr_osd_messages so the autotune
+    fires before any per-connection throttle saturates.
   default: 100
   services:
   - mgr
   see_also:
   - mgr_stats_period
+- name: mgr_dispatch_throttle_bytes
+  type: size
+  level: advanced
+  desc: Limit total size of messages waiting in the mgr dispatch queue
+  long_desc: Sets the maximum total bytes of messages queued in the mgr dispatch
+    throttle. Overrides ms_dispatch_throttle_bytes for the mgr process. Should be
+    set to at least the sum of all per-type byte throttles (mgr_osd_bytes +
+    mgr_mds_bytes + mgr_client_bytes) to avoid the dispatch queue becoming a
+    bottleneck before per-connection throttles engage.
+  default: 1_G
+  services:
+  - mgr
+  see_also:
+  - ms_dispatch_throttle_bytes
 - name: mgr_client_bytes
   type: size
   level: dev

diff --git a/src/mgr/DaemonServer.cc b/src/mgr/DaemonServer.cc
index a7ddac0868d18341d85c37b7e486fc1a17ed56f3..822b508cbd90f30b2cf72d6c710f673773559a92 100644 (file)
--- a/src/mgr/DaemonServer.cc
+++ b/src/mgr/DaemonServer.cc
@@ -176,6 +176,8 @@ int DaemonServer::init(uint64_t gid, entity_addrvec_t client_addrs)
                           entity_name_t::MGR(gid),
                           "mgr",
                           Messenger::get_random_nonce());
+  msgr->set_dispatch_throttle_size(
+      g_conf().get_val<Option::size_t>("mgr_dispatch_throttle_bytes"));
   msgr->set_default_policy(Messenger::Policy::stateless_server(0));
   // throttle policy
   msgr->set_policy(entity_name_t::TYPE_OSD,

diff --git a/src/msg/Messenger.h b/src/msg/Messenger.h
index 8dc49e2483cd4fda72568b1e92420fd6277dd9d3..88c84ea9069c28fabdb80b59857070e33b335b32 100644 (file)
--- a/src/msg/Messenger.h
+++ b/src/msg/Messenger.h
@@ -373,6 +373,13 @@ public:
    * you must not destroy them before you destroy the Messenger.
    */
   virtual void set_policy_throttlers(int type, Throttle *bytes, Throttle *msgs=NULL) = 0;
+  /**
+   * set the maximum size of the dispatch queue throttle
+   *
+   * This is an init-time function and must be called *before* calling
+   * start().
+   */
+  virtual void set_dispatch_throttle_size(uint64_t size) {}
   /**
    * set the default send priority
    *

diff --git a/src/msg/async/AsyncMessenger.h b/src/msg/async/AsyncMessenger.h
index 3edbe00dcad766ed158da1aad0e1d671f7b1bac5..53ba370c4ef68ea978d48f2748d5a42160b18c54 100644 (file)
--- a/src/msg/async/AsyncMessenger.h
+++ b/src/msg/async/AsyncMessenger.h
@@ -125,6 +125,10 @@ public:
   double get_dispatch_queue_max_age(utime_t now) const override {
     return dispatch_queue.get_max_age(now);
   }
+
+  void set_dispatch_throttle_size(uint64_t size) override {
+    dispatch_queue.dispatch_throttler.reset_max(size);
+  }
   /** @} Accessors */
 
   /**