From: Nitzan Mordechai <nmordech@ibm.com>
Date: Mon, 8 Jun 2026 05:18:55 +0000 (+0000)
Subject: mgr: fix dispatch throttle bottleneck causing OSD connection timeouts
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=948de224df12b8b383460b86ba520b191dc275bc;p=ceph.git

mgr: fix dispatch throttle bottleneck causing OSD connection timeouts

The mgr dispatch throttle (ms_dispatch_throttle_bytes, default 100MB)
was far smaller than the OSD throttle (mgr_osd_bytes, 512MB),
making the dispatch queue the effective bottleneck on large clusters.

When the dispatch queue fills, OSD connections stall waiting for
throttle space. These connections time out and reconnect, causing
connection churn and heap growth from repeated connection setup/teardown.
The MGR also falls behind on PG stats processing, leading to unknown
and inactive PGs.

Add mgr_dispatch_throttle_bytes (default 1GB) and a
Messenger::set_dispatch_throttle_size() API to configure it at init time.
The new default is sized to accommodate the sum of all per-type policy
throttles, ensuring the dispatch queue is never the bottleneck.

Fixes: https://tracker.ceph.com/issues/66310
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
---

diff --git a/doc/mgr/administrator.rst b/doc/mgr/administrator.rst
index 6d60e897b58..5b24ba60c70 100644
--- a/doc/mgr/administrator.rst
+++ b/doc/mgr/administrator.rst
@@ -161,9 +161,9 @@ The Manager automatically adjusts :confval:`mgr_stats_period` based on message q
 depth to prevent overload during high cluster activity. This feature is enabled by
 default and can be controlled with the following settings:
 
-- :confval:`mgr_stats_period_autotune` (boolean, default: true): Enable or disable
+- :confval:`mgr_stats_period_autotune` : Enable or disable
   automatic tuning of the stats period.
-- :confval:`mgr_stats_period_autotune_queue_threshold` (integer, default: 100):
+- :confval:`mgr_stats_period_autotune_queue_threshold` :
   The message queue depth threshold that triggers an increase in the stats period.
 
 When the queue depth exceeds this threshold, the stats period is increased to
diff --git a/src/common/options/mgr.yaml.in b/src/common/options/mgr.yaml.in
index b9fa2a9b0c3..4c3e65229df 100644
--- a/src/common/options/mgr.yaml.in
+++ b/src/common/options/mgr.yaml.in
@@ -56,12 +56,28 @@ options:
   long_desc: When mgr_stats_period_autotune is enabled, the Manager will increase
     the stats reporting period if the incoming message queue exceeds this threshold.
     Higher values make the system less sensitive to temporary queue spikes but may
-    allow longer periods of Manager overload.
+    allow longer periods of Manager overload. Should be well below the smallest
+    of mgr_mon_messages, mgr_mds_messages, and mgr_osd_messages so the autotune
+    fires before any per-connection throttle saturates.
   default: 100
   services:
   - mgr
   see_also:
   - mgr_stats_period
+- name: mgr_dispatch_throttle_bytes
+  type: size
+  level: advanced
+  desc: Limit total size of messages waiting in the mgr dispatch queue
+  long_desc: Sets the maximum total bytes of messages queued in the mgr dispatch
+    throttle. Overrides ms_dispatch_throttle_bytes for the mgr process. Should be
+    set to at least the sum of all per-type byte throttles (mgr_osd_bytes +
+    mgr_mds_bytes + mgr_client_bytes) to avoid the dispatch queue becoming a
+    bottleneck before per-connection throttles engage.
+  default: 1_G
+  services:
+  - mgr
+  see_also:
+  - ms_dispatch_throttle_bytes
 - name: mgr_client_bytes
   type: size
   level: dev
diff --git a/src/mgr/DaemonServer.cc b/src/mgr/DaemonServer.cc
index a7ddac0868d..822b508cbd9 100644
--- a/src/mgr/DaemonServer.cc
+++ b/src/mgr/DaemonServer.cc
@@ -176,6 +176,8 @@ int DaemonServer::init(uint64_t gid, entity_addrvec_t client_addrs)
 			   entity_name_t::MGR(gid),
 			   "mgr",
 			   Messenger::get_random_nonce());
+  msgr->set_dispatch_throttle_size(
+      g_conf().get_val<Option::size_t>("mgr_dispatch_throttle_bytes"));
   msgr->set_default_policy(Messenger::Policy::stateless_server(0));
   // throttle policy
   msgr->set_policy(entity_name_t::TYPE_OSD,
diff --git a/src/msg/Messenger.h b/src/msg/Messenger.h
index 8dc49e2483c..88c84ea9069 100644
--- a/src/msg/Messenger.h
+++ b/src/msg/Messenger.h
@@ -373,6 +373,13 @@ public:
    * you must not destroy them before you destroy the Messenger.
    */
   virtual void set_policy_throttlers(int type, Throttle *bytes, Throttle *msgs=NULL) = 0;
+  /**
+   * set the maximum size of the dispatch queue throttle
+   *
+   * This is an init-time function and must be called *before* calling
+   * start().
+   */
+  virtual void set_dispatch_throttle_size(uint64_t size) {}
   /**
    * set the default send priority
    *
diff --git a/src/msg/async/AsyncMessenger.h b/src/msg/async/AsyncMessenger.h
index 3edbe00dcad..53ba370c4ef 100644
--- a/src/msg/async/AsyncMessenger.h
+++ b/src/msg/async/AsyncMessenger.h
@@ -125,6 +125,10 @@ public:
   double get_dispatch_queue_max_age(utime_t now) const override {
     return dispatch_queue.get_max_age(now);
   }
+
+  void set_dispatch_throttle_size(uint64_t size) override {
+    dispatch_queue.dispatch_throttler.reset_max(size);
+  }
   /** @} Accessors */
 
   /**