]> git-server-git.apps.pok.os.sepia.ceph.com Git - ceph.git/commit
mgr: fix dispatch throttle bottleneck causing OSD connection timeouts 69941/head
authorNitzan Mordechai <nmordech@ibm.com>
Mon, 8 Jun 2026 05:18:55 +0000 (05:18 +0000)
committerNitzan Mordechai <nmordech@redhat.com>
Sun, 5 Jul 2026 12:16:02 +0000 (15:16 +0300)
commitdbfce796f07df79c700902908271d07f70949cd3
tree0ec7e18ca6bc6983cf24669749c104327b634ab9
parent7c8dae21fa0400abee8da5011a76f3b747dec84e
mgr: fix dispatch throttle bottleneck causing OSD connection timeouts

The mgr dispatch throttle (ms_dispatch_throttle_bytes, default 100MB)
was far smaller than the OSD throttle (mgr_osd_bytes, 512MB),
making the dispatch queue the effective bottleneck on large clusters.

When the dispatch queue fills, OSD connections stall waiting for
throttle space. These connections time out and reconnect, causing
connection churn and heap growth from repeated connection setup/teardown.
The MGR also falls behind on PG stats processing, leading to unknown
and inactive PGs.

Add mgr_dispatch_throttle_bytes (default 1GB) and a
Messenger::set_dispatch_throttle_size() API to configure it at init time.
The new default is sized to accommodate the sum of all per-type policy
throttles, ensuring the dispatch queue is never the bottleneck.

Fixes: https://tracker.ceph.com/issues/66310
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
(cherry picked from commit 948de224df12b8b383460b86ba520b191dc275bc)
doc/mgr/administrator.rst
src/common/options/mgr.yaml.in
src/mgr/DaemonServer.cc
src/msg/Messenger.h
src/msg/async/AsyncMessenger.h