msgr: fix throttle deadlock

Do msgr throttle after peer policy throttle.  The msgr (dispatch) throttle
is shortlived and won't deadlock (unless dispatch blocks), so it's safe to
take last.  In contrast, the policy throttle carries over the lifetime of
the message, and may block until replication completes or whatever else.

diff --git a/src/msg/SimpleMessenger.cc b/src/msg/SimpleMessenger.cc
index 25aa41bcc8bbb47d65a8e51f225c666eccd5adbe..2ba249526ead6348b84870c3e743856c9dd288b7 100644 (file)
--- a/src/msg/SimpleMessenger.cc
+++ b/src/msg/SimpleMessenger.cc
@@ -1751,9 +1751,14 @@ Message *SimpleMessenger::Pipe::read_message()
   uint64_t message_size = header.front_len  + header.middle_len
     + header.data_len;
   if (message_size) {
-    messenger->message_throttler.get(message_size);
     if (policy.throttler)
       policy.throttler->get(message_size);
+
+    // throttle total bytes waiting for dispatch.  do this _after_ the
+    // policy throttle, as this one does not deadlock (unless dispatch
+    // blocks indefinitely, which it shouldn't).  in contrast, the
+    // policy throttle carries for the lifetime of the message.
+    messenger->message_throttler.get(message_size);
   }
 
   // read front