objecter: use ordered map<> for tracking tids to preserve order on resend

We are using a hash_map<> to map tids to Op*'s.  In handle_osd_map(),
we will recalc_op_target() on each Op in a random (hash) order.  These
will get put in a temp map<tid,Op*> to ensure they are resent in the
correct order, but their order on the session->ops list will be random.

Then later, if we reset an OSD connection, we will resend everything for
that session in ops order, which is be incorrect.

Fix this by explicitly reordering the requests to resend in
kick_requests(), much like we do in handle_osd_map().  This lets us
continue to use a hash_map<>, which is faster for reasonable numbers of
requests.  A simpler but slower fix would be to just use map<> instead.

This is one of many bugs contributing to #2947.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 1113a6c56739a56871f01fa13da881dab36a32c4)

diff --git a/src/osdc/Objecter.cc b/src/osdc/Objecter.cc
index 2fb41a4b58491887f5280d8285a5f8291bdea154..666c7d98b8093b4d5bb7e350cc3cedf0f3c7ea79 100644 (file)
--- a/src/osdc/Objecter.cc
+++ b/src/osdc/Objecter.cc
@@ -758,21 +758,31 @@ void Objecter::kick_requests(OSDSession *session)
   ldout(cct, 10) << "kick_requests for osd." << session->osd << dendl;
 
   // resend ops
+  map<tid_t,Op*> resend;  // resend in tid order
   for (xlist<Op*>::iterator p = session->ops.begin(); !p.end();) {
     Op *op = *p;
     ++p;
     logger->inc(l_osdc_op_resend);
     if (op->should_resend) {
-      send_op(op);
+      resend[op->tid] = op;
     } else {
       cancel_op(op);
     }
   }
+  while (!resend.empty()) {
+    send_op(resend.begin()->second);
+    resend.erase(resend.begin());
+  }
 
   // resend lingers
+  map<uint64_t, LingerOp*> lresend;  // resend in order
   for (xlist<LingerOp*>::iterator j = session->linger_ops.begin(); !j.end(); ++j) {
     logger->inc(l_osdc_linger_resend);
-    send_linger(*j);
+    lresend[(*j)->linger_id] = *j;
+  }
+  while (!lresend.empty()) {
+    send_linger(lresend.begin()->second);
+    lresend.erase(lresend.begin());
   }
 }