osd/ReplicatedPG: fix enqueue_front race

When requeuing and item at the front, we need to shuffle the items in
pg_for_processing if there is an entry for this PG there.  If so, we need
to hold the qlock for the duration of the requeue of the shuffled item
back into the primary queue in order to avoid reshuffling items.  For
example, consider the queue has

 A B C D

 - dequeue1 gets (pg, A), puts A in the processing list
 - dequeue1 tries to lock pg, blocks
 - enqueue_front on X takes qlock, swaps it for A, drops qlock
 - dequeue2 gets (pg, B), puts B in the processing list
 - enqueue_front pushes X back into the original list

 so we have processing: X B  queue: A C D

 - dequeue* get X, then B, then A C D

If we whole qlock for the duration of the enqueue_front, we avoid dequeu2
from sneaking in an shuffling B into the processing list before we have
crammed A back onto the front of the list.

This may have caused #7712.

Backport: emperor, dumpling
Signed-off-by: Sage Weil <sage@inktank.com>

diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc
index 8aee20185f99a996edd674a7074cb4de2351c2b0..54e0febeeb1d82cc2a52012f4e9ea97c305d7374 100644 (file)
--- a/src/osd/OSD.cc
+++ b/src/osd/OSD.cc
@@ -7530,13 +7530,11 @@ void OSD::OpWQ::_enqueue(pair<PGRef, OpRequestRef> item)
 
 void OSD::OpWQ::_enqueue_front(pair<PGRef, OpRequestRef> item)
 {
-  {
-    Mutex::Locker l(qlock);
-    if (pg_for_processing.count(&*(item.first))) {
-      pg_for_processing[&*(item.first)].push_front(item.second);
-      item.second = pg_for_processing[&*(item.first)].back();
-      pg_for_processing[&*(item.first)].pop_back();
-    }
+  Mutex::Locker l(qlock);
+  if (pg_for_processing.count(&*(item.first))) {
+    pg_for_processing[&*(item.first)].push_front(item.second);
+    item.second = pg_for_processing[&*(item.first)].back();
+    pg_for_processing[&*(item.first)].pop_back();
   }
   unsigned priority = item.second->get_req()->get_priority();
   unsigned cost = item.second->get_req()->get_cost();