osd: peering: detect when log source osd goes down

The Peering state has a generic check based on the prior set osds that
will restart peering if one of them goes down (or one of the interesting
down ones comes up).  The GetLog state, however, can pull the log from
a peer that is not in the prior set if it got a notify from them (e.g., an
osd in an old interval that was down when the prior set was calculated).
If that osd goes down, we don't detect it and will block forward.

Fix by adding a simple check in GetLog for the newest_update_osd going
down.

(BTW GetMissing does not suffer from this problem because
peer_missing_requested is a subset of the prior set, so the Peering check
is sufficient.)

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

diff --git a/src/osd/PG.cc b/src/osd/PG.cc
index 70e83c92b97a0412a5f821dcbb9b2ff10529a906..5b49ff7dfd912353e289d5a52b3f67363d197e48 100644 (file)
--- a/src/osd/PG.cc
+++ b/src/osd/PG.cc
@@ -5216,6 +5216,21 @@ PG::RecoveryState::GetLog::GetLog(my_context ctx) :
               pg->get_osdmap()->get_epoch()));
 }
 
+boost::statechart::result PG::RecoveryState::GetLog::react(const AdvMap& advmap)
+{
+  // make sure our log source didn't go down.  we need to check
+  // explicitly because it may not be part of the prior set, which
+  // means the Peering state check won't catch it going down.
+  if (!advmap.osdmap->is_up(newest_update_osd)) {
+    dout(10) << "GetLog: newest_update_osd osd." << newest_update_osd << " went down" << dendl;
+    post_event(advmap);
+    return transit< Reset >();
+  }
+
+  // let the Peering state do its checks.
+  return forward_event();
+}
+
 boost::statechart::result PG::RecoveryState::GetLog::react(const MLogRec& logevt)
 {
   assert(!msg);

diff --git a/src/osd/PG.h b/src/osd/PG.h
index b70ae0e3f44bb9bb46b13f791c687bbc29bb9b06..ceb85b7ac6c96577e3c77e03cbd8bc2d4b394f43 100644 (file)
--- a/src/osd/PG.h
+++ b/src/osd/PG.h
@@ -1306,8 +1306,10 @@ public:
        boost::statechart::custom_reaction< QueryState >,
        boost::statechart::custom_reaction< MLogRec >,
        boost::statechart::custom_reaction< GotLog >,
+       boost::statechart::custom_reaction< AdvMap >,
        boost::statechart::transition< IsIncomplete, Incomplete >
        > reactions;
+      boost::statechart::result react(const AdvMap&);
       boost::statechart::result react(const QueryState& q);
       boost::statechart::result react(const MLogRec& logevt);
       boost::statechart::result react(const GotLog&);