osd: peering: detect when log source osd goes down

The Peering state has a generic check based on the prior set osds that
will restart peering if one of them goes down (or one of the interesting
down ones comes up).  The GetLog state, however, can pull the log from
a peer that is not in the prior set if it got a notify from them (e.g., an
osd in an old interval that was down when the prior set was calculated).
If that osd goes down, we don't detect it and will block forward.

Fix by adding a simple check in GetLog for the newest_update_osd going
down.

(BTW GetMissing does not suffer from this problem because
peer_missing_requested is a subset of the prior set, so the Peering check
is sufficient.)

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

diff --git a/src/osd/PG.cc b/src/osd/PG.cc
index c68e49c8c9c59234ed399062991a153fa9de17a0..a80d95dcbaea0b02c4b8e0d6a36cc57037815b7d 100644 (file)
--- a/src/osd/PG.cc
+++ b/src/osd/PG.cc
@@ -4758,6 +4758,21 @@ PG::RecoveryState::GetLog::GetLog(my_context ctx) :
                                        pg_query_t(pg_query_t::LOG, request_log_from, pg->info.history));
 }
 
+boost::statechart::result PG::RecoveryState::GetLog::react(const AdvMap& advmap)
+{
+  // make sure our log source didn't go down.  we need to check
+  // explicitly because it may not be part of the prior set, which
+  // means the Peering state check won't catch it going down.
+  if (!advmap.osdmap->is_up(newest_update_osd)) {
+    dout(10) << "GetLog: newest_update_osd osd." << newest_update_osd << " went down" << dendl;
+    post_event(advmap);
+    return transit< Reset >();
+  }
+
+  // let the Peering state do its checks.
+  return forward_event();
+}
+
 boost::statechart::result PG::RecoveryState::GetLog::react(const MLogRec& logevt)
 {
   assert(!msg);

diff --git a/src/osd/PG.h b/src/osd/PG.h
index 4b843aa14bd3b7b40bad9e59709a31820e58d023..4a292f6b3d09c2acc63d253d3273b8edc10d6994 100644 (file)
--- a/src/osd/PG.h
+++ b/src/osd/PG.h
@@ -1154,8 +1154,10 @@ public:
        boost::statechart::custom_reaction< QueryState >,
        boost::statechart::custom_reaction< MLogRec >,
        boost::statechart::custom_reaction< GotLog >,
+       boost::statechart::custom_reaction< AdvMap >,
        boost::statechart::transition< IsIncomplete, Incomplete >
        > reactions;
+      boost::statechart::result react(const AdvMap&);
       boost::statechart::result react(const QueryState& q);
       boost::statechart::result react(const MLogRec& logevt);
       boost::statechart::result react(const GotLog&);