osd/PeeringState: separate history's pruub from pg's

(pruub = prior_readable_until_ub [upper bound])

During peering, a primary may conclude that it does not need to wait for
the prior interval(s)' read lease because it will query all such osds.
However, it is dangerous to reflect that local inference about future
peering effects in the info.history, which is freely shared with other
OSDs.  For example, if the primary cleared the history pruub, shared it,
and then failed, the next primary may conclude that it does not need to
wait for the lease to expire.

Instead, track the pruub in the conventional way.  Only at the end of
peering do we clear it (if there are no prior_interval_down_osds) before
ending our final activate infos out, when we have already talked to the
peer osds and they know the prior interval has finished.

Fixes: https://tracker.ceph.com/issues/53326
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 96d6bf22a5dbf14da1de0ee6128585b9dd9d60d8)

diff --git a/src/osd/PeeringState.cc b/src/osd/PeeringState.cc
index ac951a4a9f391f672dad6964c7bbaf9e8377f685..6fc97f23f7830112accf10066c8edb1eac827bf6 100644 (file)
--- a/src/osd/PeeringState.cc
+++ b/src/osd/PeeringState.cc
@@ -2782,6 +2782,11 @@ void PeeringState::activate(
     info.purged_snaps.swap(purged);
 
     // start up replicas
+    if (prior_readable_down_osds.empty()) {
+      dout(10) << __func__ << " no prior_readable_down_osds to wait on, clearing ub"
+              << dendl;
+      clear_prior_readable_until_ub();
+    }
     info.history.refresh_prior_readable_until_ub(pl->get_mnow(),
                                                 prior_readable_until_ub);
 
@@ -6768,10 +6773,10 @@ PeeringState::GetInfo::GetInfo(my_context ctx)
 
   prior_set = ps->build_prior();
   ps->prior_readable_down_osds = prior_set.down;
+
   if (ps->prior_readable_down_osds.empty()) {
-    psdout(10) << " no prior_set down osds, clearing prior_readable_until_ub"
+    psdout(10) << " no prior_set down osds, will clear prior_readable_until_ub before activating"
               << dendl;
-    ps->clear_prior_readable_until_ub();
   }
 
   ps->reset_min_peer_features();

diff --git a/src/osd/PeeringState.h b/src/osd/PeeringState.h
index 2a000fb965a8659bf968f24ccc406dbf1ee948be..2cc340cb9df9a8ee60940432e58eab44fe519438 100644 (file)
--- a/src/osd/PeeringState.h
+++ b/src/osd/PeeringState.h
@@ -2075,6 +2075,7 @@ public:
   void clear_prior_readable_until_ub() {
     prior_readable_until_ub = ceph::signedspan::zero();
     prior_readable_down_osds.clear();
+    info.history.prior_readable_until_ub = ceph::signedspan::zero();
   }
 
   void renew_lease(ceph::signedspan now) {