osd: Fix issue where it is possible for stats to be recovered incorrectly during...

To hit the problem, after you take a snapshot, you:
* Perform a write
* Perform a partial write that only involves the primary
* Perform a partial write that only involves a non-primary
* Primary goes down
* Primary comes up
* Primary goes through peering and chooses a non-primary shard as its peering partner

The result of these operations is the stats reporting a size difference equal to the partial write that only involves the primary, as the non-primary is not aware of the clone operation by design and so that is missing update is copied to the osd. This commit prevents it by invalidating the stats in the case where this happens. There will be a future commit to further narrow the set of cases where stats invalidations can happen.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>
(cherry picked from commit ebc6f270681a5f602b4979ef8a96bee1ca0d1e73)

diff --git a/src/osd/PeeringState.cc b/src/osd/PeeringState.cc
index 667ac2accd96c2e5d43785f4563ec82821851d49..ce0e190e7969aec9c8510ea80356ee037b9f0599 100644 (file)
--- a/src/osd/PeeringState.cc
+++ b/src/osd/PeeringState.cc
@@ -3482,6 +3482,10 @@ void PeeringState::proc_master_log(
   // log to be authoritative (i.e., their entries are by definitely
   // non-divergent).
   merge_log(t, oinfo, std::move(olog), from);
+  if (info.last_backfill.is_max() &&
+      pool.info.is_nonprimary_shard(from.shard)) {
+    invalidate_stats = true;
+  }
   info.stats.stats_invalid |= invalidate_stats;
   peer_info[from] = oinfo;
   psdout(10) << " peer osd." << from << " now " << oinfo