From: Sage Weil <sage@redhat.com>
Date: Fri, 11 Dec 2015 16:50:22 +0000 (-0500)
Subject: osd/OSDMap: clear osd_info, osd_xinfo on osd deletion
X-Git-Tag: v10.0.3~205^2
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=4e28f9e63644835c72ff19d445d40520ff54f8e5;p=ceph.git

osd/OSDMap: clear osd_info, osd_xinfo on osd deletion

If we destroy an OSD in the map, clear not just the uuid but also
all the metadata about it.

Specifically, we care about up_from, which can prevent a new
OSD from booting if it starts with a map prior to the deletion
when it sends it boot.  Specifically, the osd epoch may be 0 and
if the latest osd epoch is also small the osd decide it is "close
enough" to the latest epoch and sends the boot message.  In
practice this problem wouldn't surface on any cluster that isn't
brand new.

Note that this changes the result of applying an incremental.
As such, it will cause lots of old OSDs to request full maps
from the mon, spiking load during an upgrade.  This is as it
should be.

Fixes: #13988
Signed-off-by: Sage Weil <sage@redhat.com>
---

diff --git a/src/osd/OSDMap.cc b/src/osd/OSDMap.cc
index ebe73d694e54..fb75a54634ea 100644
--- a/src/osd/OSDMap.cc
+++ b/src/osd/OSDMap.cc
@@ -1347,9 +1347,21 @@ int OSDMap::apply_incremental(const Incremental &inc)
       osd_xinfo[i->first].down_stamp = modified;
     }
     if ((osd_state[i->first] & CEPH_OSD_EXISTS) &&
-	(s & CEPH_OSD_EXISTS))
+	(s & CEPH_OSD_EXISTS)) {
+      // osd is destroyed; clear out anything interesting.
       (*osd_uuid)[i->first] = uuid_d();
-    osd_state[i->first] ^= s;
+      osd_info[i->first] = osd_info_t();
+      osd_xinfo[i->first] = osd_xinfo_t();
+      osd_weight[i->first] = CEPH_OSD_IN;
+      set_primary_affinity(i->first, CEPH_OSD_DEFAULT_PRIMARY_AFFINITY);
+      osd_addrs->client_addr[i->first].reset(new entity_addr_t());
+      osd_addrs->cluster_addr[i->first].reset(new entity_addr_t());
+      osd_addrs->hb_front_addr[i->first].reset(new entity_addr_t());
+      osd_addrs->hb_back_addr[i->first].reset(new entity_addr_t());
+      osd_state[i->first] = 0;
+    } else {
+      osd_state[i->first] ^= s;
+    }
   }
   for (map<int32_t,entity_addr_t>::const_iterator i = inc.new_up_client.begin();
        i != inc.new_up_client.end();