crimson/osd: fix boot crash when a peer OSD was purged across an osdmap batch

OSD::committed_osd_maps() captures old_map, then for each peer in
old_map->get_all_osds() that transitioned up->down it marks the peer's
cluster address down. The address was fetched from the *new* osdmap via
osdmap->get_cluster_addrs(osd_id), but osd_id comes from old_map. If that
peer was purged (fully removed) in the new map, get_cluster_addrs hits
ceph_assert(exists()) and the OSD aborts on boot.

This makes it impossible to decommission a crimson cluster: purging any
OSD leaves surviving crimson OSDs unable to reboot once they replay an
osdmap epoch range spanning that purge, since they abort on this assert
during boot.

Look the address up in old_map instead: it is guaranteed to contain
osd_id (that is where the id came from) and holds its last-known cluster
address, which is exactly what we want to mark down.

Signed-off-by: Vitaly Goot <vgoot@akamai.com>

diff --git a/src/crimson/osd/osd.cc b/src/crimson/osd/osd.cc
index 25096a1fb8a8c52cbdf25730bc124181c939e6d1..556850d18f51f9cd14e87cccc81dbd1574116be2 100644 (file)
--- a/src/crimson/osd/osd.cc
+++ b/src/crimson/osd/osd.cc
@@ -1243,8 +1243,11 @@ seastar::future<> OSD::committed_osd_maps(
         co_return;
       }
       DEBUG("osd.{}: mark osd.{} down", whoami, osd_id);
+      // osd_id came from old_map->get_all_osds(); if it was purged in the new
+      // osdmap, osdmap->get_cluster_addrs(osd_id) would ceph_assert(exists()).
+      // old_map is guaranteed to contain osd_id and holds its last-known addr.
       co_await cluster_msgr->mark_down(
-        osdmap->get_cluster_addrs(osd_id).front());
+        old_map->get_cluster_addrs(osd_id).front());
     });
 
     co_await pg_shard_manager.update_map(std::move(o));