osd: shut down the MgrClient before osd_fast_shutdown

We should shut down the MgrClient before osd_fast_shutdown.

This approach avoids a race condition that sometimes occurs
when the MgrClient attempts to gather `pg_stats` from the database
after it has been unmounted, or when it is in the process of being
unmounted.

Fixes: https://tracker.ceph.com/issues/56101
Signed-off-by: Laura Flores <lflores@redhat.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc
index dc3f7f5da8bd2dc8e8e397a72ac0a811e632410e..a16ec82d7282c5e3c8251a94e540e916f9d05337 100644 (file)
--- a/src/osd/OSD.cc
+++ b/src/osd/OSD.cc
@@ -4353,6 +4353,12 @@ int OSD::shutdown()
     cct->_conf.apply_changes(nullptr);
   }
 
+  // stop MgrClient earlier as it's more like an internal consumer of OSD
+  // 
+  // should occur before unmounting the database in fast-shutdown to avoid
+  // a race condition (see https://tracker.ceph.com/issues/56101)
+  mgrc.shutdown();
+
   if (cct->_conf->osd_fast_shutdown) {
     // first, stop new task from being taken from op_shardedwq
     // and clear all pending tasks
@@ -4392,9 +4398,6 @@ int OSD::shutdown()
     _exit(0);
   }
 
-  // stop MgrClient earlier as it's more like an internal consumer of OSD
-  mgrc.shutdown();
-
   service.start_shutdown();
 
   // stop sending work to pgs.  this just prevents any new work in _process