osd: shut down the MgrClient before osd_fast_shutdown

We should shut down the MgrClient before osd_fast_shutdown.

This approach avoids a race condition that sometimes occurs
when the MgrClient attempts to gather `pg_stats` from the database
after it has been unmounted, or when it is in the process of being
unmounted.

Fixes: https://tracker.ceph.com/issues/56101
Signed-off-by: Laura Flores <lflores@redhat.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 77ebbf580bee3bf83de0da03487b6ebb08c17d76)

diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc
index 839c4fd364dcf4d719dd7ff694b9f41ce3660641..df9e61db8bd805394ca8f43feec09e63c6009884 100644 (file)
--- a/src/osd/OSD.cc
+++ b/src/osd/OSD.cc
@@ -4311,6 +4311,12 @@ int OSD::shutdown()
     cct->_conf.apply_changes(nullptr);
   }
 
+  // stop MgrClient earlier as it's more like an internal consumer of OSD
+  // 
+  // should occur before unmounting the database in fast-shutdown to avoid
+  // a race condition (see https://tracker.ceph.com/issues/56101)
+  mgrc.shutdown();
+
   if (cct->_conf->osd_fast_shutdown) {
     // first, stop new task from being taken from op_shardedwq
     // and clear all pending tasks
@@ -4350,9 +4356,6 @@ int OSD::shutdown()
     _exit(0);
   }
 
-  // stop MgrClient earlier as it's more like an internal consumer of OSD
-  mgrc.shutdown();
-
   service.start_shutdown();
 
   // stop sending work to pgs.  this just prevents any new work in _process