mgr: avoid false alarm of MGR_MODULE_ERROR

mgr sends healthy report periodically, the report includes the
information whether the always-on modules are loaded or not. but the
modules are loaded with two steps:

1. load the options and command exposed by modules. the options and
   commands are registered using static methods of the subclasss of
   MgrModule.
2. create an instance of the subclass of MgrModule. this is performed
   in background by a Finisher thread. upon finishing of the construction
   of the instance, ActivePyModules::start_one() adds the module which
   successfully creates the class to `modules`.

but there is chance that when mgr sends healthy report, the always-on
module is still creating its instance of MgrModule subclass, or that
task is still pending in the finisher thread. in that case, mgr would
add a false error message like
```
4 mgr modules have failed (MGR_MODULE_ERROR)
```
in the healthy report

in this change, the number of modules in pending state is tracked,
and mgr will not take the missing always-on modules into account unless
the number of pending modules is 0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

diff --git a/src/mgr/ActivePyModules.cc b/src/mgr/ActivePyModules.cc
index 27d11f27af98fd08eacb8f4c514114f90f635a53..210f7755090857e448761c9d11170dd7dad2452a 100644 (file)
--- a/src/mgr/ActivePyModules.cc
+++ b/src/mgr/ActivePyModules.cc
@@ -448,15 +448,17 @@ void ActivePyModules::start_one(PyModuleRef py_module)
   const auto name = py_module->get_name();
   auto active_module = std::make_shared<ActivePyModule>(py_module, clog);
 
+  pending_modules.insert(name);
   // Send all python calls down a Finisher to avoid blocking
   // C++ code, and avoid any potential lock cycles.
   finisher.queue(new LambdaContext([this, active_module, name](int) {
     int r = active_module->load(this);
+    std::lock_guard l(lock);
+    pending_modules.erase(name);
     if (r != 0) {
       derr << "Failed to run module in active mode ('" << name << "')"
            << dendl;
     } else {
-      std::lock_guard l(lock);
       auto em = modules.emplace(name, active_module);
       ceph_assert(em.second); // actually inserted
 

diff --git a/src/mgr/ActivePyModules.h b/src/mgr/ActivePyModules.h
index 192360400c762bc0c8cd82110bdc296c8b62c21e..4892f2705fcda596095f2a527395c5e516f76fc8 100644 (file)
--- a/src/mgr/ActivePyModules.h
+++ b/src/mgr/ActivePyModules.h
@@ -39,6 +39,9 @@ class PyModuleRegistry;
 
 class ActivePyModules
 {
+  // module class instances not yet created
+  std::set<std::string, std::less<>> pending_modules;
+  // module class instances already created
   std::map<std::string, std::shared_ptr<ActivePyModule>> modules;
   PyModuleConfig &module_config;
   std::map<std::string, std::string> store_cache;
@@ -158,6 +161,9 @@ public:
                   const std::string &notify_id);
   void notify_all(const LogEntry &log_entry);
 
+  bool is_pending(std::string_view name) const {
+    return pending_modules.count(name) > 0;
+  }
   bool module_exists(const std::string &name) const
   {
     return modules.count(name) > 0;

diff --git a/src/mgr/PyModuleRegistry.cc b/src/mgr/PyModuleRegistry.cc
index 2e2e080aa76c00939b8f8ca4c96219cd91850cc4..5540339d6b0abde20dce63642a9a955ab45aeab1 100644 (file)
--- a/src/mgr/PyModuleRegistry.cc
+++ b/src/mgr/PyModuleRegistry.cc
@@ -377,6 +377,9 @@ void PyModuleRegistry::get_health_checks(health_check_map_t *checks)
       if (obsolete_modules.count(name)) {
        continue;
       }
+      if (active_modules->is_pending(name)) {
+       continue;
+      }
       if (!active_modules->module_exists(name)) {
         if (failed_modules.find(name) == failed_modules.end() &&
             dependency_modules.find(name) == dependency_modules.end()) {