mgr: update mon metadata when monmap is updated

there is chance that some monitor(s) is updated / upgraded in a single
monmap update without being removed from cluster state's metata first,
so, without this change, we will not update the metadata associated with
that monitor, hence the mgr modules which consumes the metadata is not
updated accordingly and keep reporting the stale information.

in this change, we always update the metadata associated with all monitor
included by the latest monmap. multiple "mon metadata" commands are sent
to monitor for retrieving their updated metadata, instead of sending a
single one, so that we can reuse "MetadataUpdate" to update the metadata
of a given daemon. as the number of monitors in a typical cluster is
relatively small, and the frequency of monmap update is low, so this
overhead should be fine.

unlike other places where we ask mon for metadata in Mgr class, the code
sending the mon command for updated monitor metata is located outside of
`cluster_state.with_monmap()` block, the reason is that `with_monmap()`
is guraded by the monc_lock under the hood, while `start_mon_command()`
also need to acquire the monc_lock, which is not a recursive lock. so we
have to do this out of the `with_monmap()` block.

Fixes: https://tracker.ceph.com/issues/48905
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c037f4cb5d7436879d58c34748ef516b5269781f)

backport:
  - path: src/mgr/Mgr.cc
    comment: nautilus don't declared `fmt`

diff --git a/src/mgr/Mgr.cc b/src/mgr/Mgr.cc
index 888d93892ece652aab5fe1455ac52fa636a1abe3..5dee63267ea9f563adbb6d16b1906aede353872f 100644 (file)
--- a/src/mgr/Mgr.cc
+++ b/src/mgr/Mgr.cc
@@ -509,6 +509,19 @@ void Mgr::handle_mon_map()
       names_exist.insert(monmap.get_name(i));
     }
   });
+  for (const auto& name : names_exist) {
+    const auto k = DaemonKey{"osd", name};
+    if (daemon_state.is_updating(k)) {
+      continue;
+    }
+    auto c = new MetadataUpdate(daemon_state, k);
+    std::ostringstream cmd;
+    cmd << "{\"prefix\": \"mon metadata\", \"id\": \""
+       << name << "\"}";
+    monc->start_mon_command(
+       {cmd.str()},
+       {}, &c->outbl, &c->outs, c);
+  }
   daemon_state.cull("mon", names_exist);
 }