cephadm: bootstrap: verify orch module is running before setting cephadm backend

There have been occasional failures when bootstrapping with error
messages like

```
Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
```

despite the orchestrator being meant to be an always on module.

Since trying to enable an already enabled module causes no issues,
trying to enable it ourselves seems like a sensible workaround.

Fixes: https://tracker.ceph.com/issues/67969
Signed-off-by: Adam King <adking@redhat.com>

diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py
index e71addf7bfa5a6a05cf2bb08b3bd4d66f35affe1..a0c7c7f31e0859d85fdcba6442ed26dbcfe78a8d 100755 (executable)
--- a/src/cephadm/cephadm.py
+++ b/src/cephadm/cephadm.py
@@ -2408,6 +2408,12 @@ def enable_cephadm_mgr_module(
     logger.info('Enabling cephadm module...')
     cli(['mgr', 'module', 'enable', 'cephadm'])
     wait_for_mgr_restart()
+    # https://tracker.ceph.com/issues/67969
+    # luckily `ceph mgr module enable <module>` returns
+    # a zero rc when the module is already enabled so
+    # this is no issue even if it is unnecessary
+    logger.info('Verifying orchestrator module is enabled...')
+    cli(['mgr', 'module', 'enable', 'orchestrator'])
     logger.info('Setting orchestrator backend to cephadm...')
     cli(['orch', 'set', 'backend', 'cephadm'])