mgr: fix MgrModuleRecoverDB to honor its retry budget on db cleanup failure

Both close_db() and open_db() during retry were unguarded, so a failure
in either escaped the decorator immediately instead of retrying up to
MAX_DBCLEANUP_RETRIES.

Wrapped both in the same try/except, and logged the
failed attempt.

Fixes: https://tracker.ceph.com/issues/77635
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

diff --git a/src/pybind/mgr/mgr_module.py b/src/pybind/mgr/mgr_module.py
index 192eefbf96ee9c296b16c01f5a31b14580d7b7b6..3ba6cb3277ca9225df1d03367faf319a803903cb 100644 (file)
--- a/src/pybind/mgr/mgr_module.py
+++ b/src/pybind/mgr/mgr_module.py
@@ -619,8 +619,13 @@ def MgrModuleRecoverDB(func: Callable) -> Callable:
                 if retries > MAX_DBCLEANUP_RETRIES:
                     raise
                 self.log.debug("attempting reopen of database")
-                self.close_db()
-                self.open_db()
+                try:
+                    self.close_db()
+                    self.open_db()
+                except sqlite3.DatabaseError as e2:
+                    self.log.warning(
+                        f"reopen attempt {retries}/{MAX_DBCLEANUP_RETRIES} failed: {e2}"
+                    )
                 # allow retry of func(...)
     check.__signature__ = inspect.signature(func)  # type: ignore[attr-defined]
     return check