pybind/mgr/devicehealth: skip legacy objects that cannot be loaded

Log looks like after test:

    2023-12-21T16:09:28.804+0000 7fbe7fd86700  0 [devicehealth DEBUG root] loading object ABC_DEADB33F_FA
    2023-12-21T16:09:28.805+0000 7fbe7fd86700  0 [devicehealth DEBUG root] object rados.Object(ioctx=<rados.Ioctx object at 0x7fbeee0c4668>,key=ABC_DEADB33F_FA,nspace=--default--,locator=None) does not exist because it is deleted in HEAD
    2023-12-21T16:09:28.805+0000 7fbe7fd86700  0 [devicehealth DEBUG root] finished reading legacy pool, complete = True

Credit to Greg Farnum for postulating the cause.

Fixes: https://tracker.ceph.com/issues/63882
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5e6fc0bf5f52732966d5cf2987e679abee8a384d)

diff --git a/src/pybind/mgr/devicehealth/module.py b/src/pybind/mgr/devicehealth/module.py
index 86ea1d280720c58ed0fa5540f135c3fd00ae5cd2..e4356175c6a4298a07c117b9d779e14651a9cb14 100644 (file)
--- a/src/pybind/mgr/devicehealth/module.py
+++ b/src/pybind/mgr/devicehealth/module.py
@@ -327,6 +327,11 @@ CREATE TABLE DeviceHealthMetrics (
                         count += 1
                 except json.decoder.JSONDecodeError:
                     pass
+                except rados.ObjectNotFound:
+                    # https://tracker.ceph.com/issues/63882
+                    # Sometimes an object appears in the pool listing but cannot be interacted with?
+                    self.log.debug(f"object {obj} does not exist because it is deleted in HEAD")
+                    pass
                 if count >= 10:
                     break
             done = count < 10