mgr/dashboard: don't mutate the cached osd_map in CephService

test_pool_list fails intermittently:

  Traceback (most recent call last):
    File "qa/tasks/mgr/dashboard/test_pool.py", line 182, in test_pool_list
      self.assertNotIn('pg_status', pool)
  AssertionError: 'pg_status' unexpectedly found in
    {'pool': 1, 'pool_name': 'rbd', ..., 'pg_status': {'active+clean': 1}, ...}

mgr.get('osd_map') defaults to mutable=False, so cacheable_get_python()
returns the mgr's shared cached object rather than a copy.
get_pool_list_with_stats() writes pool['pg_status'] and pool['stats']
into those cached dicts, and get_erasure_code_profiles() sets ecp['name']
and rewrites ecp['k']/['m'] to int. The writes outlive the request, so
once a stats=true call has run, GET /api/pool with stats=false still
returns pools carrying pg_status and the assertion above fails. It only
triggers while the cache stays valid between the two requests, hence the
flakiness.

Audited the other dashboard readers of cached mgr.get() keys: these two
are the only sites that mutate the result; the rest only read, and
health.py already copies its osd_map before editing.

Copy the dicts before stamping them; the cache stays clean.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

diff --git a/src/pybind/mgr/dashboard/services/ceph_service.py b/src/pybind/mgr/dashboard/services/ceph_service.py
index 0dc2c03421a45f0f0c5ba0816015cc48f1bd3cc8..a35195a96007d66e8203dede1d0b9e93efc86433 100644 (file)
--- a/src/pybind/mgr/dashboard/services/ceph_service.py
+++ b/src/pybind/mgr/dashboard/services/ceph_service.py
@@ -129,7 +129,9 @@ class CephService(object):
     @classmethod
     def get_pool_list_with_stats(cls, application=None):
         # pylint: disable=too-many-locals
-        pools = cls.get_pool_list(application)
+        # copy each pool: get_pool_list returns mgr's cached osd_map dicts, and
+        # stamping pg_status/stats below would otherwise leak into stats=False callers
+        pools = [dict(pool) for pool in cls.get_pool_list(application)]
 
         pools_w_stats = []
 
@@ -166,8 +168,10 @@ class CephService(object):
             return ecp
 
         ret = []
+        # copy each ecp: mgr.get('osd_map') returns cached dicts, and _serialize_ecp
+        # mutates name/k/m in place, which would pollute the shared cache otherwise
         for name, ecp in mgr.get('osd_map').get('erasure_code_profiles', {}).items():
-            ret.append(_serialize_ecp(name, ecp))
+            ret.append(_serialize_ecp(name, dict(ecp)))
         return ret
 
     @classmethod