From: Sage Weil <sage@inktank.com>
Date: Sun, 23 Jun 2013 23:21:45 +0000 (-0700)
Subject: dump_stuck: fix race with osd start
X-Git-Tag: 1.1.0~2101^2
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=refs%2Fpull%2F16%2Fhead;p=teuthology.git

dump_stuck: fix race with osd start

Occasionally we don't wait long enough for the osd to start and
mark itself up.  Keep trying until flush succeeds.

Fixes: #5431
Signed-off-by: Sage Weil <sage@inktank.com>
---

diff --git a/teuthology/task/dump_stuck.py b/teuthology/task/dump_stuck.py
index d6fd1b098..d7dbf7daf 100644
--- a/teuthology/task/dump_stuck.py
+++ b/teuthology/task/dump_stuck.py
@@ -112,9 +112,14 @@ def task(ctx, config):
     for id_ in teuthology.all_roles_of_type(ctx.cluster, 'osd'):
         manager.revive_osd(id_)
         manager.mark_in_osd(id_)
-    time.sleep(timeout)
-    manager.raw_cluster_cmd('tell', 'osd.0', 'flush_pg_stats')
-    manager.raw_cluster_cmd('tell', 'osd.1', 'flush_pg_stats')
+    while True:
+        try:
+            manager.raw_cluster_cmd('tell', 'osd.0', 'flush_pg_stats')
+            manager.raw_cluster_cmd('tell', 'osd.1', 'flush_pg_stats')
+            break
+        except:
+            log.debug('osds must not be started yet, waiting...')
+            time.sleep(1)
     manager.wait_for_clean(timeout)
 
     check_stuck(