run: do not block on greenlets after command exits

The stdout/stderr greenlets will not necessarily exit when the command does if
child processes are stuck in an uninterruptible sleep. For example, the fsx.sh
workunit spawns fsx processes that may be left behind in the D state after
/bin/timeout kills fsx.sh. These are connected to the stdout/stderr pipes which
prevent the greenlets from exiting normally.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

diff --git a/teuthology/orchestra/run.py b/teuthology/orchestra/run.py
index 50d01ae7464ceac0a1214e8bef3f2d7aee69a5dd..7a0fe160658621eed0c9363665ce7884d65a21a0 100644 (file)
--- a/teuthology/orchestra/run.py
+++ b/teuthology/orchestra/run.py
@@ -139,10 +139,16 @@ class RemoteProcess(object):
 
         :returns: self.returncode
         """
-        for greenlet in self.greenlets:
-            greenlet.get()
 
         status = self._get_exitstatus()
+        if status != 0:
+            log.debug("got remote process result: {}".format(status))
+        for greenlet in self.greenlets:
+            try:
+                greenlet.get(block=True,timeout=60)
+            except gevent.Timeout:
+                log.debug("timed out waiting; will kill: {}".format(greenlet))
+                greenlet.kill(block=False)
         for stream in ('stdout', 'stderr'):
             if hasattr(self, stream):
                 stream_obj = getattr(self, stream)