qa/tasks: force umount during kclient teardown

Previously we could readily end up hanging on teardown
when something had gone wrong with umount.  Forcing
is a big hammer (umount_wait will power cycle the node
if umount isn't working), so if we had to do that
then raise an exception to indicate that something
was wrong with the test.

Fixes: http://tracker.ceph.com/issues/18663
Signed-off-by: John Spray <john.spray@redhat.com>

diff --git a/qa/tasks/kclient.py b/qa/tasks/kclient.py
index c680ba78bfae1b72a7d37b144a2fbae5b0665832..454300a2a7c254b5477cacccdc14a600b510ddc8 100644 (file)
--- a/qa/tasks/kclient.py
+++ b/qa/tasks/kclient.py
@@ -104,11 +104,32 @@ def task(ctx, config):
 
         kernel_mount.mount()
 
+
+    def umount_all():
+        log.info('Unmounting kernel clients...')
+
+        forced = False
+        for mount in mounts.values():
+            if mount.is_mounted():
+                try:
+                    mount.umount()
+                except CommandFailedError:
+                    log.warn("Ordinary umount failed, forcing...")
+                    forced = True
+                    mount.umount_wait(force=True)
+
+        return forced
+
     ctx.mounts = mounts
     try:
         yield mounts
+    except:
+        umount_all()  # ignore forced retval, we are already in error handling
     finally:
-        log.info('Unmounting kernel clients...')
-        for mount in mounts.values():
-            if mount.is_mounted():
-                mount.umount()
+
+        forced = umount_all()
+        if forced:
+            # The context managers within the kclient manager worked (i.e.
+            # the test workload passed) but for some reason we couldn't
+            # umount, so turn this into a test failure.
+            raise RuntimeError("Kernel mounts did not umount cleanly")