Add interactive-on-error, to pause and explore on error.

Closes: http://tracker.newdream.net/issues/1291

diff --git a/README.rst b/README.rst
index 0330aa65c28ed99e73ab65dc989d75f37591a331..2fdb134a09018b711e5afc870100c8bfa2788258 100644 (file)
--- a/README.rst
+++ b/README.rst
@@ -165,3 +165,25 @@ reverse order, both on success and failure. A nice way of writing
 context managers is the ``contextlib.contextmanager`` decorator; look
 for that string in the existing tasks to see examples, and note where
 they use ``yield``.
+
+
+Troubleshooting
+===============
+
+Sometimes when a bug triggers, instead of automatic cleanup, you want
+to explore the system as is. Adding a top-level::
+
+       interactive-on-error: true
+
+as a config file for ``teuthology`` will make that possible. With that
+option, any *task* that fails, will have the ``interactive`` task
+called after it. This means that before any cleanup happens, you get a
+change to inspect the system -- both through Teuthology and via extra
+SSH connections -- and the cleanup completes only when you choose so.
+Just exit the interactive Python session to continue the cleanup.
+
+TODO: this only catches exceptions *between* the tasks. If a task
+calls multiple subtasks, e.g. with ``contextutil.nested``, those
+cleanups *will* be performed. Later on, we can let tasks communicate
+the subtasks they wish to invoke to the top-level runner, avoiding
+this issue.

diff --git a/teuthology/run_tasks.py b/teuthology/run_tasks.py
index fde6cb6f8a7e85b2ac09bebd1f5076bbed228da7..dcd94689e6f69ba831a6cd4f4498c6102d41c88a 100644 (file)
--- a/teuthology/run_tasks.py
+++ b/teuthology/run_tasks.py
@@ -29,6 +29,10 @@ def run_tasks(tasks, ctx):
     except:
         ctx.summary['success'] = False
         log.exception('Saw exception from tasks')
+        if ctx.config.get('interactive-on-error'):
+            from .task import interactive
+            log.warning('Saw failure, going into interactive mode...')
+            interactive.task(ctx=ctx, config=None)
     finally:
         try:
             exc_info = sys.exc_info()