Re-create guest if it doesn't come up right.

In some rare cases (mainly centos/rhel after creating the
guest with downburst it does not come up right. It
gets a kernel panic at boot. Usually just turning it off
and then back on again is enough but to be on the safe
side I figured it should be re-created instead. This
insures you don't get hung jobs from a guest that didn't
come up correctly.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

diff --git a/teuthology/task/internal.py b/teuthology/task/internal.py
index 96a54e240e24ad578bf75555b5acbaf4366fa60d..2e5372616e0819aa662049ec490775843a630da0 100644 (file)
--- a/teuthology/task/internal.py
+++ b/teuthology/task/internal.py
@@ -103,10 +103,20 @@ def lock_machines(ctx, config):
             if vmlist:
                 log.info('Waiting for virtual machines to come up')
                 keyscan_out = ''
+                loopcount=0
                 while len(keyscan_out.splitlines()) != len(vmlist):
+                    loopcount += 1
                     time.sleep(10)
                     keyscan_out, current_locks = lock.keyscan_check(ctx, vmlist)
                     log.info('virtual machine is stil unavailable')
+                    if loopcount == 30:
+                        loopcount = 0
+                        log.info('virtual machine(s) still not up, recreating unresponsive ones.')
+                        for guest in vmlist:
+                            if guest not in keyscan_out:
+                                log.info('recreating: ' + guest)
+                                lock.destroy_if_vm(ctx, 'ubuntu@' + guest)
+                                lock.create_if_vm(ctx, 'ubuntu@' + guest)
                 if lock.update_keys(ctx, keyscan_out, current_locks):
                     log.info("Error in virtual machine keys")
                 newscandict = {}