openstack: flock timeout on openstack server create

The global lock used to run openstack server create must timeout
otherwise it may hang forever. The timeout is set to 8h. Assuming one
server create --wait takes ~1mn to return, that should be more than
enough to absorbe a burst of 200 jobs provisioning instances at the same
time.

The goal here is to ensure the queue does not stay blocked forever, not
to timeout jobs because they cannot get a server in a given delay.

http://tracker.ceph.com/issues/14243 Fixes: #14243

Signed-off-by: Loic Dachary <loic@dachary.org>

diff --git a/teuthology/provision.py b/teuthology/provision.py
index 9a5b80473f289e836c9cfe173f42b53e4250a1ab..4d2db6684672e3c7c8e32c1561b76d920e58d091 100644 (file)
--- a/teuthology/provision.py
+++ b/teuthology/provision.py
@@ -307,7 +307,8 @@ class ProvisionOpenStack(OpenStack):
             net = ''
         flavor = self.flavor(resources_hint['machine'],
                              config['openstack'].get('flavor-select-regexp'))
-        misc.sh("flock --close /tmp/teuthology-server-create.lock openstack server create" +
+        misc.sh("flock --close /tmp/teuthology-server-create.lock --timeout 28800" +
+                " openstack server create" +
                 " " + config['openstack'].get('server-create', '') +
                 " -f json " +
                 " --image '" + str(image) + "'" +