provision/openstack: put "add volume" in a retry loop

Attempted workaround for this issue:

2017-03-23T12:33:20.684 DEBUG:teuthology.misc::sh: openstack --quiet volume show  -f json target217182140122-1
2017-03-23T12:34:22.140 DEBUG:teuthology.misc:{
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"size": 10,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"status": "available",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"properties": "ownedby='217.182.143.170'",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"user_id": "122b249e82994f5aa8ec6a31b7891df5",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"description": null,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"availability_zone": "nova",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"bootable": "false",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"encrypted": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"created_at": "2017-03-23T12:33:17.000000",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"multiattach": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:driver_data": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"name": "target217182140122-1",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"snapshot_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"consistencygroup_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"replication_status": "disabled",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-vol-tenant-attr:tenant_id": "43583fef7b734bb1b6fb5f86d48abe5e",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"source_volid": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:extended_status": null,
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"type": "classic",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"id": "e6949180-87fe-4f4c-b0a0-e3ea35bdb2e2",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"attachments": []
2017-03-23T12:34:25.036 DEBUG:teuthology.misc:}
2017-03-23T12:34:25.042 DEBUG:teuthology.misc::sh: openstack server add volume target217182140122 target217182140122-1
2017-03-23T12:35:30.536 DEBUG:teuthology.misc:No volume with a name or ID of 'target217182140122-1' exists.
2017-03-23T12:35:34.512 ERROR:teuthology.provision.openstack:Command 'openstack server add volume target217182140122 target217182140122-1' returned non-zero exit status 1

Fixes: http://tracker.ceph.com/issues/17659
Signed-off-by: Nathan Cutler <ncutler@suse.com>

diff --git a/teuthology/provision/openstack.py b/teuthology/provision/openstack.py
index 3ff7849e672b47f5e0d2583fb5c7b960ffa93fb3..eeef9a3972eb024d405d0d73240a8fbbdce589f4 100644 (file)
--- a/teuthology/provision/openstack.py
+++ b/teuthology/provision/openstack.py
@@ -92,7 +92,14 @@ class ProvisionOpenStack(OpenStack):
                                      " not information available yet")
             # do not use OpenStack().run because its
             # bugous for volume
-            misc.sh("openstack server add volume " + name + " " + volume_name)
+            with safe_while(sleep=20, increment=20, tries=10,
+                            action="add volume " + volume_name) as proceed:
+                while proceed():
+                    try:
+                        misc.sh("openstack server add volume " + name + " " + volume_name)
+                        break
+                    except subprocess.CalledProcessError:
+                        log.warning("openstack add volume failed unexpectedly; retrying")
 
     @staticmethod
     def ip2name(prefix, ip):