From: Nathan Cutler <ncutler@suse.com>
Date: Thu, 23 Mar 2017 13:47:29 +0000 (+0100)
Subject: provision/openstack: put "add volume" in a retry loop
X-Git-Tag: 1.1.0~211^2~25
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=5f25e89a44e9a0511b610426e054f50e77b1f5e6;p=teuthology.git

provision/openstack: put "add volume" in a retry loop

Attempted workaround for this issue:

2017-03-23T12:33:20.684 DEBUG:teuthology.misc::sh: openstack --quiet volume show  -f json target217182140122-1
2017-03-23T12:34:22.140 DEBUG:teuthology.misc:{
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"size": 10,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"status": "available",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"properties": "ownedby='217.182.143.170'",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"user_id": "122b249e82994f5aa8ec6a31b7891df5",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"description": null,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"availability_zone": "nova",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"bootable": "false",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"encrypted": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"created_at": "2017-03-23T12:33:17.000000",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"multiattach": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:driver_data": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"name": "target217182140122-1",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"snapshot_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"consistencygroup_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"replication_status": "disabled",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-vol-tenant-attr:tenant_id": "43583fef7b734bb1b6fb5f86d48abe5e",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"source_volid": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:extended_status": null,
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"type": "classic",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"id": "e6949180-87fe-4f4c-b0a0-e3ea35bdb2e2",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"attachments": []
2017-03-23T12:34:25.036 DEBUG:teuthology.misc:}
2017-03-23T12:34:25.042 DEBUG:teuthology.misc::sh: openstack server add volume target217182140122 target217182140122-1
2017-03-23T12:35:30.536 DEBUG:teuthology.misc:No volume with a name or ID of 'target217182140122-1' exists.
2017-03-23T12:35:34.512 ERROR:teuthology.provision.openstack:Command 'openstack server add volume target217182140122 target217182140122-1' returned non-zero exit status 1

Fixes: http://tracker.ceph.com/issues/17659
Signed-off-by: Nathan Cutler <ncutler@suse.com>
---

diff --git a/teuthology/provision/openstack.py b/teuthology/provision/openstack.py
index 3ff7849e6..eeef9a397 100644
--- a/teuthology/provision/openstack.py
+++ b/teuthology/provision/openstack.py
@@ -92,7 +92,14 @@ class ProvisionOpenStack(OpenStack):
                                      " not information available yet")
             # do not use OpenStack().run because its
             # bugous for volume
-            misc.sh("openstack server add volume " + name + " " + volume_name)
+            with safe_while(sleep=20, increment=20, tries=10,
+                            action="add volume " + volume_name) as proceed:
+                while proceed():
+                    try:
+                        misc.sh("openstack server add volume " + name + " " + volume_name)
+                        break
+                    except subprocess.CalledProcessError:
+                        log.warning("openstack add volume failed unexpectedly; retrying")
 
     @staticmethod
     def ip2name(prefix, ip):