ceph-disk: retry cryptsetup remove

Retry a cryptsetup remove ten times. After the ceph-osd terminates, the
device is released asyncrhonously and an attempt to cryptsetup remove
will may fail because it is considered busy. Although a few attempts are
made before giving up, the number of attempts / the duration of the
attempts cannot be controlled with a cryptsetup option. The workaround
is to increase this by trying a few times.

If cryptsetup remove fails for a reason that is unrelated to timeout,
the error will be repeated a few times. There is no undesirable side
effect. It will not hide a problem.

Signed-off-by: Loic Dachary <loic@dachary.org>

diff --git a/src/ceph-disk b/src/ceph-disk
index 3ce7a7acc9fd5c548a6e495076211e00f76a309a..bb819582c10a716bffd888369b5ef504989bc2a1 100755 (executable)
--- a/src/ceph-disk
+++ b/src/ceph-disk
@@ -1088,17 +1088,17 @@ def dmcrypt_unmap(
     """
     Removes the dmcrypt device with the given UUID.
     """
-    args = [
-        'cryptsetup',
-        'remove',
-        _uuid
-    ]
-
-    try:
-        command_check_call(args)
-
-    except subprocess.CalledProcessError as e:
-        raise Error('unable to unmap device', _uuid, e)
+    retries = 0
+    while True:
+        try:
+            command_check_call(['cryptsetup', 'remove', _uuid])
+            break
+        except subprocess.CalledProcessError as e:
+            if retries == 10:
+                raise Error('unable to unmap device', _uuid, e)
+            else:
+                time.sleep(0.5 + retries * 1.0)
+                retries += 1
 
 
 def mount(