tests: Protecting rados bench against endless loop

If the cluster dies during the rados bench, the maximum running time is
no more considered and all emitted aios are pending.

rados bench never quits and the global testing timeout (3600 sec : 1
hour) have to be reach to get a failure.

This situation is dramatic for a background test or a CI run as it locks
the whole job for too long for an event that will never occurs.

This ideal solution would be having 'rados bench' considering a failure
once the timeout is reached when aios are pending.

A possible workaround here is to put use the system command 'timeout'
before calling rados bench and fail if rados didn't completed on time.

To avoid side effects, this patch is doubling rados timeout. If rados
didn't completed after twice the expected time, it have to fail to avoid
locking the whole testing job.

Please find below the way it worked on a real test case.
We can see no IO after t>2 but despite timeout=4 the bench continue.
Thanks to this patch, the bench is stopped at t=8 and return 1.

5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon:  timeout 8 rados -p foo bench 4 write -b 4096 --no-cleanup
5: hints = 1
5: Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 4 seconds or 0 objects
5: Object prefix: benchmark_data_mr-meeseeks_184960
5:   sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
5:     0       0         0         0         0         0           -           0
5:     1      16      1144      1128   4.40538   4.40625  0.00412965   0.0141116
5:     2      16      2147      2131   4.16134   3.91797  0.00985654   0.0109079
5:     3      16      2147      2131   2.77424         0           -   0.0109079
5:     4      16      2147      2131    2.0807         0           -   0.0109079
5:     5      16      2147      2131   1.66456         0           -   0.0109079
5:     6      16      2147      2131   1.38714         0           -   0.0109079
5:     7      16      2147      2131   1.18897         0           -   0.0109079
5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon:  return 1
5: /home/erwan/ceph/src/test/smoke.sh:18: run:  return 1

Signed-off-by: Erwan Velu <erwan@redhat.com>

diff --git a/qa/standalone/osd/osd-dup.sh b/qa/standalone/osd/osd-dup.sh
index 9675d917fa2f8ad0bfd8a0987efb660a990220ec..324840b04b4ee79edf7239898a39d9885fc6fafa 100755 (executable)
--- a/qa/standalone/osd/osd-dup.sh
+++ b/qa/standalone/osd/osd-dup.sh
@@ -43,7 +43,7 @@ function TEST_filestore_to_bluestore() {
     create_pool foo 16
 
     # write some objects
-    rados bench -p foo 10 write -b 4096 --no-cleanup || return 1
+    timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup || return 1
 
     # kill
     while kill $osd_pid; do sleep 1 ; done

diff --git a/qa/standalone/osd/osd-fast-mark-down.sh b/qa/standalone/osd/osd-fast-mark-down.sh
index 8443d1da06f9b137e59f08d70488bef2bcc51792..6a95b226744617872d6d0085d4c5d27e8dc0985b 100755 (executable)
--- a/qa/standalone/osd/osd-fast-mark-down.sh
+++ b/qa/standalone/osd/osd-fast-mark-down.sh
@@ -61,7 +61,7 @@ function test_fast_kill() {
    create_rbd_pool || return 1
 
    # make some objects so osds to ensure connectivity between osds
-   rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup
+   timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup || return 1
    sleep 1
 
    killid=0

diff --git a/src/test/smoke.sh b/src/test/smoke.sh
index 1c79416cbd91c4bd4f235a83787a84c2c7360e87..d6b4cb8a8f9a653db7a3d6f0848a372c37e0a89a 100755 (executable)
--- a/src/test/smoke.sh
+++ b/src/test/smoke.sh
@@ -52,7 +52,7 @@ function TEST_multimon() {
     ceph osd out 0
     wait_for_clean
 
-    rados -p foo bench 4 write -b 4096 --no-cleanup
+    timeout 8 rados -p foo bench 4 write -b 4096 --no-cleanup || return 1
     wait_for_clean
 
     ceph osd in 0