openstack: throttle job scheduling by default

When running a suite with 50 jobs, it will schedule about 100 instance
creation within less than a minute. It is likely to exceed the API
quotas of the OpenStack provider (number of instance creation per
minute) and lead to instance creation failures.

Set the teuthology-suite to be called with --throttle 15 by default,
that is about 60 / 15 = 4 * 2 = ~8 server creation per minute.

http://tracker.ceph.com/issues/12977 Fixes: #12977

Signed-off-by: Loic Dachary <loic@dachary.org>

diff --git a/scripts/openstack.py b/scripts/openstack.py
index 4038efd2d90b60728ab7da41bf57efe7a9d36356..e0b9e049b2d213d92c4c6d145e73f1e501031580 100644 (file)
--- a/scripts/openstack.py
+++ b/scripts/openstack.py
@@ -157,5 +157,14 @@ and analyze results.
               'the keywords in the comma separated keyword '
               'string specified. ')
     )
+    parser.add_argument(
+        '--throttle',
+        help=('When scheduling, wait SLEEP seconds between jobs. '
+              'Useful to avoid bursts that may be too hard on '
+              'the underlying infrastructure or exceed OpenStack API '
+              'limits (server creation per minute for instance).'),
+        type=int,
+        default=15,
+    )
 
     return parser.parse_args(argv)

diff --git a/teuthology/openstack/test/test_openstack.py b/teuthology/openstack/test/test_openstack.py
index 9d8dcc6d7a4853730296b66d62861d308df79990..20f2cd3c36352a003ebdb59d9f5861661a5ab129 100644 (file)
--- a/teuthology/openstack/test/test_openstack.py
+++ b/teuthology/openstack/test/test_openstack.py
@@ -93,6 +93,7 @@ openstack keypair delete {key_name} || true
             '--timeout', '234',
             '--filter', 'trasher',
             '--filter-out', 'erasure-code',
+            '--throttle', '3',
         ]
         argv = (self.options +
                 ['--upload',