From c4d3040b70d7911088ab5e60d5e1d11dec109f90 Mon Sep 17 00:00:00 2001 From: Patrick Donnelly Date: Wed, 9 Feb 2022 21:34:22 -0500 Subject: [PATCH] doc/dev: document nested subsets in teuthology See also: https://github.com/ceph/teuthology/pull/1704 Signed-off-by: Patrick Donnelly --- ...s-integration-testing-teuthology-intro.rst | 40 ++++++++++++++++++- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst index dc1cd4f0f95..ac8a66f2b01 100644 --- a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst +++ b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst @@ -316,8 +316,8 @@ signifies concatenation. Convolution operator ^^^^^^^^^^^^^^^^^^^^ -The convolution operator, implemented as an empty file called ``%``, tells -teuthology to construct a test matrix from yaml facets found in +The convolution operator, implemented as a (typically empty) file called ``%``, +tells teuthology to construct a test matrix from yaml facets found in subdirectories below the directory containing the operator. For example, the `ceph-deploy suite @@ -384,6 +384,34 @@ An individual test from the `ceph-deploy suite`_ can be run by adding the be combined with ``--filter``. This is because the ``--suite`` option understands POSIX relative paths only. +Nested Subsets +^^^^^^^^^^^^^^ + +Suites can get quite large with the combinatorial explosion of yaml +configurations. At the time of writing, the ``rados``` suite is more than +100,000 jobs. For this reason, scheduling often uses the ``--subset`` option to +only run a subset of the jobs (see also: :ref:`subset`). However, this applies +only at the top-level of the suite being run (e.g. ``fs``). That may +incidentally inflate the ratio of jobs for some larger sub-suites (like +``fs:workload``) vs. smaller but critical suites (like ``fs:volumes``). + +It is therefore attractive to automatically subset some sub-suites which are +never run fully. This is done by providing an integer divisor for the ``%`` +convolution operator file instead of leaving it empty. That divisor +automatically subsets the resulting matrix. For example, if the convolution +file ``%`` contains ``2``, the matrix will be divided into two using the same +logic as the ``--subset`` mechanism. + +Note the numerator is not specified as with the ``--subset`` option as there is +no meaningful way to express this when there could be several layers of +nesting. Instead, a random subset is selected (1 of 2 in our example). The +choice is based off the random seed (``--seed``) used for the scheduling. +Remember that seed is saved in the results so that a ``--rerun`` of failed +tests will still preserve the correct numerator (subset of subsets). + +You can disable nested subsets using the ``--no-nested-subset`` argument to +``teuthology-suite``. + Concatenation operator ^^^^^^^^^^^^^^^^^^^^^^ @@ -508,6 +536,9 @@ or Each string is looked up anywhere in the test description and has to be an exact match: they are not regular expressions. + +.. _subset: + Reducing the number of tests ---------------------------- @@ -537,6 +568,11 @@ teuthology will still ensure that all files in the suite are in at least one test. Understanding the actual logic that drives this requires reading the teuthology source code. +Note: some suites are now using a **nested subset** feature that automatically +applies a subset to a carefully chosen set of YAML configurations. You may +disable this behavior (for some custom filtering, perhaps) using the +``--no-nested-subset`` option. + The ``--limit`` option only runs the first ``N`` tests in the suite: this is rarely useful, however, because there is no way to control which test will be first. -- 2.39.5