doc/dev: document nested subsets in teuthology

See also: https://github.com/ceph/teuthology/pull/1704

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

diff --git a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst
index dc1cd4f0f95e2eb1a576c599ba99430606779aa6..ac8a66f2b01fe5815350840d765f5334a8378712 100644 (file)
--- a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst
+++ b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst
@@ -316,8 +316,8 @@ signifies concatenation.
 Convolution operator
 ^^^^^^^^^^^^^^^^^^^^
 
-The convolution operator, implemented as an empty file called ``%``, tells
-teuthology to construct a test matrix from yaml facets found in
+The convolution operator, implemented as a (typically empty) file called ``%``,
+tells teuthology to construct a test matrix from yaml facets found in
 subdirectories below the directory containing the operator.
 
 For example, the `ceph-deploy suite
@@ -384,6 +384,34 @@ An individual test from the `ceph-deploy suite`_ can be run by adding the
    be combined with ``--filter``. This is because the ``--suite`` option
    understands POSIX relative paths only.
 
+Nested Subsets
+^^^^^^^^^^^^^^
+
+Suites can get quite large with the combinatorial explosion of yaml
+configurations. At the time of writing, the ``rados``` suite is more than
+100,000 jobs. For this reason, scheduling often uses the ``--subset`` option to
+only run a subset of the jobs (see also: :ref:`subset`). However, this applies
+only at the top-level of the suite being run (e.g. ``fs``). That may
+incidentally inflate the ratio of jobs for some larger sub-suites (like
+``fs:workload``) vs.  smaller but critical suites (like ``fs:volumes``).
+
+It is therefore attractive to automatically subset some sub-suites which are
+never run fully. This is done by providing an integer divisor for the ``%``
+convolution operator file instead of leaving it empty. That divisor
+automatically subsets the resulting matrix. For example, if the convolution
+file ``%`` contains ``2``, the matrix will be divided into two using the same
+logic as the ``--subset`` mechanism.
+
+Note the numerator is not specified as with the ``--subset`` option as there is
+no meaningful way to express this when there could be several layers of
+nesting.  Instead, a random subset is selected (1 of 2 in our example). The
+choice is based off the random seed (``--seed``) used for the scheduling.
+Remember that seed is saved in the results so that a ``--rerun`` of failed
+tests will still preserve the correct numerator (subset of subsets).
+
+You can disable nested subsets using the ``--no-nested-subset`` argument to
+``teuthology-suite``.
+
 Concatenation operator
 ^^^^^^^^^^^^^^^^^^^^^^
 
@@ -508,6 +536,9 @@ or
 Each string is looked up anywhere in the test description and has to
 be an exact match: they are not regular expressions.
 
+
+.. _subset:
+
 Reducing the number of tests
 ----------------------------
 
@@ -537,6 +568,11 @@ teuthology will still ensure that all files in the suite are in at
 least one test. Understanding the actual logic that drives this
 requires reading the teuthology source code.
 
+Note: some suites are now using a **nested subset** feature that automatically
+applies a subset to a carefully chosen set of YAML configurations. You may
+disable this behavior (for some custom filtering, perhaps) using the
+``--no-nested-subset`` option.
+
 The ``--limit`` option only runs the first ``N`` tests in the suite:
 this is rarely useful, however, because there is no way to control which
 test will be first.