From: Loic Dachary Date: Wed, 16 Dec 2015 18:50:55 +0000 (+0100) Subject: doc: dev: document ceph-qa-suite X-Git-Tag: v10.0.3~113^2 X-Git-Url: http://git.apps.os.sepia.ceph.com/?a=commitdiff_plain;h=279e9fb1e72a8122f3f059ea37d3cc724ebdf3b5;p=ceph.git doc: dev: document ceph-qa-suite Signed-off-by: Loic Dachary --- diff --git a/doc/dev/index.rst b/doc/dev/index.rst index c53ffd61e8a74..6a726963bbb03 100644 --- a/doc/dev/index.rst +++ b/doc/dev/index.rst @@ -211,7 +211,7 @@ workflows. \----------------/ | ^ | fix changes | | make check - | review | ceph--workbench ceph--qa--suite + | review | teuthology-suite | | git commit | v /--------------\ /-------------\ @@ -310,6 +310,266 @@ regressions, or to analyze test failures when they do occur. .. _`teuthology framework`: https://github.com/ceph/teuthology .. _`ceph-workbench ceph-qa-suite`: http://ceph-workbench.readthedocs.org/ +Integration tests : ceph-qa-suite +================================= + +This is an introduction to integration tests. A detailed description +of each option is available from ``teuthology-suite --help``. + +Reading a standalone integration test +------------------------------------- + +A test is defined by yaml files found in the ``suites`` subdirectory +of the `ceph-qa-suite repository`_ and implemented by python code +found in the ``tasks`` subdirectory. Here is a commented example using +`rados/singleton/all/admin-socket.yaml `_ :: + + roles: + - - mon.a + - osd.0 + - osd.1 + tasks: + - install: + - ceph: + - admin_socket: + osd.0: + version: + git_version: + help: + config show: + config set filestore_dump_file /tmp/foo: + perf dump: + perf schema: + +The ``roles`` array determines the composition of the cluster (how +many MONs, OSDs, etc.) on which this test is designed to run, as well +as how these roles will be distributed over the machines in the +testing cluster. In this case, there is only one element in the +top-level array: therefore, only one machine is allocated to the +test. The nested array declares that this machine shall run a MON with +id ``a`` (that is the ``mon.a`` in the list of roles) and two OSDs +(``osd.0`` and ``osd.1``). + +The body of the test is in the ``tasks`` array: each element is +evaluated in order and runs the corresponding python file found in the +``tasks`` subdirectory of the `teuthology repository`_ or +`ceph-qa-suite repository`_. The `install +`_ +task comes first and installs the Ceph packages on each machine (as +defined by the ``roles`` array). A full description of the ``install`` +task is `found in the python file +`_. + +The `ceph task +`_ +starts OSDs and MONs as required by the ``roles`` array. It will start +one MON (``mon.a``) and two OSDs (``osd.0`` and ``osd.1``), on the same machine. + +Once the Ceph cluster is healthy, the `admin_socket task +`_ +starts. The parameter of the ``admin_socket`` task (and any other +task) is a structure which is interpreted as documented in the +task. In this example the parameters are a set of commands to be sent +to the admin socket of ``osd.0``. The task verifies that each of them returns +on success (i.e. exit code zero). + +This test can be run with:: + + teuthology-suite --suite rados/singleton/all/admin-socket.yaml + +How are tests built from directories ? +-------------------------------------- + +Most tests are not a single file but the concatenation of files +collected from a tree. For instance, the `ceph-disk suite +`_ +is as follows:: + + directory: ceph-disk/basic + file: % + directory: distros + file: centos_7.0.yaml + file: ubuntu_14.04.yaml + directory: tasks + file: ceph-disk.yaml + +This is interpreted as two tests: + +* the concatenation of centos_7.0.yaml and ceph-disk.yaml +* the concatenation of ubuntu_14.04.yaml and ceph-disk.yaml + +Meaning the task found in ``ceph-disk.yaml`` is intended to run on +both CentOS 7.0 and Ubuntu 14.04. + +The special file percent (``%``) is interpreted as a requirement to +generate tests combining all files found in the current directory and +in its direct subdirectories. Without the file percent, the +``ceph-disk`` tree would create three independant tests: + +* ceph-disk/basic/distros/centos_7.0.yaml +* ceph-disk/basic/distros/ubuntu_14.04.yaml +* ceph-disk/basic/distros/ceph-disk.yaml + +To share parts of the test description between suites, the special +file plus (``+``) can be used to concatenate them. For instance:: + + directory: rbd/thrash + file: % + directory: clusters + file: + + file: fixed-2.yaml + file: openstack.yaml + directory: workloads + file: rbd_api_tests_copy_on_read.yaml + file: rbd_api_tests.yaml + +creates two tests: + +* rbd/thrash/{clusters/fixed-2.yaml, clusters/openstack.yaml, + workloads/rbd_api_tests_copy_on_read.yaml} +* rbd/thrash/{clusters/fixed-2.yaml, clusters/openstack.yaml, + workloads/rbd_api_tests.yaml} + +Because of the special file plus (``+``), ``fixed-2.yaml`` and +``openstack.yaml`` are concatenated together and treated as a single +file. Without the special file plus, they would have been combined +with the files from the workloads directory to create four tests: + +* rbd/thrash/{clusters/openstack.yaml, workloads/rbd_api_tests_copy_on_read.yaml} +* rbd/thrash/{clusters/openstack.yaml, workloads/rbd_api_tests.yaml} +* rbd/thrash/{clusters/fixed-2.yaml, workloads/rbd_api_tests_copy_on_read.yaml} +* rbd/thrash/{clusters/fixed-2.yaml, workloads/rbd_api_tests.yaml} + +The ``clusters/fixed-2.yaml`` file is shared among many suites to +define the following ``roles``:: + + roles: + - [mon.a, mon.c, osd.0, osd.1, osd.2, client.0] + - [mon.b, osd.3, osd.4, osd.5, client.1] + +The tests generated from the ``ceph-disk`` directory can be run with:: + + teuthology-suite --suite ceph-disk + +.. _`teuthology repository`: https://github.com/ceph/teuthology/ + +Test descriptions are unique identifiers +---------------------------------------- + +Each test is uniquely identified by its description which is made of +the names of all files concatenated together. For instance the test:: + + ceph-disk/basic/{distros/centos_7.0.yaml tasks/ceph-disk.yaml} + +is the concatenation of the files: + +* ceph-disk/basic/distros/centos_7.0.yaml +* ceph-disk/basic/tasks/ceph-disk.yaml + +Filtering tests by their description +------------------------------------ + +When a few jobs fail and need to be run again, the ``--filter`` option +will select the tests with a matching description. For instance if the +``rados`` suite fails the `all/peer.yaml `_ test, the following will only run the tests that contain this file:: + + teuthology-suite --suite rados --filter all/peer.yaml + +The ``--filter-out`` option does the opposite (it matches test that do +not contain a given string), and can be combined with the ``--filter`` +option. + +Both --filter and --filter-out take a comma-separated list of strings (which +means comma are implicitly forbidden in filenames found in the +`ceph-qa-suite repository`_). For instance:: + + teuthology-suite --suite rados --filter all/peer.yaml,all/rest-api.yaml + +will run tests that contain either +`all/peer.yaml `_ +or +`all/rest-api.yaml `_ + +Each string is looked up anywhere in the test description and has to +be an exact match: they are not regular expressions. + +Reducing the number of tests +---------------------------- + +The rados suite generates thousands of tests out of a few hundred +files. For instance all tests in the `rados/thrash suite `_ run for ``ext4``, ``xfs`` and ``btrfs`` because they are combined (the ``%`` file system) +with the `fs directory `_ + +All these tests are required before a Ceph release is published but it +is too much when verifying a contribution can be merged without +risking a trivial regression. The --subset option can be used to +reduce the number of tests that are triggered. For instance:: + + teuthology-suite --suite rados --subset 0/4000 + +will run as few tests as possible. The tradeoff is that some tests +will only run on ``ext4`` and not on ``btrfs``, but all files in the +suite will be in at least one test. + +The ``--limit`` option only runs the first ``N`` tests in the suite: +this is however rarely useful because there is no way to control which test +will be first. + +Inventory +--------- + +The ``suites`` directory of the `ceph-qa-suite repository`_ contains +all the integration tests, for all the Ceph components. + +`ceph-deploy `_ + install a Ceph cluster with `ceph-deploy`_ + +`ceph-disk `_ + verify init scripts (upstart etc.) and udev integration with + `ceph-disk`_, with and without dmcrypt support. + +`dummy `_ + get a machine, do nothing and return success (commonly used to + verify the integration testing infrastructure works as expected) + expected + +`fs `_ + test CephFS + +`kcephfs `_ + test the CephFS kernel module + +`krbd `_ + test the RBD kernel module + +`powercycle `_ + verify the Ceph cluster behaves when machines are powered off + and on again + +`rados `_ + run Ceph clusters including OSDs and MONs, under various conditions of + stress + +`rbd `_ + run RBD tests using actual Ceph clusters, with and without qemu + +`rgw `_ + run RGW tests using actual Ceph clusters + +`smoke `_ + run test that exercise the Ceph API with an actual Ceph cluster + +`teuthology `_ + verify that teuthology can run integration tests, with and without OpenStack + +`upgrade `_ + for various versions of Ceph, verify that upgrades can happen + without disrupting an ongoing workload + +.. _`ceph-qa-suite repository`: https://github.com/ceph/ceph-qa-suite/ +.. _`ceph-deploy`: ../../man/8/ceph-deploy +.. _`ceph-disk`: ../../man/8/ceph-disk + Architecture ============