Integration Tests using Teuthology Workflow
===========================================
-Scheduling Test Run
--------------------
+Infrastructure
+--------------
+
+Components:
+
+1. `ceph-ci`_: Clone of the main Ceph repository, used for triggering Jenkins
+ Ceph builds for development.
+
+2. `Ceph Jenkins`_: Responsible for triggering builds, uploading packages
+ to Chacra, and pushing updates about the build to Shaman.
+
+3. `Shaman`_: UI Interface used to check build status. In its backend,
+ it is a REST API to query and store build information.
+
+4. `Chacra`_: Service where packages are uploaded. The binaries uploaded
+ here can be downloaded and used by anyone.
+
+5. `Teuthology CLI`_: Developers can use various Teuthology commands to schedule
+ and manage test runs.
+
+6. Teuthology: This component is responsible for pushing test jobs to
+ the Beanstalk queue and Paddles. It also picks jobs from
+ the queue and runs tests.
+
+7. Beanstalk queue: A priority queue containing all the queued jobs.
+ Developers typically do not need to interact with it.
+
+8. Paddles: A backend service that stores all test run information.
+ Developers typically do not need to interact with it.
+
+9. `Pulpito`_: A UI interface (for information stored in Paddles) that allows
+ developers to see detailed information about their scheduled tests,
+ including status and results.
+
+10. Testnodes: A cluster of various machines that are used for running tests.
+ Developers usually schedule tests to run on `smithi`_ machines, which are
+ dedicated test nodes for Teuthology integration testing.
+
+Each Teuthology test *run* contains multiple test *jobs*. Each job runs in an
+environment isolated from other jobs, on a different collection of test nodes.
+
+To test a change in Ceph, follow these steps:
+
+1. Getting binaries - Build Ceph.
+2. Scheduling Test Run:
+
+ a. About Test Suites.
+ b. Triggering Teuthology Tests.
+ c. Testing QA changes (without re-building binaries).
+ d. Filtering Tests.
-Getting binaries
-****************
+3. Viewing Test Results:
+
+ a. Pulpito Dashboard.
+ b. Teuthology Archives (Reviewing Logs).
+
+4. Killing tests.
+5. Re-running tests.
+
+
+Getting binaries - Build Ceph
+-----------------------------
Ceph binaries must be built for your branch before you can use teuthology to run integration tests on them. Follow these steps to build the Ceph binaries:
.. _the Chacra site: https://shaman.ceph.com/api/search/?status=ready&project=ceph
-Triggering Tests
-****************
+Naming the ceph-ci branch
+*************************
+Prepend your branch with your name before you push it to ceph-ci. For example,
+a branch named ``feature-x`` should be named ``wip-$yourname-feature-x``, where
+``$yourname`` is replaced with your name. Identifying your branch with your
+name makes your branch easily findable on Shaman and Pulpito.
+
+If you are using one of the stable branches (`quincy`, `pacific`, etc.), include
+the name of that stable branch in your ceph-ci branch name.
+For example, the ``feature-x`` PR branch should be named
+``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.*
+
+You can choose to only trigger a CentOS 9.Stream build (excluding other distro like ubuntu)
+by adding "centos9-only" at the end of the ceph-ci branch name. For example,
+``wip-$yourname-feature-centos9-only``. This helps to get quicker builds and save resources
+when you don't require binaries for other distros.
+
+Delete the branch from ceph-ci when you no longer need it. If you are
+logged in to GitHub, all your branches on ceph-ci can be found here:
+https://github.com/ceph/ceph-ci/branches.
+
+
+Scheduling Test Run
+-------------------
+
+About Test Suites
+*****************
+
+Integration tests are organized into “suites”, which are defined in ``qa/suites``
+sub-directory of the Ceph repository. These test suites can be run with the teuthology-suite
+command.
+
+See `Suites Inventory`_ for a list of available suites of integration tests.
+
+More details understanding of how these test suites are defined can be found on `Integration Test Introduction Page`_.
+
+Triggering Teuthology Tests
+***************************
After you have built Ceph binaries for your branch, you can run tests using
teuthology. This procedure explains how to run tests using teuthology.
ssh <username>@teuthology.front.sepia.ceph.com
This requires Sepia lab access. To request access to the Sepia lab, see:
- https://ceph.github.io/sepia/adding_users/
+ https://ceph.github.io/sepia/adding_users/.
+
+#. For initial setup, follow `teuthology installation guide`_ to setup teuthology for
+ your user on teuthology machine. This will enable you to run teuthology commands.
#. Run the ``teuthology-suite`` command:
-s fs \
-p 110 \
--filter "cephfs-shell" \
- -e foo@gmail.com \
+ -e foo@gmail.com
The options in the above command are defined here:
`Pulpito`_ where the test results can be viewed.
+The ``--dry-run`` option allows you to demo-run ``teuthology-suite`` command without
+actually scheduling teuthology tests. This is helpful to check how many jobs and which jobs
+a command will schedule.
Other frequently used/useful options are ``-d`` (or ``--distro``),
-``--distroversion``, ``--filter-out``, ``--timeout``, ``flavor``, ``-rerun``,
-``-l`` (for limiting number of jobs) , ``-N`` (for how many times the job will
+``--distro-version``, ``--filter-out``, ``--timeout``, ``flavor``, ``--rerun``,
+``--limit`` (for limiting number of jobs) , ``-N`` (for how many times the job will
run), and ``--subset`` (used to reduce the number of tests that are triggered). Run
``teuthology-suite --help`` to read descriptions of these and other options.
for the builds to finish, then triggering tests and waiting for
the test results.
-About Suites and Filters
-************************
-
-See `Suites Inventory`_ for a list of available suites of integration tests.
-Each directory under ``qa/suites`` in the Ceph repository is an integration
-test suite, and arguments appropriate to follow ``-s`` can be found there.
+Filtering Tests
+***************
+Test suites includes combinations of many yaml files which can results in massive
+amount of jobs being scheduled for a suite. So filters can help to reduce the amount
+of jobs or schedule particular jobs within a suite.
+
Keywords for filtering tests can be found in
-``qa/suites/<suite-name>/<subsuite-name>/tasks`` and can be used as arguments
+``qa/suites/<suite-name>/<subsuite-name>/tasks`` in Ceph repository and can be used as arguments
for ``--filter``. Each YAML file in that directory can trigger tests; using the
name of the file without its filename extension as an argument to the
``--filter`` triggers those tests.
the ``modules`` attribute is ``tasks.cephfs.test_cephfs_shell``. This means
that it triggers all tests in ``qa/tasks/cephfs/test_cephfs_shell.py``.
+Read more about how to `Filter Tests by their Description`_.
+
Viewing Test Results
---------------------
*******************
After the tests have finished running, the log for the job can be obtained by
-clicking on the job ID at the Pulpito page associated with your tests. It's
+clicking on the job ID at the Pulpito run page associated with your tests. It's
more convenient to download the log and then view it rather than viewing it in
-an internet browser since these logs can easily be up to 1 GB in size. It is
-easier to ssh into the teuthology machine (``teuthology.front.sepia.ceph.com``)
-and access the following path::
+an internet browser since these logs can easily be up to 1 GB in size.
+It is also possible to ssh into a `developer playground machine`_ and access the following path::
- /ceph/teuthology-archive/<test-id>/<job-id>/teuthology.log
+ /teuthology/<run-name>/<job-id>/teuthology.log
For example: for the above test ID, the path is::
- /ceph/teuthology-archive/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
+ /teuthology/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
This method can be used to view the log more quickly than would be possible through a browser.
-In addition to ``teuthology.log``, some other files are included for debugging
-purposes:
+To view ceph logs (cephadm, ceph monitors, ceph-mgr, etc) or system logs,
+remove ``teuthology.log`` from the job's teuthology log url on browser and then navigate
+to ``remote/<machine>/log/``. System logs can be found at ``remote/<machine>/syslog/``.
+Similarly, these logs can be found on developer playground machines at
+``/teuthology/<test-id>/<job-id>/remote/<machine>/``.
+
+Some other files that are included for debugging purposes:
* ``unit_test_summary.yaml``: Provides a summary of all unit test failures.
Generated (optionally) when the ``unit_test_scan`` configuration option is
* ``valgrind.yaml``: Summarizes any Valgrind errors that may occur.
.. note:: To access archives more conveniently, ``/a/`` has been symbolically
- linked to ``/ceph/teuthology-archive/``. For instance, to access the previous
+ linked to ``/teuthology/``. For instance, to access the previous
example, we can use something like::
/a/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
.. prompt:: bash $
- teuthology-kill -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
+ teuthology-kill -p -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi -m smithi -o scheduled_teuthology@teuthology
-Let's call the argument passed to ``-r`` as test ID. It can be found
+The argument passed to ``-r`` is run name. It can be found
easily in the link to the Pulpito page for the tests you triggered. For
example, for the above test ID, the link is - http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/
'waiting'. Default value: 'fail,dead'
======================= ===============================================
-Naming the ceph-ci branch
--------------------------
-Prepend your branch with your name before you push it to ceph-ci. For example,
-a branch named ``feature-x`` should be named ``wip-$yourname-feature-x``, where
-``$yourname`` is replaced with your name. Identifying your branch with your
-name makes your branch easily findable on Shaman and Pulpito.
-
-If you are using one of the stable branches (`quincy`, `pacific`, etc.), include
-the name of that stable branch in your ceph-ci branch name.
-For example, the ``feature-x`` PR branch should be named
-``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.*
-
-Delete the branch from ceph-ci when you no longer need it. If you are
-logged in to GitHub, all your branches on ceph-ci can be found here:
-https://github.com/ceph/ceph-ci/branches.
-
.. _ceph-ci: https://github.com/ceph/ceph-ci
+.. _Ceph Jenkins: https://jenkins.ceph.com/
+.. _Teuthology CLI: https://docs.ceph.com/projects/teuthology/en/latest/commands/list.html
.. _Chacra: https://github.com/ceph/chacra/blob/master/README.rst
.. _Pulpito: http://pulpito.front.sepia.ceph.com/
.. _Running Your First Test: ../../running-tests-locally/#running-your-first-test
.. _Suites Inventory: ../tests-integration-testing-teuthology-intro/#suites-inventory
.. _Testing Priority: ../tests-integration-testing-teuthology-intro/#testing-priority
.. _Triggering Tests: ../tests-integration-testing-teuthology-workflow/#triggering-tests
+.. _Integration Test Introduction Page: ../tests-integration-testing-teuthology-intro/#how-integration-tests-are-defined
.. _tests-sentry-developers-guide: ../tests-sentry-developers-guide/
+.. _smithi: https://wiki.sepia.ceph.com/doku.php?id=hardware:smithi
+.. _teuthology installation guide: https://docs.ceph.com/projects/teuthology/en/latest/INSTALL.html#installation-and-setup
+.. _Filter Tests by their Description: ../tests-integration-testing-teuthology-intro/#filtering-tests-by-their-description
+.. _developer playground machine: https://wiki.sepia.ceph.com/doku.php?id=devplayground
projects / ceph.git / commitdiff