+++ /dev/null
-===================================================
-`Teuthology` -- The Ceph integration test framework
-===================================================
-
-``teuthology`` is an automation framework for `Ceph
-<https://github.com/ceph/ceph>`__, written in `Python
-<https://www.python.org/>`__. It is used to run the vast majority of its tests
-and was developed because the unique requirements of testing such a highly
-distributed system with active kernel development meant that no other framework
-existed that could do its job.
-
-The name '`teuthology <http://en.wikipedia.org/wiki/Teuthology>`__' refers to the
-study of cephalopods.
-
-
-Overview
-========
-
-The general mode of operation of ``teuthology`` is to remotely orchestrate
-operations on remote hosts over SSH, as implemented by `Paramiko
-<http://www.lag.net/paramiko/>`__. A typical `job` consists of multiple nested
-`tasks`, each of which perform operations on a remote host over the network.
-
-When testing, it is common to group many `jobs` together to form a `test run`.
-
-
-Provided Utilities
-==================
-* ``teuthology`` - Run individual jobs
-* ``teuthology-coverage`` - Analyze code coverage via lcov
-* ``teuthology-kill`` - Kill running jobs or entire runs
-* ``teuthology-lock`` - Lock, unlock, and update status of machines
-* ``teuthology-ls`` - List job results by examining an archive directory
-* ``teuthology-openstack`` - Use OpenStack backend (wrapper around ``teuthology-suite``)
-* ``teuthology-nuke`` - Attempt to return a machine to a pristine state
-* ``teuthology-queue`` - List, or delete, jobs in the queue
-* ``teuthology-report`` - Submit test results to a web service (we use `paddles <https://github.com/ceph/paddles/>`__)
-* ``teuthology-results`` - Examing a finished run and email results
-* ``teuthology-schedule`` - Schedule a single job
-* ``teuthology-suite`` - Schedule a full run based on a suite (see `suites` in `ceph-qa-suite <https://github.com/ceph/ceph-qa-suite>`__)
-* ``teuthology-updatekeys`` - Update SSH host keys for a mchine
-* ``teuthology-worker`` - Worker daemon to monitor the queue and execute jobs
-
-For a description of the distinct services that utilities interact with see
-:ref:`components`.
-
-Installation
-============
-
-See :ref:`installation_and_setup`.
-
-
-Infrastructure
-==============
-
-The examples in this document are based on the lab machine configuration used
-by the Red Hat Ceph development and quality assurance teams (see
-:ref:`lab_setup`). Other instances of a Ceph Lab being used in a
-development or testing environment may differ from these examples.
-
-
-Test configuration
-==================
-
-An integration test run takes three items of configuration:
-
-- ``targets``: what hosts to run on; this is a dictionary mapping
- hosts to ssh host keys, like:
- "username@hostname.example.com: ssh-rsa long_hostkey_here"
- It is possible to configure your installation so that if the targets line
- and host keys are omitted and teuthology is run with the --lock option,
- then teuthology will grab machines from a pool of available
- test machines.
-- ``roles``: how to use the hosts; this is a list of lists, where each
- entry lists all the roles to be run on a single host. For example, a
- single entry might say ``[mon.1, osd.1]``.
-- ``tasks``: how to set up the cluster and what tests to run on it;
- see below for examples
-
-The format for this configuration is `YAML <http://yaml.org/>`__, a
-structured data format that is still human-readable and editable.
-
-For example, a full config for a test run that sets up a three-machine
-cluster, mounts Ceph via ``ceph-fuse``, and leaves you at an interactive
-Python prompt for manual exploration (and enabling you to SSH in to
-the nodes & use the live cluster ad hoc), might look like this::
-
- roles:
- - [mon.0, mds.0, osd.0]
- - [mon.1, osd.1]
- - [mon.2, client.0]
- targets:
- ubuntu@host07.example.com: ssh-rsa host07_ssh_key
- ubuntu@host08.example.com: ssh-rsa host08_ssh_key
- ubuntu@host09.example.com: ssh-rsa host09_ssh_key
- tasks:
- - install:
- - ceph:
- - ceph-fuse: [client.0]
- - interactive:
-
-The number of entries under ``roles`` and ``targets`` must match.
-
-Note the colon after every task name in the ``tasks`` section. Also note the
-dashes before each task. This is the YAML syntax for an ordered list and
-specifies the order in which tasks are executed.
-
-The ``install`` task needs to precede all other tasks.
-
-The listed targets need resolvable hostnames. If you do not have a DNS server
-running, you can add entries to ``/etc/hosts``. You also need to be able to SSH
-in to the listed targets without passphrases, and the remote user needs to have
-passwordless `sudo` access. Note that the ssh keys at the end of the
-``targets`` entries are the public ssh keys for the hosts. These are
-located in /etc/ssh/ssh_host_rsa_key.pub
-
-If you saved the above file as ``example.yaml``, you could run
-teuthology on it like this::
-
- ./virtualenv/bin/teuthology example.yaml
-
-You could also pass the ``-v`` option for more verbose execution. See
-``teuthology --help`` for more options.
-
-
-Multiple config files
----------------------
-
-You can pass multiple files as arguments to teuthology. Each one
-will be read as a config file, and their contents will be merged. This
-allows you to share definitions of what a "simple 3 node cluster"
-is. The source tree comes with ``roles/3-simple.yaml``, so we could
-skip the ``roles`` section in the above ``example.yaml`` and then
-run::
-
- ./virtualenv/bin/teuthology roles/3-simple.yaml example.yaml
-
-
-Reserving target machines
--------------------------
-
-Teuthology automatically locks nodes for you if you specify the
-``--lock`` option. Without this option, you must specify machines to
-run on in a ``targets.yaml`` file, and lock them using
-teuthology-lock.
-
-Note that the default owner of a machine is of the form: USER@HOST where USER
-is the user who issued the lock command and host is the machine on which the
-lock command was run.
-
-You can override this with the ``--owner`` option when running
-teuthology or teuthology-lock.
-
-With teuthology-lock you can also add a description, so you can
-remember which tests you were running. This can be done when
-locking or unlocking machines, or as a separate action with the
-``--update`` option. To lock 3 machines and set a description, run::
-
- ./virtualenv/bin/teuthology-lock --lock-many 3 --desc 'test foo'
-
-If machines become unusable for some reason, you can mark them down::
-
- ./virtualenv/bin/teuthology-lock --update --status down machine1 machine2
-
-To see the status of all machines, use the ``--list`` option. This can
-be restricted to particular machines as well::
-
- ./virtualenv/bin/teuthology-lock --list machine1 machine2
-
-
-Choosing machines for a job
----------------------------
-
-It is possible to run jobs against machines of one or more ``machine_type``
-values. It is also possible to tell ``teuthology`` to only select those
-machines which match the following criteria specified in the job's YAML:
-
-* ``os_type`` (e.g. 'rhel', 'ubuntu')
-* ``os_version`` (e.g. '7.0', '14.04')
-* ``arch`` (e.g. 'x86_64')
-
-
-Tasks
-=====
-
-A task is a Python module in the ``teuthology.task`` package, with a
-callable named ``task``. It gets the following arguments:
-
-- ``ctx``: a context that is available through the lifetime of the
- test run, and has useful attributes such as ``cluster``, letting the
- task access the remote hosts. Tasks can also store their internal
- state here. (TODO beware of namespace collisions.)
-- ``config``: the data structure after the colon in the config file,
- e.g. for the above ``ceph-fuse`` example, it would be a list like
- ``["client.0"]``.
-
-Tasks can be simple functions, called once in the order they are
-listed in ``tasks``. But sometimes it makes sense for a task to be
-able to clean up after itself: for example, unmounting the filesystem
-after a test run. A task callable that returns a Python `context
-manager
-<http://docs.python.org/library/stdtypes.html#typecontextmanager>`__
-will have the manager added to a stack, and the stack will be unwound
-at the end of the run. This means the cleanup actions are run in
-reverse order, both on success and failure. A nice way of writing
-context managers is the ``contextlib.contextmanager`` decorator; look
-for that string in the existing tasks to see examples, and note where
-they use ``yield``.
-
-Further details on some of the more complex tasks such as install or workunit
-can be obtained via python help. For example::
-
- >>> import teuthology.task.workunit
- >>> help(teuthology.task.workunit)
-
-displays a page of more documentation and more concrete examples.
-
-Some of the more important / commonly used tasks include:
-
-* ``ansible``: Run the ansible task.
-* ``install``: by default, the install task goes to gitbuilder and installs the
- results of the latest build. You can, however, add additional parameters to
- the test configuration to cause it to install any branch, SHA, archive or
- URL. The following are valid parameters.
-
- - ``branch``: specify a branch (firefly, giant...)
-
- - ``flavor``: specify a flavor (next, unstable...). Flavors can be thought of
- as subsets of branches. Sometimes (unstable, for example) they may have a
- predefined meaning.
-
- - ``project``: specify a project (ceph, samba...)
-
- - ``sha1``: install the build with this sha1 value.
-
- - ``tag``: specify a tag/identifying text for this build (v47.2, v48.1...)
-
-* ``ceph``: Bring up Ceph
-
-* ``overrides``: override behavior. Typically, this includes sub-tasks being
- overridden. Overrides technically is not a task (there is no 'def task' in
- an overrides.py file), but from a user's standpoint can be described as
- behaving like one.
- Sub-tasks can nest further information. For example, overrides
- of install tasks are project specific, so the following section of a yaml
- file would cause all ceph installations to default to using the cuttlefish
- branch::
-
- overrides:
- install:
- ceph:
- branch: cuttlefish
-
-* ``workunit``: workunits are a way of grouping tasks and behavior on targets.
-* ``sequential``: group the sub-tasks into a unit where the sub-tasks run
- sequentially as listed.
-* ``parallel``: group the sub-tasks into a unit where the sub-tasks all run in
- parallel.
-
-Sequential and parallel tasks can be nested. Tasks run sequentially unless
-specified otherwise.
-
-The above list is a very incomplete description of the tasks available on
-teuthology. The teuthology/task subdirectory contains the teuthology-specific
-python files that implement tasks.
-
-Extra tasks used by teuthology can be found in ceph-qa-suite/tasks. These
-tasks are not needed for teuthology to run, but do test specific independent
-features. A user who wants to define a test for a new feature can implement
-new tasks in this directory.
-
-Many of these tasks are used to run shell scripts that are defined in the
-ceph/ceph-qa-suite.
-
-If machines were locked as part of the run (with the --lock switch),
-teuthology normally leaves them locked when there is any task failure
-for investigation of the machine state. When developing new teuthology
-tasks, sometimes this behavior is not useful. The ``unlock_on_failure``
-global option can be set to true to make the unlocking happen unconditionally.
-
-Troubleshooting
-===============
-
-Sometimes when a bug triggers, instead of automatic cleanup, you want
-to explore the system as is. Adding a top-level::
-
- interactive-on-error: true
-
-as a config file for teuthology will make that possible. With that
-option, any *task* that fails, will have the ``interactive`` task
-called after it. This means that before any cleanup happens, you get a
-chance to inspect the system -- both through Teuthology and via extra
-SSH connections -- and the cleanup completes only when you choose so.
-Just exit the interactive Python session to continue the cleanup.
-
-Interactive task facilities
-===========================
-
-The ``interactive`` task presents a prompt for you to interact with the
-teuthology configuration. The ``ctx`` variable is available to explore,
-and a ``pprint.PrettyPrinter().pprint`` object is added for convenience as
-``pp``, so you can do things like pp(dict-of-interest) to see a formatted
-view of the dict.
-
-This is also useful to pause the execution of the test between two tasks,
-either to perform ad hoc operations, or to examine the state of the cluster.
-Hit ``control-D`` to continue when done.
-
-You need to nest ``interactive`` underneath ``tasks`` in your config. You
-can have has many ``interactive`` tasks as needed in your task list.
-
-An example::
-
- tasks:
- - ceph:
- - interactive:
-
-Test Sandbox Directory
-======================
-
-Teuthology currently places most test files and mount points in a
-sandbox directory, defaulting to ``/home/$USER/cephtest``. To change
-the location of the sandbox directory, the following option can be
-specified in ``$HOME/.teuthology.yaml``::
-
- test_path: <directory>
-
-OpenStack backend
-=================
-
-The ``teuthology-openstack`` command is a wrapper around
-``teuthology-suite`` that transparently creates the teuthology cluster
-using OpenStack virtual machines.
-
-Prerequisites
--------------
-
-An OpenStack tenant with access to the nova and cinder API. If the
-cinder API is not available, some jobs won't run because they expect
-volumes attached to each instance.
-
-Setup OpenStack at OVH
-----------------------
-
-Each instance has a public IP by default.
-
-* `create an account <https://www.ovh.com/fr/support/new_nic.xml>`_
-* get $HOME/openrc.sh from `the horizon dashboard <https://horizon.cloud.ovh.net/project/access_and_security/?tab=access_security_tabs__api_access_tab>`_
-
-Setup
------
-
-* Get and configure teuthology::
-
- $ git clone http://github.com/ceph/teuthology
- $ cd teuthology ; ./bootstrap install
- $ source virtualenv/bin/activate
-
-Get OpenStack credentials and test it
--------------------------------------
-
-* follow the `OpenStack API Quick Start <http://docs.openstack.org/api/quick-start/content/index.html#cli-intro>`_
-* source $HOME/openrc.sh
-* verify the OpenStack client works::
-
- $ nova list
- +----+------------+--------+------------+-------------+-------------------------+
- | ID | Name | Status | Task State | Power State | Networks |
- +----+------------+--------+------------+-------------+-------------------------+
- +----+------------+--------+------------+-------------+-------------------------+
-* create a passwordless ssh public key with::
-
- $ openstack keypair create myself > myself.pem
- +-------------+-------------------------------------------------+
- | Field | Value |
- +-------------+-------------------------------------------------+
- | fingerprint | e0:a3:ab:5f:01:54:5c:1d:19:40:d9:62:b4:b3:a1:0b |
- | name | myself |
- | user_id | 5cf9fa21b2e9406b9c4108c42aec6262 |
- +-------------+-------------------------------------------------+
- $ chmod 600 myself.pem
-
-Usage
------
-
-* Create a passwordless ssh public key::
-
- $ openstack keypair create myself > myself.pem
- $ chmod 600 myself.pem
-
-* Run the dummy suite. It does nothing useful but shows all works as
- expected. Note that the first time it is run, it can take a long
- time (from a few minutes to half an hour or so) because it downloads
- and uploads a cloud image to the OpenStack provider. ::
-
- $ teuthology-openstack --key-filename myself.pem --key-name myself --suite dummy
- Job scheduled with name ubuntu-2015-07-24_09:03:29-dummy-master---basic-openstack and ID 1
- 2015-07-24 09:03:30,520.520 INFO:teuthology.suite:ceph sha1: dedda6245ce8db8828fdf2d1a2bfe6163f1216a1
- 2015-07-24 09:03:31,620.620 INFO:teuthology.suite:ceph version: v9.0.2-829.gdedda62
- 2015-07-24 09:03:31,620.620 INFO:teuthology.suite:teuthology branch: master
- 2015-07-24 09:03:32,196.196 INFO:teuthology.suite:ceph-qa-suite branch: master
- 2015-07-24 09:03:32,197.197 INFO:teuthology.repo_utils:Fetching from upstream into /home/ubuntu/src/ceph-qa-suite_master
- 2015-07-24 09:03:33,096.096 INFO:teuthology.repo_utils:Resetting repo at /home/ubuntu/src/ceph-qa-suite_master to branch master
- 2015-07-24 09:03:33,157.157 INFO:teuthology.suite:Suite dummy in /home/ubuntu/src/ceph-qa-suite_master/suites/dummy generated 1 jobs (not yet filtered)
- 2015-07-24 09:03:33,158.158 INFO:teuthology.suite:Scheduling dummy/{all/nop.yaml}
- 2015-07-24 09:03:34,045.045 INFO:teuthology.suite:Suite dummy in /home/ubuntu/src/ceph-qa-suite_master/suites/dummy scheduled 1 jobs.
- 2015-07-24 09:03:34,046.046 INFO:teuthology.suite:Suite dummy in /home/ubuntu/src/ceph-qa-suite_master/suites/dummy -- 0 jobs were filtered out.
-
- 2015-07-24 11:03:34,104.104 INFO:teuthology.openstack:
- web interface: http://167.114.242.13:8081/
- ssh access : ssh ubuntu@167.114.242.13 # logs in /usr/share/nginx/html
-
-* Visit the web interface (the URL is displayed at the end of the
- teuthology-openstack output) to monitor the progress of the suite.
-
-* The virtual machine running the suite will persist for forensic
- analysis purposes. To destroy it run::
-
- $ teuthology-openstack --key-filename myself.pem --key-name myself --teardown
-
-* The test results can be uploaded to a publicly accessible location
- with the ``--upload`` flag::
-
- $ teuthology-openstack --key-filename myself.pem --key-name myself \
- --suite dummy --upload
-
-
-Troubleshooting
----------------
-
-Debian Jessie users may face the following error::
-
- NameError: name 'PROTOCOL_SSLv3' is not defined
-
-The `workaround
-<https://github.com/mistio/mist.io/issues/434#issuecomment-86484952>`_
-suggesting to replace ``PROTOCOL_SSLv3`` with ``PROTOCOL_SSLv23`` in
-the ssl.py has been reported to work.
-
-Running the OpenStack backend integration tests
------------------------------------------------
-
-The easiest way to run the integration tests is to first run a dummy suite::
-
- $ teuthology-openstack --key-name myself --suite dummy
- ...
- ssh access : ssh ubuntu@167.114.242.13
-
-This will create a virtual machine suitable for the integration
-test. Login wih the ssh access displayed at the end of the
-``teuthology-openstack`` command and run the following::
-
- $ pkill -f teuthology-worker
- $ cd teuthology ; pip install "tox>=1.9"
- $ tox -v -e openstack-integration
- integration/openstack-integration.py::TestSuite::test_suite_noop PASSED
- ...
- ========= 9 passed in 2545.51 seconds ========
- $ tox -v -e openstack
- integration/test_openstack.py::TestTeuthologyOpenStack::test_create PASSED
- ...
- ========= 1 passed in 204.35 seconds =========
-
-Defining instances flavor and volumes
--------------------------------------
-
-Each target (i.e. a virtual machine or instance in the OpenStack
-parlance) created by the OpenStack backend are exactly the same. By
-default they have at least 8GB RAM, 20GB disk, 1 cpus and no disk
-attached. It is equivalent to having the following in the
-`~/.teuthology.yaml <https://github.com/ceph/teuthology/blob/master/docs/siteconfig.rst>`_ file::
-
- openstack:
- ...
- machine:
- disk: 20 # GB
- ram: 8000 # MB
- cpus: 1
- volumes:
- count: 0
- size: 1 # GB
-
-If a job needs more RAM or disk etc. the following can be included in
-an existing facet (yaml file in the teuthology parlance)::
-
- openstack:
- - machine:
- disk: 100 # GB
- volumes:
- count: 4
- size: 10 # GB
-
-Teuthology interprets this as the minimimum requirements, on top of
-the defaults found in the ``~/.teuthology.yaml`` file and the job will
-be given instances with at least 100GB root disk, 8GB RAM, 1 cpus and
-four 10GB volumes attached. The highest value wins: if the job claims
-to need 4GB RAM and the defaults are 8GB RAM, the targets will all
-have 8GB RAM.
-
-Note the dash before the ``machine`` key: the ``openstack`` element is
-an array with one value. If the dash is missing, it is a dictionary instead.
-It matters because there can be multiple entries per job such as::
-
- openstack:
- - machine:
- disk: 40 # GB
- ram: 8000 # MB
-
- openstack:
- - machine:
- ram: 32000 # MB
-
- openstack:
- - volumes: # attached to each instance
- count: 3
- size: 200 # GB
-
-When a job is composed with these, theuthology aggregates them as::
-
- openstack:
- - machine:
- disk: 40 # GB
- ram: 8000 # MB
- - machine:
- ram: 32000 # MB
- - volumes: # attached to each instance
- count: 3
- size: 200 # GB
-
-i.e. all entries are grouped in a list in the same fashion ``tasks`` are.
-The resource requirement is the maximum of the resources found in each
-element (including the default values). In the example above it is equivalent to::
-
- openstack:
- machine:
- disk: 40 # GB
- ram: 32000 # MB
- volumes: # attached to each instance
- count: 3
- size: 200 # GB
-
-VIRTUAL MACHINE SUPPORT
-=======================
-
-Teuthology also supports virtual machines, which can function like
-physical machines but differ in the following ways:
-
-VPSHOST:
---------
-The following description is based on the Red Hat lab used by the Ceph
-development and quality assurance teams.
-
-The teuthology database of available machines contains a vpshost field.
-For physical machines, this value is null. For virtual machines, this entry
-is the name of the physical machine that that virtual machine resides on.
-
-There are fixed "slots" for virtual machines that appear in the teuthology
-database. These slots have a machine type of vps and can be locked like
-any other machine. The existence of a vpshost field is how teuthology
-knows whether or not a database entry represents a physical or a virtual
-machine.
-
-In order to get the right virtual machine associations, the following needs
-to be set in ~/.config/libvirt/libvirt.conf or for some older versions
-of libvirt (like ubuntu precise) in ~/.libvirt/libvirt.conf::
-
- uri_aliases = [
- 'mira001=qemu+ssh://ubuntu@mira001.front.sepia.ceph.com/system?no_tty=1',
- 'mira003=qemu+ssh://ubuntu@mira003.front.sepia.ceph.com/system?no_tty=1',
- 'mira004=qemu+ssh://ubuntu@mira004.front.sepia.ceph.com/system?no_tty=1',
- 'mira006=qemu+ssh://ubuntu@mira006.front.sepia.ceph.com/system?no_tty=1',
- 'mira007=qemu+ssh://ubuntu@mira007.front.sepia.ceph.com/system?no_tty=1',
- 'mira008=qemu+ssh://ubuntu@mira008.front.sepia.ceph.com/system?no_tty=1',
- 'mira009=qemu+ssh://ubuntu@mira009.front.sepia.ceph.com/system?no_tty=1',
- 'mira010=qemu+ssh://ubuntu@mira010.front.sepia.ceph.com/system?no_tty=1',
- 'mira011=qemu+ssh://ubuntu@mira011.front.sepia.ceph.com/system?no_tty=1',
- 'mira013=qemu+ssh://ubuntu@mira013.front.sepia.ceph.com/system?no_tty=1',
- 'mira014=qemu+ssh://ubuntu@mira014.front.sepia.ceph.com/system?no_tty=1',
- 'mira015=qemu+ssh://ubuntu@mira015.front.sepia.ceph.com/system?no_tty=1',
- 'mira017=qemu+ssh://ubuntu@mira017.front.sepia.ceph.com/system?no_tty=1',
- 'mira018=qemu+ssh://ubuntu@mira018.front.sepia.ceph.com/system?no_tty=1',
- 'mira020=qemu+ssh://ubuntu@mira020.front.sepia.ceph.com/system?no_tty=1',
- 'mira024=qemu+ssh://ubuntu@mira024.front.sepia.ceph.com/system?no_tty=1',
- 'mira029=qemu+ssh://ubuntu@mira029.front.sepia.ceph.com/system?no_tty=1',
- 'mira036=qemu+ssh://ubuntu@mira036.front.sepia.ceph.com/system?no_tty=1',
- 'mira043=qemu+ssh://ubuntu@mira043.front.sepia.ceph.com/system?no_tty=1',
- 'mira044=qemu+ssh://ubuntu@mira044.front.sepia.ceph.com/system?no_tty=1',
- 'mira074=qemu+ssh://ubuntu@mira074.front.sepia.ceph.com/system?no_tty=1',
- 'mira079=qemu+ssh://ubuntu@mira079.front.sepia.ceph.com/system?no_tty=1',
- 'mira081=qemu+ssh://ubuntu@mira081.front.sepia.ceph.com/system?no_tty=1',
- 'mira091=qemu+ssh://ubuntu@mira091.front.sepia.ceph.com/system?no_tty=1',
- 'mira098=qemu+ssh://ubuntu@mira098.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi01=qemu+ssh://ubuntu@vercoi01.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi02=qemu+ssh://ubuntu@vercoi02.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi03=qemu+ssh://ubuntu@vercoi03.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi04=qemu+ssh://ubuntu@vercoi04.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi05=qemu+ssh://ubuntu@vercoi05.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi06=qemu+ssh://ubuntu@vercoi06.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi07=qemu+ssh://ubuntu@vercoi07.front.sepia.ceph.com/system?no_tty=1',
- 'vercoi08=qemu+ssh://ubuntu@vercoi08.front.sepia.ceph.com/system?no_tty=1',
- 'senta01=qemu+ssh://ubuntu@senta01.front.sepia.ceph.com/system?no_tty=1',
- 'senta02=qemu+ssh://ubuntu@senta02.front.sepia.ceph.com/system?no_tty=1',
- 'senta03=qemu+ssh://ubuntu@senta03.front.sepia.ceph.com/system?no_tty=1',
- 'senta04=qemu+ssh://ubuntu@senta04.front.sepia.ceph.com/system?no_tty=1',
- ]
-
-DOWNBURST:
-----------
-
-When a virtual machine is locked, downburst is run on that machine to install a
-new image. This allows the user to set different virtual OSes to be installed
-on the newly created virtual machine. Currently the default virtual machine is
-ubuntu (precise). A different vm installation can be set using the
-``--os-type`` and ``--os-version`` options in ``teuthology.lock``.
-
-When a virtual machine is unlocked, downburst destroys the image on the
-machine.
-
-Temporary yaml files are used to downburst a virtual machine. A typical
-yaml file will look like this::
-
- downburst:
- cpus: 1
- disk-size: 30G
- distro: centos
- networks:
- - {source: front}
- ram: 4G
-
-These values are used by downburst to create the virtual machine.
-
-When locking a file, a downburst meta-data yaml file can be specified by using
-the downburst-conf parameter on the command line.
-
-To find the downburst executable, teuthology first checks the PATH environment
-variable. If not defined, teuthology next checks for
-src/downburst/virtualenv/bin/downburst executables in the user's home
-directory, /home/ubuntu, and /home/teuthology. This can all be overridden if
-the user specifies a downburst field in the user's .teuthology.yaml file.
-
-HOST KEYS:
-----------
-
-Because teuthology reinstalls a new machine, a new hostkey is generated. After
-locking, once a connection is established to the new machine,
-``teuthology-lock`` with the ``--list`` or ``--list-targets`` options will
-display the new keys. When vps machines are locked using the ``--lock-many``
-option, a message is displayed indicating that ``--list-targets`` should be run
-later.
-
-ASSUMPTIONS:
-------------
-
-It is assumed that downburst is on the user's ``$PATH``.
-
-
-Test Suites
-===========
-
-Most of the current teuthology test suite execution scripts automatically
-download their tests from the master branch of the appropriate github
-repository. People who want to run experimental test suites usually modify the
-download method in the ``teuthology/task`` script to use some other branch or
-repository. This should be generalized in later teuthology releases.
-Teuthology QA suites can be found in ``src/ceph-qa-suite``. Make sure that this
-directory exists in your source tree before running the test suites.
-
-Each suite name is determined by the name of the directory in ``ceph-qa-suite``
-that contains that suite. The directory contains subdirectories and yaml files,
-which, when assembled, produce valid tests that can be run. The test suite
-application generates combinations of these files and thus ends up running a
-set of tests based off the data in the directory for the suite.
-
-To run a suite, enter::
-
- teuthology-suite -s <suite> [-c <ceph>] [-k <kernel>] [-e email] [-f flavor] [-t <teuth>] [-m <mtype>]
-
-where:
-
-* ``suite``: the name of the suite (the directory in ceph-qa-suite).
-* ``ceph``: ceph branch to be used.
-* ``kernel``: version of the kernel to be used.
-* ``email``: email address to send the results to.
-* ``flavor``: the kernel flavor to run against
-* ``teuth``: version of teuthology to run
-* ``mtype``: machine type of the run
-* ``templates``: template file used for further modifying the suite (optional)
-
-For example, consider::
-
- teuthology-suite -s rbd -c wip-fix -k cuttlefish -e bob.smith@foo.com -f basic -t cuttlefish -m plana
-
-The above command runs the rbd suite using the wip-fix branch of ceph, the
-cuttlefish kernel, with a 'basic' kernel flavor, and the teuthology
-cuttlefish branch will be used. It will run on plana machines and send an email
-to bob.smith@foo.com when it's completed. For more details on
-``teuthology-suite``, please consult the output of ``teuthology-suite --help``.
-
-In order for a queued task to be run, a teuthworker thread on
-``teuthology.front.sepia.ceph.com`` needs to remove the task from the queue.
-On ``teuthology.front.sepia.ceph.com``, run ``ps aux | grep teuthology-worker``
-to view currently running tasks. If no processes are reading from the test
-version that you are running, additonal teuthworker tasks need to be started.
-To start these tasks:
-
-* copy your build tree to ``/home/teuthworker`` on ``teuthology.front.sepia.ceph.com``.
-* Give it a unique name (in this example, xxx)
-* start up some number of worker threads (as many as machines you are testing with, there are 60 running for the default queue)::
-
- /home/virtualenv/bin/python
- /var/lib/teuthworker/xxx/virtualenv/bin/teuthworker
- /var/lib/teuthworker/archive --tube xxx
- --log-dir /var/lib/teuthworker/archive/worker_logs
-
- Note: The threads on teuthology.front.sepia.ceph.com are started via
- ~/teuthworker/start.sh. You can use that file as a model for your
- own threads, or add to this file if you want your threads to be
- more permanent.
-
-Once the suite completes, an email message is sent to the users specified, and
-a large amount of information is left on ``teuthology.front.sepia.ceph.com`` in
-``/var/lib/teuthworker/archive``.
-
-This is symbolically linked to /a for convenience. A new directory is created
-whose name consists of a concatenation of the date and time that the suite was
-started, the name of the suite, the ceph branch tested, the kernel used, and
-the flavor. For every test run there is a directory whose name is the pid
-number of the pid of that test. Each of these directory contains a copy of the
-``teuthology.log`` for that process. Other information from the suite is
-stored in files in the directory, and task-specific yaml files and other logs
-are saved in the subdirectories.
-
-These logs are also publically available at
-``http://qa-proxy.ceph.com/teuthology/``.
--- /dev/null
+===================================================
+`Teuthology` -- The Ceph integration test framework
+===================================================
+
+``teuthology`` is an automation framework for `Ceph
+<https://github.com/ceph/ceph>`__, written in `Python
+<https://www.python.org/>`__. It is used to run the vast majority of its tests
+and was developed because the unique requirements of testing such a highly
+distributed system with active kernel development meant that no other framework
+existed that could do its job.
+
+The name '`teuthology <http://en.wikipedia.org/wiki/Teuthology>`__' refers to the
+study of cephalopods.
+
+
+Overview
+========
+
+The general mode of operation of ``teuthology`` is to remotely orchestrate
+operations on remote hosts over SSH, as implemented by `Paramiko
+<http://www.lag.net/paramiko/>`__. A typical `job` consists of multiple nested
+`tasks`, each of which perform operations on a remote host over the network.
+
+When testing, it is common to group many `jobs` together to form a `test run`.
+
+
+Provided Utilities
+==================
+* ``teuthology`` - Run individual jobs
+* ``teuthology-coverage`` - Analyze code coverage via lcov
+* ``teuthology-kill`` - Kill running jobs or entire runs
+* ``teuthology-lock`` - Lock, unlock, and update status of machines
+* ``teuthology-ls`` - List job results by examining an archive directory
+* ``teuthology-openstack`` - Use OpenStack backend (wrapper around ``teuthology-suite``)
+* ``teuthology-nuke`` - Attempt to return a machine to a pristine state
+* ``teuthology-queue`` - List, or delete, jobs in the queue
+* ``teuthology-report`` - Submit test results to a web service (we use `paddles <https://github.com/ceph/paddles/>`__)
+* ``teuthology-results`` - Examing a finished run and email results
+* ``teuthology-schedule`` - Schedule a single job
+* ``teuthology-suite`` - Schedule a full run based on a suite (see `suites` in `ceph-qa-suite <https://github.com/ceph/ceph-qa-suite>`__)
+* ``teuthology-updatekeys`` - Update SSH host keys for a mchine
+* ``teuthology-worker`` - Worker daemon to monitor the queue and execute jobs
+
+For a description of the distinct services that utilities interact with see
+:ref:`components`.
+
+Installation
+============
+
+See :ref:`installation_and_setup`.
+
+
+Infrastructure
+==============
+
+The examples in this document are based on the lab machine configuration used
+by the Red Hat Ceph development and quality assurance teams (see
+:ref:`lab_setup`). Other instances of a Ceph Lab being used in a
+development or testing environment may differ from these examples.
+
+
+Test configuration
+==================
+
+An integration test run takes three items of configuration:
+
+- ``targets``: what hosts to run on; this is a dictionary mapping
+ hosts to ssh host keys, like:
+ "username@hostname.example.com: ssh-rsa long_hostkey_here"
+ It is possible to configure your installation so that if the targets line
+ and host keys are omitted and teuthology is run with the --lock option,
+ then teuthology will grab machines from a pool of available
+ test machines.
+- ``roles``: how to use the hosts; this is a list of lists, where each
+ entry lists all the roles to be run on a single host. For example, a
+ single entry might say ``[mon.1, osd.1]``.
+- ``tasks``: how to set up the cluster and what tests to run on it;
+ see below for examples
+
+The format for this configuration is `YAML <http://yaml.org/>`__, a
+structured data format that is still human-readable and editable.
+
+For example, a full config for a test run that sets up a three-machine
+cluster, mounts Ceph via ``ceph-fuse``, and leaves you at an interactive
+Python prompt for manual exploration (and enabling you to SSH in to
+the nodes & use the live cluster ad hoc), might look like this::
+
+ roles:
+ - [mon.0, mds.0, osd.0]
+ - [mon.1, osd.1]
+ - [mon.2, client.0]
+ targets:
+ ubuntu@host07.example.com: ssh-rsa host07_ssh_key
+ ubuntu@host08.example.com: ssh-rsa host08_ssh_key
+ ubuntu@host09.example.com: ssh-rsa host09_ssh_key
+ tasks:
+ - install:
+ - ceph:
+ - ceph-fuse: [client.0]
+ - interactive:
+
+The number of entries under ``roles`` and ``targets`` must match.
+
+Note the colon after every task name in the ``tasks`` section. Also note the
+dashes before each task. This is the YAML syntax for an ordered list and
+specifies the order in which tasks are executed.
+
+The ``install`` task needs to precede all other tasks.
+
+The listed targets need resolvable hostnames. If you do not have a DNS server
+running, you can add entries to ``/etc/hosts``. You also need to be able to SSH
+in to the listed targets without passphrases, and the remote user needs to have
+passwordless `sudo` access. Note that the ssh keys at the end of the
+``targets`` entries are the public ssh keys for the hosts. These are
+located in /etc/ssh/ssh_host_rsa_key.pub
+
+If you saved the above file as ``example.yaml``, you could run
+teuthology on it like this::
+
+ ./virtualenv/bin/teuthology example.yaml
+
+You could also pass the ``-v`` option for more verbose execution. See
+``teuthology --help`` for more options.
+
+
+Multiple config files
+---------------------
+
+You can pass multiple files as arguments to teuthology. Each one
+will be read as a config file, and their contents will be merged. This
+allows you to share definitions of what a "simple 3 node cluster"
+is. The source tree comes with ``roles/3-simple.yaml``, so we could
+skip the ``roles`` section in the above ``example.yaml`` and then
+run::
+
+ ./virtualenv/bin/teuthology roles/3-simple.yaml example.yaml
+
+
+Reserving target machines
+-------------------------
+
+Teuthology automatically locks nodes for you if you specify the
+``--lock`` option. Without this option, you must specify machines to
+run on in a ``targets.yaml`` file, and lock them using
+teuthology-lock.
+
+Note that the default owner of a machine is of the form: USER@HOST where USER
+is the user who issued the lock command and host is the machine on which the
+lock command was run.
+
+You can override this with the ``--owner`` option when running
+teuthology or teuthology-lock.
+
+With teuthology-lock you can also add a description, so you can
+remember which tests you were running. This can be done when
+locking or unlocking machines, or as a separate action with the
+``--update`` option. To lock 3 machines and set a description, run::
+
+ ./virtualenv/bin/teuthology-lock --lock-many 3 --desc 'test foo'
+
+If machines become unusable for some reason, you can mark them down::
+
+ ./virtualenv/bin/teuthology-lock --update --status down machine1 machine2
+
+To see the status of all machines, use the ``--list`` option. This can
+be restricted to particular machines as well::
+
+ ./virtualenv/bin/teuthology-lock --list machine1 machine2
+
+
+Choosing machines for a job
+---------------------------
+
+It is possible to run jobs against machines of one or more ``machine_type``
+values. It is also possible to tell ``teuthology`` to only select those
+machines which match the following criteria specified in the job's YAML:
+
+* ``os_type`` (e.g. 'rhel', 'ubuntu')
+* ``os_version`` (e.g. '7.0', '14.04')
+* ``arch`` (e.g. 'x86_64')
+
+
+Tasks
+=====
+
+A task is a Python module in the ``teuthology.task`` package, with a
+callable named ``task``. It gets the following arguments:
+
+- ``ctx``: a context that is available through the lifetime of the
+ test run, and has useful attributes such as ``cluster``, letting the
+ task access the remote hosts. Tasks can also store their internal
+ state here. (TODO beware of namespace collisions.)
+- ``config``: the data structure after the colon in the config file,
+ e.g. for the above ``ceph-fuse`` example, it would be a list like
+ ``["client.0"]``.
+
+Tasks can be simple functions, called once in the order they are
+listed in ``tasks``. But sometimes it makes sense for a task to be
+able to clean up after itself: for example, unmounting the filesystem
+after a test run. A task callable that returns a Python `context
+manager
+<http://docs.python.org/library/stdtypes.html#typecontextmanager>`__
+will have the manager added to a stack, and the stack will be unwound
+at the end of the run. This means the cleanup actions are run in
+reverse order, both on success and failure. A nice way of writing
+context managers is the ``contextlib.contextmanager`` decorator; look
+for that string in the existing tasks to see examples, and note where
+they use ``yield``.
+
+Further details on some of the more complex tasks such as install or workunit
+can be obtained via python help. For example::
+
+ >>> import teuthology.task.workunit
+ >>> help(teuthology.task.workunit)
+
+displays a page of more documentation and more concrete examples.
+
+Some of the more important / commonly used tasks include:
+
+* ``ansible``: Run the ansible task.
+* ``install``: by default, the install task goes to gitbuilder and installs the
+ results of the latest build. You can, however, add additional parameters to
+ the test configuration to cause it to install any branch, SHA, archive or
+ URL. The following are valid parameters.
+
+ - ``branch``: specify a branch (firefly, giant...)
+
+ - ``flavor``: specify a flavor (next, unstable...). Flavors can be thought of
+ as subsets of branches. Sometimes (unstable, for example) they may have a
+ predefined meaning.
+
+ - ``project``: specify a project (ceph, samba...)
+
+ - ``sha1``: install the build with this sha1 value.
+
+ - ``tag``: specify a tag/identifying text for this build (v47.2, v48.1...)
+
+* ``ceph``: Bring up Ceph
+
+* ``overrides``: override behavior. Typically, this includes sub-tasks being
+ overridden. Overrides technically is not a task (there is no 'def task' in
+ an overrides.py file), but from a user's standpoint can be described as
+ behaving like one.
+ Sub-tasks can nest further information. For example, overrides
+ of install tasks are project specific, so the following section of a yaml
+ file would cause all ceph installations to default to using the cuttlefish
+ branch::
+
+ overrides:
+ install:
+ ceph:
+ branch: cuttlefish
+
+* ``workunit``: workunits are a way of grouping tasks and behavior on targets.
+* ``sequential``: group the sub-tasks into a unit where the sub-tasks run
+ sequentially as listed.
+* ``parallel``: group the sub-tasks into a unit where the sub-tasks all run in
+ parallel.
+
+Sequential and parallel tasks can be nested. Tasks run sequentially unless
+specified otherwise.
+
+The above list is a very incomplete description of the tasks available on
+teuthology. The teuthology/task subdirectory contains the teuthology-specific
+python files that implement tasks.
+
+Extra tasks used by teuthology can be found in ceph-qa-suite/tasks. These
+tasks are not needed for teuthology to run, but do test specific independent
+features. A user who wants to define a test for a new feature can implement
+new tasks in this directory.
+
+Many of these tasks are used to run shell scripts that are defined in the
+ceph/ceph-qa-suite.
+
+If machines were locked as part of the run (with the --lock switch),
+teuthology normally leaves them locked when there is any task failure
+for investigation of the machine state. When developing new teuthology
+tasks, sometimes this behavior is not useful. The ``unlock_on_failure``
+global option can be set to true to make the unlocking happen unconditionally.
+
+Troubleshooting
+===============
+
+Sometimes when a bug triggers, instead of automatic cleanup, you want
+to explore the system as is. Adding a top-level::
+
+ interactive-on-error: true
+
+as a config file for teuthology will make that possible. With that
+option, any *task* that fails, will have the ``interactive`` task
+called after it. This means that before any cleanup happens, you get a
+chance to inspect the system -- both through Teuthology and via extra
+SSH connections -- and the cleanup completes only when you choose so.
+Just exit the interactive Python session to continue the cleanup.
+
+Interactive task facilities
+===========================
+
+The ``interactive`` task presents a prompt for you to interact with the
+teuthology configuration. The ``ctx`` variable is available to explore,
+and a ``pprint.PrettyPrinter().pprint`` object is added for convenience as
+``pp``, so you can do things like pp(dict-of-interest) to see a formatted
+view of the dict.
+
+This is also useful to pause the execution of the test between two tasks,
+either to perform ad hoc operations, or to examine the state of the cluster.
+Hit ``control-D`` to continue when done.
+
+You need to nest ``interactive`` underneath ``tasks`` in your config. You
+can have has many ``interactive`` tasks as needed in your task list.
+
+An example::
+
+ tasks:
+ - ceph:
+ - interactive:
+
+Test Sandbox Directory
+======================
+
+Teuthology currently places most test files and mount points in a
+sandbox directory, defaulting to ``/home/$USER/cephtest``. To change
+the location of the sandbox directory, the following option can be
+specified in ``$HOME/.teuthology.yaml``::
+
+ test_path: <directory>
+
+OpenStack backend
+=================
+
+The ``teuthology-openstack`` command is a wrapper around
+``teuthology-suite`` that transparently creates the teuthology cluster
+using OpenStack virtual machines.
+
+Prerequisites
+-------------
+
+An OpenStack tenant with access to the nova and cinder API. If the
+cinder API is not available, some jobs won't run because they expect
+volumes attached to each instance.
+
+Setup OpenStack at OVH
+----------------------
+
+Each instance has a public IP by default.
+
+* `create an account <https://www.ovh.com/fr/support/new_nic.xml>`_
+* get $HOME/openrc.sh from `the horizon dashboard <https://horizon.cloud.ovh.net/project/access_and_security/?tab=access_security_tabs__api_access_tab>`_
+
+Setup
+-----
+
+* Get and configure teuthology::
+
+ $ git clone http://github.com/ceph/teuthology
+ $ cd teuthology ; ./bootstrap install
+ $ source virtualenv/bin/activate
+
+Get OpenStack credentials and test it
+-------------------------------------
+
+* follow the `OpenStack API Quick Start <http://docs.openstack.org/api/quick-start/content/index.html#cli-intro>`_
+* source $HOME/openrc.sh
+* verify the OpenStack client works::
+
+ $ nova list
+ +----+------------+--------+------------+-------------+-------------------------+
+ | ID | Name | Status | Task State | Power State | Networks |
+ +----+------------+--------+------------+-------------+-------------------------+
+ +----+------------+--------+------------+-------------+-------------------------+
+* create a passwordless ssh public key with::
+
+ $ openstack keypair create myself > myself.pem
+ +-------------+-------------------------------------------------+
+ | Field | Value |
+ +-------------+-------------------------------------------------+
+ | fingerprint | e0:a3:ab:5f:01:54:5c:1d:19:40:d9:62:b4:b3:a1:0b |
+ | name | myself |
+ | user_id | 5cf9fa21b2e9406b9c4108c42aec6262 |
+ +-------------+-------------------------------------------------+
+ $ chmod 600 myself.pem
+
+Usage
+-----
+
+* Create a passwordless ssh public key::
+
+ $ openstack keypair create myself > myself.pem
+ $ chmod 600 myself.pem
+
+* Run the dummy suite. It does nothing useful but shows all works as
+ expected. Note that the first time it is run, it can take a long
+ time (from a few minutes to half an hour or so) because it downloads
+ and uploads a cloud image to the OpenStack provider. ::
+
+ $ teuthology-openstack --key-filename myself.pem --key-name myself --suite dummy
+ Job scheduled with name ubuntu-2015-07-24_09:03:29-dummy-master---basic-openstack and ID 1
+ 2015-07-24 09:03:30,520.520 INFO:teuthology.suite:ceph sha1: dedda6245ce8db8828fdf2d1a2bfe6163f1216a1
+ 2015-07-24 09:03:31,620.620 INFO:teuthology.suite:ceph version: v9.0.2-829.gdedda62
+ 2015-07-24 09:03:31,620.620 INFO:teuthology.suite:teuthology branch: master
+ 2015-07-24 09:03:32,196.196 INFO:teuthology.suite:ceph-qa-suite branch: master
+ 2015-07-24 09:03:32,197.197 INFO:teuthology.repo_utils:Fetching from upstream into /home/ubuntu/src/ceph-qa-suite_master
+ 2015-07-24 09:03:33,096.096 INFO:teuthology.repo_utils:Resetting repo at /home/ubuntu/src/ceph-qa-suite_master to branch master
+ 2015-07-24 09:03:33,157.157 INFO:teuthology.suite:Suite dummy in /home/ubuntu/src/ceph-qa-suite_master/suites/dummy generated 1 jobs (not yet filtered)
+ 2015-07-24 09:03:33,158.158 INFO:teuthology.suite:Scheduling dummy/{all/nop.yaml}
+ 2015-07-24 09:03:34,045.045 INFO:teuthology.suite:Suite dummy in /home/ubuntu/src/ceph-qa-suite_master/suites/dummy scheduled 1 jobs.
+ 2015-07-24 09:03:34,046.046 INFO:teuthology.suite:Suite dummy in /home/ubuntu/src/ceph-qa-suite_master/suites/dummy -- 0 jobs were filtered out.
+
+ 2015-07-24 11:03:34,104.104 INFO:teuthology.openstack:
+ web interface: http://167.114.242.13:8081/
+ ssh access : ssh ubuntu@167.114.242.13 # logs in /usr/share/nginx/html
+
+* Visit the web interface (the URL is displayed at the end of the
+ teuthology-openstack output) to monitor the progress of the suite.
+
+* The virtual machine running the suite will persist for forensic
+ analysis purposes. To destroy it run::
+
+ $ teuthology-openstack --key-filename myself.pem --key-name myself --teardown
+
+* The test results can be uploaded to a publicly accessible location
+ with the ``--upload`` flag::
+
+ $ teuthology-openstack --key-filename myself.pem --key-name myself \
+ --suite dummy --upload
+
+
+Troubleshooting
+---------------
+
+Debian Jessie users may face the following error::
+
+ NameError: name 'PROTOCOL_SSLv3' is not defined
+
+The `workaround
+<https://github.com/mistio/mist.io/issues/434#issuecomment-86484952>`_
+suggesting to replace ``PROTOCOL_SSLv3`` with ``PROTOCOL_SSLv23`` in
+the ssl.py has been reported to work.
+
+Running the OpenStack backend integration tests
+-----------------------------------------------
+
+The easiest way to run the integration tests is to first run a dummy suite::
+
+ $ teuthology-openstack --key-name myself --suite dummy
+ ...
+ ssh access : ssh ubuntu@167.114.242.13
+
+This will create a virtual machine suitable for the integration
+test. Login wih the ssh access displayed at the end of the
+``teuthology-openstack`` command and run the following::
+
+ $ pkill -f teuthology-worker
+ $ cd teuthology ; pip install "tox>=1.9"
+ $ tox -v -e openstack-integration
+ integration/openstack-integration.py::TestSuite::test_suite_noop PASSED
+ ...
+ ========= 9 passed in 2545.51 seconds ========
+ $ tox -v -e openstack
+ integration/test_openstack.py::TestTeuthologyOpenStack::test_create PASSED
+ ...
+ ========= 1 passed in 204.35 seconds =========
+
+Defining instances flavor and volumes
+-------------------------------------
+
+Each target (i.e. a virtual machine or instance in the OpenStack
+parlance) created by the OpenStack backend are exactly the same. By
+default they have at least 8GB RAM, 20GB disk, 1 cpus and no disk
+attached. It is equivalent to having the following in the
+`~/.teuthology.yaml <https://github.com/ceph/teuthology/blob/master/docs/siteconfig.rst>`_ file::
+
+ openstack:
+ ...
+ machine:
+ disk: 20 # GB
+ ram: 8000 # MB
+ cpus: 1
+ volumes:
+ count: 0
+ size: 1 # GB
+
+If a job needs more RAM or disk etc. the following can be included in
+an existing facet (yaml file in the teuthology parlance)::
+
+ openstack:
+ - machine:
+ disk: 100 # GB
+ volumes:
+ count: 4
+ size: 10 # GB
+
+Teuthology interprets this as the minimimum requirements, on top of
+the defaults found in the ``~/.teuthology.yaml`` file and the job will
+be given instances with at least 100GB root disk, 8GB RAM, 1 cpus and
+four 10GB volumes attached. The highest value wins: if the job claims
+to need 4GB RAM and the defaults are 8GB RAM, the targets will all
+have 8GB RAM.
+
+Note the dash before the ``machine`` key: the ``openstack`` element is
+an array with one value. If the dash is missing, it is a dictionary instead.
+It matters because there can be multiple entries per job such as::
+
+ openstack:
+ - machine:
+ disk: 40 # GB
+ ram: 8000 # MB
+
+ openstack:
+ - machine:
+ ram: 32000 # MB
+
+ openstack:
+ - volumes: # attached to each instance
+ count: 3
+ size: 200 # GB
+
+When a job is composed with these, theuthology aggregates them as::
+
+ openstack:
+ - machine:
+ disk: 40 # GB
+ ram: 8000 # MB
+ - machine:
+ ram: 32000 # MB
+ - volumes: # attached to each instance
+ count: 3
+ size: 200 # GB
+
+i.e. all entries are grouped in a list in the same fashion ``tasks`` are.
+The resource requirement is the maximum of the resources found in each
+element (including the default values). In the example above it is equivalent to::
+
+ openstack:
+ machine:
+ disk: 40 # GB
+ ram: 32000 # MB
+ volumes: # attached to each instance
+ count: 3
+ size: 200 # GB
+
+VIRTUAL MACHINE SUPPORT
+=======================
+
+Teuthology also supports virtual machines, which can function like
+physical machines but differ in the following ways:
+
+VPSHOST:
+--------
+The following description is based on the Red Hat lab used by the Ceph
+development and quality assurance teams.
+
+The teuthology database of available machines contains a vpshost field.
+For physical machines, this value is null. For virtual machines, this entry
+is the name of the physical machine that that virtual machine resides on.
+
+There are fixed "slots" for virtual machines that appear in the teuthology
+database. These slots have a machine type of vps and can be locked like
+any other machine. The existence of a vpshost field is how teuthology
+knows whether or not a database entry represents a physical or a virtual
+machine.
+
+In order to get the right virtual machine associations, the following needs
+to be set in ~/.config/libvirt/libvirt.conf or for some older versions
+of libvirt (like ubuntu precise) in ~/.libvirt/libvirt.conf::
+
+ uri_aliases = [
+ 'mira001=qemu+ssh://ubuntu@mira001.front.sepia.ceph.com/system?no_tty=1',
+ 'mira003=qemu+ssh://ubuntu@mira003.front.sepia.ceph.com/system?no_tty=1',
+ 'mira004=qemu+ssh://ubuntu@mira004.front.sepia.ceph.com/system?no_tty=1',
+ 'mira006=qemu+ssh://ubuntu@mira006.front.sepia.ceph.com/system?no_tty=1',
+ 'mira007=qemu+ssh://ubuntu@mira007.front.sepia.ceph.com/system?no_tty=1',
+ 'mira008=qemu+ssh://ubuntu@mira008.front.sepia.ceph.com/system?no_tty=1',
+ 'mira009=qemu+ssh://ubuntu@mira009.front.sepia.ceph.com/system?no_tty=1',
+ 'mira010=qemu+ssh://ubuntu@mira010.front.sepia.ceph.com/system?no_tty=1',
+ 'mira011=qemu+ssh://ubuntu@mira011.front.sepia.ceph.com/system?no_tty=1',
+ 'mira013=qemu+ssh://ubuntu@mira013.front.sepia.ceph.com/system?no_tty=1',
+ 'mira014=qemu+ssh://ubuntu@mira014.front.sepia.ceph.com/system?no_tty=1',
+ 'mira015=qemu+ssh://ubuntu@mira015.front.sepia.ceph.com/system?no_tty=1',
+ 'mira017=qemu+ssh://ubuntu@mira017.front.sepia.ceph.com/system?no_tty=1',
+ 'mira018=qemu+ssh://ubuntu@mira018.front.sepia.ceph.com/system?no_tty=1',
+ 'mira020=qemu+ssh://ubuntu@mira020.front.sepia.ceph.com/system?no_tty=1',
+ 'mira024=qemu+ssh://ubuntu@mira024.front.sepia.ceph.com/system?no_tty=1',
+ 'mira029=qemu+ssh://ubuntu@mira029.front.sepia.ceph.com/system?no_tty=1',
+ 'mira036=qemu+ssh://ubuntu@mira036.front.sepia.ceph.com/system?no_tty=1',
+ 'mira043=qemu+ssh://ubuntu@mira043.front.sepia.ceph.com/system?no_tty=1',
+ 'mira044=qemu+ssh://ubuntu@mira044.front.sepia.ceph.com/system?no_tty=1',
+ 'mira074=qemu+ssh://ubuntu@mira074.front.sepia.ceph.com/system?no_tty=1',
+ 'mira079=qemu+ssh://ubuntu@mira079.front.sepia.ceph.com/system?no_tty=1',
+ 'mira081=qemu+ssh://ubuntu@mira081.front.sepia.ceph.com/system?no_tty=1',
+ 'mira091=qemu+ssh://ubuntu@mira091.front.sepia.ceph.com/system?no_tty=1',
+ 'mira098=qemu+ssh://ubuntu@mira098.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi01=qemu+ssh://ubuntu@vercoi01.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi02=qemu+ssh://ubuntu@vercoi02.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi03=qemu+ssh://ubuntu@vercoi03.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi04=qemu+ssh://ubuntu@vercoi04.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi05=qemu+ssh://ubuntu@vercoi05.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi06=qemu+ssh://ubuntu@vercoi06.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi07=qemu+ssh://ubuntu@vercoi07.front.sepia.ceph.com/system?no_tty=1',
+ 'vercoi08=qemu+ssh://ubuntu@vercoi08.front.sepia.ceph.com/system?no_tty=1',
+ 'senta01=qemu+ssh://ubuntu@senta01.front.sepia.ceph.com/system?no_tty=1',
+ 'senta02=qemu+ssh://ubuntu@senta02.front.sepia.ceph.com/system?no_tty=1',
+ 'senta03=qemu+ssh://ubuntu@senta03.front.sepia.ceph.com/system?no_tty=1',
+ 'senta04=qemu+ssh://ubuntu@senta04.front.sepia.ceph.com/system?no_tty=1',
+ ]
+
+DOWNBURST:
+----------
+
+When a virtual machine is locked, downburst is run on that machine to install a
+new image. This allows the user to set different virtual OSes to be installed
+on the newly created virtual machine. Currently the default virtual machine is
+ubuntu (precise). A different vm installation can be set using the
+``--os-type`` and ``--os-version`` options in ``teuthology.lock``.
+
+When a virtual machine is unlocked, downburst destroys the image on the
+machine.
+
+Temporary yaml files are used to downburst a virtual machine. A typical
+yaml file will look like this::
+
+ downburst:
+ cpus: 1
+ disk-size: 30G
+ distro: centos
+ networks:
+ - {source: front}
+ ram: 4G
+
+These values are used by downburst to create the virtual machine.
+
+When locking a file, a downburst meta-data yaml file can be specified by using
+the downburst-conf parameter on the command line.
+
+To find the downburst executable, teuthology first checks the PATH environment
+variable. If not defined, teuthology next checks for
+src/downburst/virtualenv/bin/downburst executables in the user's home
+directory, /home/ubuntu, and /home/teuthology. This can all be overridden if
+the user specifies a downburst field in the user's .teuthology.yaml file.
+
+HOST KEYS:
+----------
+
+Because teuthology reinstalls a new machine, a new hostkey is generated. After
+locking, once a connection is established to the new machine,
+``teuthology-lock`` with the ``--list`` or ``--list-targets`` options will
+display the new keys. When vps machines are locked using the ``--lock-many``
+option, a message is displayed indicating that ``--list-targets`` should be run
+later.
+
+ASSUMPTIONS:
+------------
+
+It is assumed that downburst is on the user's ``$PATH``.
+
+
+Test Suites
+===========
+
+Most of the current teuthology test suite execution scripts automatically
+download their tests from the master branch of the appropriate github
+repository. People who want to run experimental test suites usually modify the
+download method in the ``teuthology/task`` script to use some other branch or
+repository. This should be generalized in later teuthology releases.
+Teuthology QA suites can be found in ``src/ceph-qa-suite``. Make sure that this
+directory exists in your source tree before running the test suites.
+
+Each suite name is determined by the name of the directory in ``ceph-qa-suite``
+that contains that suite. The directory contains subdirectories and yaml files,
+which, when assembled, produce valid tests that can be run. The test suite
+application generates combinations of these files and thus ends up running a
+set of tests based off the data in the directory for the suite.
+
+To run a suite, enter::
+
+ teuthology-suite -s <suite> [-c <ceph>] [-k <kernel>] [-e email] [-f flavor] [-t <teuth>] [-m <mtype>]
+
+where:
+
+* ``suite``: the name of the suite (the directory in ceph-qa-suite).
+* ``ceph``: ceph branch to be used.
+* ``kernel``: version of the kernel to be used.
+* ``email``: email address to send the results to.
+* ``flavor``: the kernel flavor to run against
+* ``teuth``: version of teuthology to run
+* ``mtype``: machine type of the run
+* ``templates``: template file used for further modifying the suite (optional)
+
+For example, consider::
+
+ teuthology-suite -s rbd -c wip-fix -k cuttlefish -e bob.smith@foo.com -f basic -t cuttlefish -m plana
+
+The above command runs the rbd suite using the wip-fix branch of ceph, the
+cuttlefish kernel, with a 'basic' kernel flavor, and the teuthology
+cuttlefish branch will be used. It will run on plana machines and send an email
+to bob.smith@foo.com when it's completed. For more details on
+``teuthology-suite``, please consult the output of ``teuthology-suite --help``.
+
+In order for a queued task to be run, a teuthworker thread on
+``teuthology.front.sepia.ceph.com`` needs to remove the task from the queue.
+On ``teuthology.front.sepia.ceph.com``, run ``ps aux | grep teuthology-worker``
+to view currently running tasks. If no processes are reading from the test
+version that you are running, additonal teuthworker tasks need to be started.
+To start these tasks:
+
+* copy your build tree to ``/home/teuthworker`` on ``teuthology.front.sepia.ceph.com``.
+* Give it a unique name (in this example, xxx)
+* start up some number of worker threads (as many as machines you are testing with, there are 60 running for the default queue)::
+
+ /home/virtualenv/bin/python
+ /var/lib/teuthworker/xxx/virtualenv/bin/teuthworker
+ /var/lib/teuthworker/archive --tube xxx
+ --log-dir /var/lib/teuthworker/archive/worker_logs
+
+ Note: The threads on teuthology.front.sepia.ceph.com are started via
+ ~/teuthworker/start.sh. You can use that file as a model for your
+ own threads, or add to this file if you want your threads to be
+ more permanent.
+
+Once the suite completes, an email message is sent to the users specified, and
+a large amount of information is left on ``teuthology.front.sepia.ceph.com`` in
+``/var/lib/teuthworker/archive``.
+
+This is symbolically linked to /a for convenience. A new directory is created
+whose name consists of a concatenation of the date and time that the suite was
+started, the name of the suite, the ceph branch tested, the kernel used, and
+the flavor. For every test run there is a directory whose name is the pid
+number of the pid of that test. Each of these directory contains a copy of the
+``teuthology.log`` for that process. Other information from the suite is
+stored in files in the directory, and task-specific yaml files and other logs
+are saved in the subdirectories.
+
+These logs are also publically available at
+``http://qa-proxy.ceph.com/teuthology/``.
.. toctree::
:maxdepth: 2
- index.rst
+ README.rst
siteconfig.rst
INSTALL.rst
LAB_SETUP.rst
+++ /dev/null
-.. include:: ../README.rst
projects / teuthology.git / commitdiff