openstack: nuke must use ID and not name

Otherwise it will fail if there are multiple instances by the same name.

http://tracker.ceph.com/issues/14238 Fixes: #14238

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #756 from dachary/wip-openstack-centos-7-1

openstack: support CentOS 7.1

Merge pull request #761 from ceph/wip-dbrst-mac

Downburst: make MAC setting persistent and remove test user's password

Reviewed-by: Dan Mick <dmick@redhat.com>

Downburst: Remove the test user's password

So that console logins are possible

Signed-off-by: Zack Cerza <zack@redhat.com>

Downburst: Make MAC setting persistent

On CentOS hosts, the MAC is being reset on some reboots. Write it to the
appropriate file.

Signed-off-by: Zack Cerza <zack@redhat.com>

openstack: support CentOS 7.1

It uses the same cloud image as centos-7.0 because it already was 7.1
anyway, only misnamed 7.0 because changing it is inconvenient.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #755 from dachary/wip-openstack-package-data-files

openstack: package data files

Merge pull request #754 from dachary/wip-suite-dry-run-verify-hash

suite: honor suite_verify_ceph_hash on --dry-run

Merge pull request #753 from dachary/wip-openstack-dead-code

openstack: remove dead code

Merge pull request #751 from dachary/wip-openstack-image-private

openstack: keep image private

Merge pull request #750 from dachary/wip-openstack-bugous-ip

openstack: remove bugous static IP regexp

Merge pull request #749 from dachary/wip-openstack-dnsmasq

dnsmasq is not actually restarting

Merge pull request #747 from dachary/wip-openstack-integration-cleanup

tests: openstack-integration is too agressive

Merge pull request #746 from dachary/wip-openstack-delegate-tests

tests: delegate integration tests to OpenStack

Merge pull request #718 from dachary/wip-get-pkg-type

misc: get_pkg_type to infer the package time from the os_type

Merge pull request #759 from ceph/wip-openstack-requests

Block requests==2.9.0

Reviewed-by: Dan Mick <dmick@redhat.com>

Block requests==2.9.0

Because of https://bugs.launchpad.net/nova/+bug/1526413

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #758 from dmick/wip-ntpservers

clock: get the servers to use for ntpdate from ntp.conf

clock: get the servers to use for ntpdate from ntp.conf

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #757 from ceph/fix-ansible-error-reporting

ansible: include the contents of the failure log in the exception

ansible: include the contents of the failure log in the exception

When parsing of the ansible failure log fails this will actually print
the contents of the malformed log in the raised AnsibleFailedError.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #734 from dachary/wip-openstack-ssh

openstack: TeuthologyOpenStack.ssh continuous output

openstack: TeuthologyOpenStack.cloud_init_wait continuous output

Instead of tailing the files every 30 seconds, wait for the
string that proves the teuthology installation completed with a tail -f
and display the result continously.

The number of attempts is reduced from 100 to 30 because there is no
need for repeated calls to tail to check for the desired string.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: display reminders before running the suite

If --wait, it is useful to display the URL/ssh reminders before waiting
for the suite to complete.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: TeuthologyOpenStack.ssh continuous output

Instead of waiting for the completion of the ssh command,
display stdout/stderr continously.

Signed-off-by: Loic Dachary <loic@dachary.org>

misc: sh displays sdterr/stdout continuously

Instead of blocking until the command completes, which makes for a
better user experience when the command takes more than a second to complete.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: package data files

Signed-off-by: Loic Dachary <loic@dachary.org>

suite: honor suite_verify_ceph_hash on --dry-run

When running dry-run, an attempt is made to figure out if the hash of
the job is already present in the gitbuilder. This must not be done
if suite_verify_ceph_hash is false.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: remove dead code

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: keep image private

Ensure a newly created image is private to this tenant by default,
regardless of whatever the openstack defaults are.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

openstack: remove bugous static IP regexp

Instead of duplicating a bugous static IP regexp, use the get_ip()
method.

Signed-off-by: Loic Dachary <loic@dachary.org>

dnsmasq is not actually restarting

The init.d fres, but does nothing.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

tests: openstack-integration is too agressive

It will wipe out all targets in the tenant which can be devastating and
unexpected when multiple teuthology clusters use the same tenant. Filter
the deletion using the ownedby property to limit the destruction to the
targets owned by the cluster in which the integration tests are run.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: delegate integration tests to OpenStack

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #733 from dachary/wip-openstack-instance

openstack: use OpenStackInstance in TeuthologyOpenStack

Merge pull request #732 from dachary/wip-orchestra-debug

orchestra: display connect_args on debug

Merge pull request #683 from dachary/wip-13520-openstack-12.04

openstack: add support for ubuntu 12.04

Merge pull request #731 from dachary/wip-openstack-verify

openstack: use flavor list to verify connectivity

Merge pull request #714 from ceph/wip-teuthology-tree

teuthology-describe-tests: a way to examine a qa suite

describe-tests: add newline before start of docstrings

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: fix formatting for flake8

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #729 from dachary/wip-typos

fix typos

Merge pull request #728 from dachary/wip-openstack-cron-openstack-stale

openstack: cleanup leftovers every 30 minutes

Merge pull request #725 from dachary/wip-gitlab

enable GitLab CI

Merge pull request #712 from dachary/wip-13581-openstack-hint

openstack: enforce hints as an array

Merge pull request #743 from dachary/wip-misc-sh-truncate

misc: truncating shell output to 1024 per line

describe-tests: specify default output format

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: rename input element from 'description' to 'meta'

teuthology-suite already uses a top-level description field that other
tools rely on. This is shorter to type too.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: add docstrings

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: handle ParseErrors from any method

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: add options for json or csv output

For simple consumption, use objects with fields in the json output,
and list the headers separately in case the original order is useful.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: add subsuite columns for combo output

This gives more context when looking at larger suites as a whole,
e.g. rbd or rados.

Sort the rows as well, so tests within subsuites, e.g. basic, thrash,
singleton, etc. are grouped together.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: add options to show the combinations of a suite

Match the behavior of teuthology-suite and generate the full product
of all fragments by default. Include the same filtering and subset
options as well. Just print an asii table for now.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test-describe-tests: remove unnecessary vars

These tests use the class-initialized attributes instead.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

describe-tests: tolerate non-dict files

Some yamls are empty to represent the absence of a setting, and parse
into None.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-describe-tests: change input format to yaml

Rather than using comments, read description metadata from a yaml
'description' section. Make it a list so that teuthology-suite
aggregates the descriptions in order, and allow multiple fields by
making the list contain a dict.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-describe-tests: rename from teuthology-tree

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-tree: show facets for each file by default

Suggested by Loic, this provides more context without
needing to remember or look back at the parent directory.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-tree: remove whitespace from metadata

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-tree: add unit tests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

fake_fs: add fake open() support

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test: move fs fakes to a common file

This will be useful for other tests shortly.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-tree: remove cruft from usage

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-tree: support multiple lines with the same prefix

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-tree: output in a table

This is easier to read since it aligns the metadata.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

scripts: teuthology-tree tool to examine a qa suite

An initial prototype, with basic formatting printing out the directory
tree and comments from the yaml fragments.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: truncating shell output to 1024 per line

Truncating by default to 128 frequently trims useful debug
information. Set the default to 1024: the intent is to avoid huge logs
when a json output is repeatedly returning large (>10k) payloads.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #742 from ceph/wip-os-version

Remote: only submit X.Y for os_version

Copying comment for future documentation:

The opsys.OS class passes the operating system's version according to
the actual host; in some cases e.g. CentOS is reporting things like
'7.1.1503'. In these cases, let's make
Remote.inventory_info['os_version'] only use the '7.1' - since we don't
care about the rest. This will allow us to continue scheduling against
CentOS smoothly.

Reviewed-by: Dan Mick <dmick@redhat.com>

Remote: only submit X.Y for os_version

The opsys.OS class passes the operating system's version according to
the actual host; in some cases e.g. CentOS is reporting things like
'7.1.1503'. In these cases, let's make
Remote.inventory_info['os_version'] only use the '7.1' - since we don't
care about the rest. This will allow us to continue scheduling against
CentOS smoothly.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #727 from dachary/wip-nuke-stale-openstack

nuke: --stale-openstack must cope with disappearing volumes / instances

nuke: --stale-openstack must cope with disappearing volumes / instances

"openstack server/volume show" typically takes a few seconds to
complete due to the provider API slowness/throttling. It is possible that
a volume/instance found with "openstack server list" no longer exists when
running "openstack server show" to get more information about it.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #741 from ceph/wip-apt-mirror

Stop hardcoding apt-mirror's IP

Stop hardcoding apt-mirror's IP

We will point to apt-mirror.sepia.ceph.com, which is
publicly-accessible.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #740 from smithfarm/wip-bootstrap-opensuse

bootstrap: fixes for openSUSE

bootstrap: fixes for openSUSE

When a package is found not to exist, we need to also check if there
is another package installed that provides a capability of the same
name.

Also, the python-devel package must be installed, or the pip phase
fails.

http://tracker.ceph.com/issues/14002 Fixes: #14002

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #737 from ceph/wip-config-port

Use the newer config API

nuke: Use newer config API

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Linter fixes

Signed-off-by: Zack Cerza <zack@redhat.com>

misc: Use newer config API

Signed-off-by: Zack Cerza <zack@redhat.com>

Implement YamlConfig.get()

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop misc.read_config()

Replace with config.set_config_attr(). This is mainly for
backward-compatibility in tasks contained in ceph-qa-suite.

Signed-off-by: Zack Cerza <zack@redhat.com>

openstack: use OpenStackInstance in TeuthologyOpenStack

The setup() function of TeuthologyOpenStack creates an OpenStackInstance
datamember to cache results and avoid running numerous duplicate
commands.

Signed-off-by: Loic Dachary <loic@dachary.org>

orchestra: display connect_args on debug

Instead of log.info because it is noisy.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: use flavor list to verify connectivity

Using the server list to verify if the connection to the OpenStack
cluster works may generate too much output if there are many
instances. Use the last two lines of flavor list instead.

Signed-off-by: Loic Dachary <loic@dachary.org>

fix typos

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: cleanup leftovers every 30 minutes

Resources are billed by the hour, it makes a difference to cleanup
leftovers once every hour. Provide a crontab that runs

  teuthology-nuke --stale-openstack

every 30 minutes.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: enforce hints as an array

All OpenStack hints have been converted to arrays (hammer, infernalis,
master), raise an error if a hint is added old style to avoid mixing
style.

http://tracker.ceph.com/issues/13581 Fixes: #13581

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add support for ubuntu 12.04

Workaround the bug in cloud-init 0.6.3 that prevents the final message
from being
displayed (https://bugs.launchpad.net/cloud-init/+bug/1511485). Also
make the Ubuntu 14.04 user-data file not a symlink since it looks like
each version will need its own.

http://tracker.ceph.com/issues/13520 Fixes: #13520

Signed-off-by: Loic Dachary <ldachary@redhat.com>

enable GitLab CI

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #723 from dachary/wip-nuke-stale-openstack

nuke: cleanup stale OpenStack resources

Merge pull request #709 from ceph/wip-epel-remove

Remove epel packages during downstream testing if present on the node

Merge pull request #710 from dachary/wip-internal-buildpackages

internal: the buildpackages task must come first

Merge pull request #719 from dachary/wip-openstack-parallel-delete

openstack: run 20 server/deletion at a time

Merge pull request #720 from dachary/wip-openstack-instance-size

openstack: teuthology for 100 or more need 32GB

Merge pull request #722 from dachary/wip-openstack-parallel

openstack: run one job at a time by default

nuke: cleanup stale OpenStack resources

Implement the --stale-openstack action to cleanup OpenStack resources
that are known to be stale because:

* instances or volumes have been created a long time ago
* instances or volumes with REMOVE-ME in the name
* nodes that are locked but did not get an instance

Only the instances and volumes known to belong to the teuthology cluster
because they have ownedby=IP in the name are considered, where IP is the
IP address of the machine running the teuthology clusters.

Signed-off-by: Loic Dachary <loic@dachary.org>

lock: locked_since_seconds converts locked_since

The node locked_since property is converted by locked_since_seconds
into the number of seconds since the present time, which is convenient
to compare with configured delays.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: introduce OpenStackInstance

To reduce the number of redundant calls and reduce the code complexity.

Signed-off-by: Loic Dachary <loic@dachary.org>

internal: the buildpackages task must come first

Otherwise the machines will be locked and remain idle while the packages
are building. When there are 20 jobs requiring a package, it means an
average of 40 machines locked idle while the packages are building.

Signed-off-by: Loic Dachary <loic@dachary.org>

misc: get_pkg_type to infer the package time from the os_type

Signed-off-by: Loic Dachary <loic@dachary.org>