openstack: only destroy target/ceph- VMs on restart

Obsolete resources will be taken care of by the teuthology-nuke
--stale-openstack cron job. We just want to avoid running into obsolete
targets.

The ceph-* targets which are created by the buildpackages task must also
be disposed of.

The timestamps dir is wiped out, except for packages-repository, if
it exists.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #791 from ceph/wip-ceph-ansible

ceph_ansible: provide devices, monitor_interface, public_network

Reviewed-by: Dan Mick <dmick@redhat.com>

Set network-related hostvars if necessary

Specifically, monitor_interface and public_network

Signed-off-by: Zack Cerza <zack@redhat.com>

Add Remote.{ip_address, interface, cidr}

Signed-off-by: Zack Cerza <zack@redhat.com>

Use misc.get_scratch_devices() when necessary

That is, if osd_auto_discovery is disabled.

Signed-off-by: Zack Cerza <zack@redhat.com>

Refactor hosts file generation

This is in preparation to enable us to write host-specific vars to the
file.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #793 from ceph/wip-suite-error

Add a warning when gitbuilder 404s

Reviewed-by: Dan Mick <dmick@redhat.com>

Add a warning when gitbuilder 404s

When determining the ceph hash, we usually ask gitbuilder. When that
fails, the error is misleading but it would take a bit of refactoring to
truly fix it. Let's at least put a useful warning before the confusing error
message.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #792 from ceph/wip-os-version-str

get_distro_version: Always return a string

Reviewed-by: Dan Mick <dmick@redhat.com>

get_distro_version: Always return a string

Signed-off-by: Zack Cerza <zack@redhat.com>

Unpin requests version

setuptools 20.2 won't parse this version requirement. 2.9.1 appears to
work just fine anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #789 from ceph/wip-ceph-ansible

Add ceph_ansible task

Reviewed-by: Dan Mick <mick@redhat.com>

Add ceph_ansible task

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Patch more aggressively in unit tests

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Don't modify plays

We were applying hosts=all to each play when using a generated playbook.
I can't remember why I thought that was a good idea, and it's tripping
up CephAnsible.

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Run ansible-playbook in repo dir

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Place generated playbooks in repo dir

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: write hosts file in _write_hosts_file()

This is to make implementing the CephAnsible task cleaner.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #788 from ceph/wip-worker-debug

worker: When running a job, log the full command

Reviewed-by: Dan Mick <dmick@redhat.com>

worker: When running a job, log the full command

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #785 from ceph/wip-timing-fix

Fix bug when running archiveless jobs

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #784 from ceph/wip-no-debug

task/install: only install debug packages if debuginfo: true

Fix bug when running archiveless jobs

We were trying to tell the timer to write to a nonexistent file

Signed-off-by: Zack Cerza <zack@redhat.com>

task/install: only install debug packages if debuginfo: true

Note that our kludge to handle extras does not include debuginfo packages,
which means we won't get those for ceph-deploy runs for now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #782 from ceph/wip-timer

Task timing

Reviewed-by: Dan Mick <dmick@redhat.com>

Set config.archive_path if it's not set already

This will only affect non-scheduled jobs

Signed-off-by: Zack Cerza <zack@redhat.com>

Collect timing data for task execution

Just before we execute a task's enter or exit method, create a new timer
mark. If the archive is enabled, each timer.mark() call writes (or
re-writes) all the timing data to timing.yaml inside the archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add teuthology.timer

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #773 from dachary/wip-subset-fix

fix a --subset bug that misses facets in some cases

matrix: reimpliment Sum

See the docstring for details on the new implementation.  The
old one didn't really satisfy the minscanlen properties at the
tail of the sequence.

Signed-off-by: Samuel Just <sjust@redhat.com>

matrix: fix Product index offset and minscanlen value

Incrementing the index on each iteration is problematic because you
may not see a particular element for 2*minscanlen indices.
Decrementing it ensures we'll see any particular submat element every
max(minscanlen) + 1 indicies.

Signed-off-by: Samuel Just <sjust@redhat.com>

suite: add randomized tests verifying subset coverage

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

matrix: add __str__

Signed-off-by: Samuel Just <sjust@redhat.com>

suite: ensure that convolutions are cycled out to satisfy mincyclicity

Signed-off-by: Samuel Just <sjust@redhat.com>

suite: fix rounding bug in _build_matrix

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #781 from ceph/wip-tidy

task/internal: list leftover files during tidy stage

Reviewed-by: Dan Mick <dmick@redhat.com>

task/internal: list leftover testdir files

The rmdir should succeed, but if it doesn't, it's hard to tell what was
there that prevented cleanup.  The find will usually print nothing but
when we do have a sloppy task that didn't clean up we'll be able to tell
what it was...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #780 from ceph/revert-745-wip-get_scratch_devices

Revert "misc: ignore some unusable block devices"

Merge pull request #713 from ceph/wip-prune-faster

prune: Process log dirs in order of ctime

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "misc: ignore some unusable block devices"

Merge pull request #777 from ceph/wip-14515

Map locking failures to 'dead' status

Reviewed-by: Dan Mick <dmick@redhat.com>

lock_machines(): Use dead status for lock failures

Signed-off-by: Zack Cerza <zack@redhat.com>

ProvisionOpenStack: raise QuotaExceededError

... if instance creation fails due to a maxed-out quota

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #776 from dmick/wip-14660

14660: additions to -nuke to undo multipath

Merge pull request #778 from ceph/wip-openstack-nuke

Fix -nuke --stale-openstack

Reviewed-by: Dan Mick <dmick@redhat.com>

nuke.py: Fix some spelling/capitalization errors.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

nuke.py: try to undo multipath devices

Use multipath -F, and remove multipath-tools and/or
device-mapper-multipath.

Fixes: #14460
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Update for openstack volume field name changes

Signed-off-by: Zack Cerza <zack@redhat.com>

Add QuotaExceededError

To represent failures to provision OpenStack instances because of a
maxed-out quota

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #775 from ceph/wip-no-ssp

Update bootstrapping and CI hooks

Install package dependencies in TravisCI

And use an Ubuntu Trusty VM

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop --no-use-wheel when invoking pip

Signed-off-by: Zack Cerza <zack@redhat.com>

Unpin setuptools

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop obsolete jenkins/ dir

Signed-off-by: Zack Cerza <zack@redhat.com>

Require mysqlclient

Even if it's just for teuthology-coverage which isn't used (and might be
broken), it's <80KB

Signed-off-by: Zack Cerza <zack@redhat.com>

Require libvirt devel package on Linux distros

Also, drop the requirement for the system libvirt Python bindings. This
effectively drops support for old distros like Ubuntu Precise.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add libffi for MacOS

Signed-off-by: Zack Cerza <zack@redhat.com>

Require libvirt-python

For teuthology.orchestra.remote.VirtualConsole

Signed-off-by: Zack Cerza <zack@redhat.com>

Rebuild virtualenvs that use system site-packages

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop setuptools version restriction

Signed-off-by: Zack Cerza <zack@redhat.com>

Always upgrade pip and setuptools

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #721 from dachary/wip-openstack-git-urls

openstack: update ~/.teuthology with --git-ceph{,qa-suite}-url

Merge pull request #738 from caibo2014/master

amendment misunderstood lock server and results server address

Merge pull request #744 from ceph/wip-subtree-osd

ceph.conf: mon osd reporter subtree level = osd

Merge pull request #745 from dreamhost/wip-get_scratch_devices

misc: ignore some unusable block devices

Merge pull request #735 from dachary/wip-suite-wait

suite: --wait to block until the suite finishes

Merge pull request #760 from dachary/wip-openstack-supported

doc: only OpenStack with OVH is supported

Drop --system-site-packages

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #774 from ceph/wip-no-rhel-vps

Don't pretend that RHEL is supported by downburst

Speed up these tests by ~20x

By patching provision.downburst_executable()

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove RHEL from downburst version list

... and associated unit test

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #772 from dachary/wip-volume-rename

openstack: a failure to rename a volume is not fatal

openstack: a failure to rename a volume is not fatal

It will be cleaned up by nuke --stale-openstack

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #771 from ceph/wip-openstack-json

nuke: Expect json output from openstack commands

Reviewed-by: Dan Mick <dmick@redhat.com>

nuke: Expect json output from openstack commands

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #752 from dachary/wip-openstack-json-dict

openstack: switch the default json output to dictionary

Merge pull request #736 from ceph/wip-se-ignore-few-denials

Ignore few se denials in log parsing

Merge pull request #770 from ceph/wip-openstack-centos

Add CentOS 7.2 for OpenStack

Ignore few se denials in audit log parsing

dmidecode issue https://bugzilla.redhat.com/show_bug.cgi?id=1289274
tracker for chronyd issue http://tracker.ceph.com/issues/14244

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add user-data for CentOS 7.2

Signed-off-by: Zack Cerza <zack@redhat.com>

Correct CentOS 7.0/7.1 images and add 7.2

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #766 from smithfarm/wip-hammer-sh-doc

doc: add descriptive comment at top of hammer.sh script

Merge pull request #769 from ceph/wip-helgrind-suppressions

valgrind.supp: suppress helgrind warnings for external libraries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #768 from ceph/wip-centos-version

Use remote.inventory_info['os_version']

Use remote.inventory_info['os_version']

... instead of remote.os.version.

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind.supp: suppress helgrind warnings for external libraries

Fixes: #14163
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #767 from ceph/wip-14336

Fix regression in flock usage

Fix regression in flock usage

Bug report: http://tracker.ceph.com/issues/14336
Regression caused in: https://github.com/ceph/teuthology/pull/764

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #762 from ktdreyer/wip-readme-rst-syntax

README: fix rst syntax for sub-bullets

Merge pull request #764 from dachary/wip-14238-openstack-stale

openstack: flock timeout on openstack server create

doc: add descriptive comment at top of hammer.sh script

This script was added by 303e863d32c077c17cd8a027172af8204434a66b - this
commit takes the description from that commit and places it in a comment at
the top of the script.

Without this descriptive comment, the only thing we have to go on is the
filename, hammer.sh, which is easily confused with Hammer as in version
0.94 of Ceph.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #765 from ceph/wip-14125

packaging.GitbuilderProject: support the wait_for_package feature

better logging in packaging._get_response

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

task.install: fix how the config option wait_for_package is set

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

packaging.GitbuilderProject will retry now when fetching package versions

If we don't get a 200, retry to account for gitbuilder finishing a
build.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

openstack: flock timeout on openstack server create

The global lock used to run openstack server create must timeout
otherwise it may hang forever. The timeout is set to 8h. Assuming one
server create --wait takes ~1mn to return, that should be more than
enough to absorbe a burst of 200 jobs provisioning instances at the same
time.

The goal here is to ensure the queue does not stay blocked forever, not
to timeout jobs because they cannot get a server in a given delay.

http://tracker.ceph.com/issues/14243 Fixes: #14243

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #763 from dachary/wip-14238-openstack-stale

openstack: nuke must use ID and not name

openstack: nuke must use ID and not name

Otherwise it will fail if there are multiple instances by the same name.

http://tracker.ceph.com/issues/14238 Fixes: #14238

Signed-off-by: Loic Dachary <loic@dachary.org>

README: fix rst syntax for sub-bullets

The install task contains its own list of sub-point bullets.  Prior to
this change, rST did not treat this list as a sub-list of the "install
task" bullet point. Instead, it rendered them at the same level as the
install task and all the other tasks.

Indent the sub-bullets under the install task's description. This will
cause rST to treat them as true sub-bullets under the install task.

Merge pull request #756 from dachary/wip-openstack-centos-7-1

openstack: support CentOS 7.1