Merge pull request #797 from dachary/wip-openstack-override-clone

openstack: allow override of openstack:clone:

Merge pull request #822 from dachary/wip-14997-openstack-stale-keypairs

openstack: protect against stale keypairs

openstack: protect against stale keypairs

http://tracker.ceph.com/issues/14997 Fixes: #14997

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #821 from ceph/wip-yum-priorities

install: Fix non-idempotent check_obsoletes tweak

Reviewed-by: Dan Mick<dmick@redhat.com>

Merge pull request #813 from smithfarm/wip-fix-siteconfig

docs/siteconfig.rst: adapt comments to reality

docs/siteconfig.rst: consistency

Change "count: 0" to "count: 3" for consistency with the comment.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

install: Fix non-idempotent check_obsoletes tweak

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #820 from ceph/wip-nrpe

selinux: Whitelist nrpe

selinux: Whitelist nagios' nrpe

Signed-off-by: Zack Cerza <zack@redhat.com>

selinux: Fix linter complaints

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: do not warn on straw calc version = 0

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #818 from dachary/wip-14799-valgrind

valgrind: add suppression for tcmalloc in libboost_thread-mt.so.1.53.0

valgrind: add suppression for tcmalloc in libboost_thread-mt.so.1.53.0

http://tracker.ceph.com/issues/14799 Fixes: #14799

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #817 from ceph/wip-valgrind-supp

valgrind: ignore boost thread-local warning

Reviewed-by: Dan Mick <dmick@redhat.com>

valgrind: ignore boost thread-local warning

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #814 from ceph/wip-15014

install task: Make package lists configurable

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #816 from ceph/wip-big-vps

lock: Pass canonicalized hostname to create_if_vm

Reviewed-by: Dan Mick <dmick@redhat.com>

lock: Pass canonicalized hostname to create_if_vm

Signed-off-by: Zack Cerza <zack@redhat.com>

install: Remove old "lib deps" cruft

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Use install_task.get_package_list()

(Properly named task.install.get_package_list())

Signed-off-by: Zack Cerza <zack@redhat.com>

install: Add get_package_list()

Move package lists out of the module and into packages.yaml. Enable
overriding package lists via the job config.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #815 from ceph/wip-14855

misc.get_wwn_id_map: not finding wwns is not an error

misc.get_wwn_id_map: not finding wwns is not an error

Fixes: #14855
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #790 from ceph/wip-fixdenials

Add whitelist option and fix for cephtest denials

Add whitelist option and fix for cephtest denials

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #812 from ceph/wip-gevent

Unpin gevent and avoid patching subprocess

Reviewed-by: Dan Mick <dmick@redhat.com>

Unpin gevent and avoid patching subprocess

Recently, gevent updated to 1.1.0, breaking a few things like misc.sh().
PR #810 pinned gevent to 1.0.2, but I wanted to properly investigate the
issue to not be stuck on an old version. Some git bisects later, I
discovered that gevent began patching the subprocess module by default
in:

https://github.com/gevent/gevent/commit/fc73e9afa34d811ede5d62531025d2e3876b8c01

Unpinning gevent and also telling it not to patch subprocess fixes the
issue.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #811 from ceph/wip-prime

mon osd prime pg temp = true

Reviewed-by: Samuel Just <sjust@redhat.com>

ceph.conf: mon osd prime pg temp = true

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #809 from smithfarm/wip_doc_fake_fs

doc: update documentation of make_fake_fstools()

Merge pull request #808 from smithfarm/wip-doc-teuos

README.rst: mention teuthology-openstack in Provided Utilities

Merge pull request #810 from dachary/wip-14990-gevent

pin gevent 1.0.2

pin gevent 1.0.2

http://tracker.ceph.com/issues/14990 Fixes: #14990

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: update documentation of make_fake_fstools()

When the documentation of make_fake_fstools() was written, the function
evidently only returned two values. Now it returns four values. Additional
return values might be added later.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

README.rst: mention teuthology-openstack in Provided Utilities

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #807 from ceph/wip-repo-set-remote

repo_utils: Call git set remote-url

repo_utils: Call git set remote-url

This will cause git repos on-disk to be updated to point at the correct
remote based on config.ceph_git_base_url

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #804 from dachary/wip-openstack-vault

openstack: create default ~/.vault_pass.txt

Merge pull request #806 from dachary/wip-openstack-sitepackage

openstack: do not use sitepackages for openstack-integration tests

Merge pull request #805 from dachary/wip-openstack-test-selection

openstack: test selection for openstack-integration

openstack: do not use sitepackages for openstack-integration tests

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: test selection for openstack-integration

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: create default ~/.vault_pass.txt

It is now required by teuthology.

http://tracker.ceph.com/issues/14914 Fixes: #14914

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: allow override of openstack:clone:

When running from a wip teuthology branch, it is inconvenient to tweak
~/.teuthology.yaml. Add --teuthology-git-url and --teuthology-branch to
take precendence and override the defaults.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #794 from dachary/wip-openstack-wait

openstack: add --wait to block until the suite finishes

Merge pull request #802 from dachary/wip-openstack-neutron

openstack: openstacklient >= 2.1.0 does not depend on neutronclient

Merge pull request #801 from dachary/wip-openstack-centos-7-2

openstack: package CentOS 7.2 data file

Merge pull request #800 from dachary/wip-openstack-cleanup

openstack: only destroy target/ceph- VMs on restart

Merge pull request #799 from dachary/wip-openstack-timeout

openstack: default to 9h job timeout

Merge pull request #796 from dachary/wip-archive-upload

suite/openstack: add --archive-upload-url

Merge pull request #795 from dachary/wip-openstack-suite-exit-code

suite/openstack: implement the exit code

openstack: openstacklient >= 2.1.0 does not depend on neutronclient

Explicitly require it python-neutronclient.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: package CentOS 7.2 data file

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: only destroy target/ceph- VMs on restart

Obsolete resources will be taken care of by the teuthology-nuke
--stale-openstack cron job. We just want to avoid running into obsolete
targets.

The ceph-* targets which are created by the buildpackages task must also
be disposed of.

The timestamps dir is wiped out, except for packages-repository, if
it exists.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: default to 9h job timeout

Because a few job take more than 6h. Ideally this could be overriden on
a case by case basis.

Signed-off-by: Loic Dachary <loic@dachary.org>

suite/openstack: add --archive-upload-url

When uploading job results with --upload, it may be non-trivial to
figure out the public facing URL where the job will be visible. When
used via teuthology-openstack, it defaults to

    http://teuthology-logs.public.ceph.com

so that using --wait --upload --teardown displays the URLs of the
uploaded job results that are accessible even after the cluster is
gone.

Signed-off-by: Loic Dachary <loic@dachary.org>

suite/openstack: implement the exit code

Make it so the exit code of the command reflects the success or failure
of the suite. This is useful when running with --wait.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add --wait to block until the suite finishes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #791 from ceph/wip-ceph-ansible

ceph_ansible: provide devices, monitor_interface, public_network

Reviewed-by: Dan Mick <dmick@redhat.com>

Set network-related hostvars if necessary

Specifically, monitor_interface and public_network

Signed-off-by: Zack Cerza <zack@redhat.com>

Add Remote.{ip_address, interface, cidr}

Signed-off-by: Zack Cerza <zack@redhat.com>

Use misc.get_scratch_devices() when necessary

That is, if osd_auto_discovery is disabled.

Signed-off-by: Zack Cerza <zack@redhat.com>

Refactor hosts file generation

This is in preparation to enable us to write host-specific vars to the
file.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #793 from ceph/wip-suite-error

Add a warning when gitbuilder 404s

Reviewed-by: Dan Mick <dmick@redhat.com>

Add a warning when gitbuilder 404s

When determining the ceph hash, we usually ask gitbuilder. When that
fails, the error is misleading but it would take a bit of refactoring to
truly fix it. Let's at least put a useful warning before the confusing error
message.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #792 from ceph/wip-os-version-str

get_distro_version: Always return a string

Reviewed-by: Dan Mick <dmick@redhat.com>

get_distro_version: Always return a string

Signed-off-by: Zack Cerza <zack@redhat.com>

Unpin requests version

setuptools 20.2 won't parse this version requirement. 2.9.1 appears to
work just fine anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #789 from ceph/wip-ceph-ansible

Add ceph_ansible task

Reviewed-by: Dan Mick <mick@redhat.com>

Add ceph_ansible task

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Patch more aggressively in unit tests

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Don't modify plays

We were applying hosts=all to each play when using a generated playbook.
I can't remember why I thought that was a good idea, and it's tripping
up CephAnsible.

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Run ansible-playbook in repo dir

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Place generated playbooks in repo dir

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: write hosts file in _write_hosts_file()

This is to make implementing the CephAnsible task cleaner.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #788 from ceph/wip-worker-debug

worker: When running a job, log the full command

Reviewed-by: Dan Mick <dmick@redhat.com>

worker: When running a job, log the full command

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #785 from ceph/wip-timing-fix

Fix bug when running archiveless jobs

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #784 from ceph/wip-no-debug

task/install: only install debug packages if debuginfo: true

Fix bug when running archiveless jobs

We were trying to tell the timer to write to a nonexistent file

Signed-off-by: Zack Cerza <zack@redhat.com>

task/install: only install debug packages if debuginfo: true

Note that our kludge to handle extras does not include debuginfo packages,
which means we won't get those for ceph-deploy runs for now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #782 from ceph/wip-timer

Task timing

Reviewed-by: Dan Mick <dmick@redhat.com>

Set config.archive_path if it's not set already

This will only affect non-scheduled jobs

Signed-off-by: Zack Cerza <zack@redhat.com>

Collect timing data for task execution

Just before we execute a task's enter or exit method, create a new timer
mark. If the archive is enabled, each timer.mark() call writes (or
re-writes) all the timing data to timing.yaml inside the archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add teuthology.timer

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #773 from dachary/wip-subset-fix

fix a --subset bug that misses facets in some cases

matrix: reimpliment Sum

See the docstring for details on the new implementation.  The
old one didn't really satisfy the minscanlen properties at the
tail of the sequence.

Signed-off-by: Samuel Just <sjust@redhat.com>

matrix: fix Product index offset and minscanlen value

Incrementing the index on each iteration is problematic because you
may not see a particular element for 2*minscanlen indices.
Decrementing it ensures we'll see any particular submat element every
max(minscanlen) + 1 indicies.

Signed-off-by: Samuel Just <sjust@redhat.com>

suite: add randomized tests verifying subset coverage

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

matrix: add __str__

Signed-off-by: Samuel Just <sjust@redhat.com>

suite: ensure that convolutions are cycled out to satisfy mincyclicity

Signed-off-by: Samuel Just <sjust@redhat.com>

suite: fix rounding bug in _build_matrix

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #781 from ceph/wip-tidy

task/internal: list leftover files during tidy stage

Reviewed-by: Dan Mick <dmick@redhat.com>

task/internal: list leftover testdir files

The rmdir should succeed, but if it doesn't, it's hard to tell what was
there that prevented cleanup.  The find will usually print nothing but
when we do have a sloppy task that didn't clean up we'll be able to tell
what it was...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #780 from ceph/revert-745-wip-get_scratch_devices

Revert "misc: ignore some unusable block devices"

Merge pull request #713 from ceph/wip-prune-faster

prune: Process log dirs in order of ctime

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "misc: ignore some unusable block devices"

Merge pull request #777 from ceph/wip-14515

Map locking failures to 'dead' status

Reviewed-by: Dan Mick <dmick@redhat.com>

lock_machines(): Use dead status for lock failures

Signed-off-by: Zack Cerza <zack@redhat.com>

ProvisionOpenStack: raise QuotaExceededError

... if instance creation fails due to a maxed-out quota

Signed-off-by: Zack Cerza <zack@redhat.com>