cloud.openstack: Exclude windows-specific sizes

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Do not log the namespace object

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud.openstack: Cache authentication tokens

Constantly causing Keystone to regenerate auth tokens was the cause of
our hitting rate limits during testing. This will let us reuse auth
tokens - including across processes - to avoid hitting those limits.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add cloud.util.AuthToken

This provides a mechanism for caching OpenStack authentication tokens

Signed-off-by: Zack Cerza <zack@redhat.com>

Move FileLock out of repo_utils

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud: Volume creation fails node creation

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud: Retry failed requests in libcloud

It's common to see "429 Rate limit exceeded", at least with OVH. When we
encounter the exception associated with that exception, backoff and
retry for an interval before eventually giving up.

Signed-off-by: Zack Cerza <zack@redhat.com>

updatekeys: Fix argument parsing

Signed-off-by: Zack Cerza <zack@redhat.com>

.gitignore: .idea

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't unlock VMs when destroy fails

Signed-off-by: Zack Cerza <zack@redhat.com>

Update SSH keys when creating VMs

Signed-off-by: Zack Cerza <zack@redhat.com>

Handle VMs missing consoles safely

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.lock a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

More intelligently implement is_vm()

Move to using one decent implementation instead of multiple (some naive)
implementations

Signed-off-by: Zack Cerza <zack@redhat.com>

Locking changes needed for libcloud

Signed-off-by: Zack Cerza <zack@redhat.com>

Add libcloud backend

Initially this supports OpenStack but will grow to support other methods
of cloud-like deployment. Some assuptions are made regarding supporting
infrastructure (FIXME document these)

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1033 from ceph/wip-wait-osd-timeout

misc.wait_until_osds_up(): timeout after 5min

misc.wait_until_osds_up(): timeout after 5min

It doesn't make any sense to wait more than a few minutes for OSDs to
come up. If they take more than five minutes, fail the job.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1030 from tchaikov/wip-suite-sha1-for-workunit

suite: use "suite_hash" as the default sha1 for workunit

Merge pull request #1032 from ceph/wip-nuke-samba

nuke: Remove samba repos

nuke: Remove samba repos

Fixes: http://tracker.ceph.com/issues/19061
Signed-off-by: David Galloway <dgallowa@redhat.com>

run: allow using alternate suite repo

Do the same thing we do for ceph repo to make ceph.git commit
1f82b9b9446d ("qa/tasks/workunit: use the suite repo for cloning
workunit") work for scheduled jobs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

suite: use "suite_hash" as the default sha1 for workunit

as "workunits" reside in ceph/qa/workunits, it's more intuitive to
respect suite-branch option when cloning workunits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1027 from ceph/wip-ceph-ansible-boot

ceph_ansible: update pip and ansible versions

Merge pull request #1013 from ceph/wip-clock

Change to use task 'clock' instead of 'clock.check'

update default ansible version to 2.2.1

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

upgrade pip first before setuptools

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1018 from jan--f/wip-bootstrap-tumbleweed

bootstrap: add support for opensuse tumbleweed.

Merge pull request #1021 from ceph/wip-suite-default

teuthology-suite: Drop default for --ceph

Merge pull request #1004 from SUSE/wip-17981

nuke: Use pkill -KILL to unconditionally wipe out hadoop processes

Merge pull request #1026 from dmick/master

prune: use the shortest time of -p or -r to decide on processing

prune: use the shortest time of -p or -r to decide on processing

invoking -p 7 -r 30 was only removing passed jobs 30 days or older.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1025 from ceph/wip-console-force-into-spy

console: force existing connections into spy mode if !readonly

console: force existing connections into spy mode if !readonly

If someone watching the console didn't think of using "console -s", we
end up power cycling the node in an attempt to get the login prompt.
This is futile -- if the watcher is still there after the node comes
back up, our connection will get dropped to spy mode again.

Use -f to temporarily force existing connections into spy mode when we
attach to save a power cycle.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1024 from ceph/wip-kclient-nuke-nosync

nuke: work around a reboot -n trouble

nuke: work around a reboot -n trouble

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

nuke: improve stale_kernel_mount() check

Commit 7db9e8b76fd5 ("nuke: bring stale kernel client handling back")
resurrected the check that was removed in commit 1d47a121b385 ("Fix
nuke, redo some cleanup functions").  It isn't sufficient though -- for
example, if a workunit already issued a umount, /etc/mtab won't have
a '^/dev/rbd' entry.

debugfs is enabled and mounted on all distros we care about.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1022 from ceph/wip-unbreak-kclient-nuke

nuke: bring stale kernel client handling back

nuke: drop remove_kernel_mounts()

Calling remove_kernel_mounts() after reboot() is pretty useless.  Also,
as explained in the previous commit, there isn't much we can do in the
krbd case, so just drop it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

nuke: bring stale kernel client handling back

Commit 1d47a121b385 ("Fix nuke, redo some cleanup functions") broke
stale kernel client map/mount handling by dropping reboot arguments.
While for kcephfs we can use 'umount -f' to avoid sync (it used to not
work, but is mostly fixed now, I believe), currently there is nothing
we can do for a local filesystem mounted on top of krbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

teuthology-suite: Drop default for --ceph

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1020 from ceph/wip-waitpid

Tell gevent not to patch os.waitpid()

Reviewed-by: Dan Mick <dmick@redhat.com>

Tell gevent not to patch os.waitpid()

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: add support for opensuse tumbleweed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #1017 from ceph/wip-manhole

Use manhole to provide a way to debug hung jobs

Reviewed-by: Dan Mick <dmick@redhat.com>

Use manhole to provide a way to debug hung jobs

https://pypi.python.org/pypi/manhole

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1014 from jcsp/wip-18594

pcp: use a timeout when downloading graphite graphs

pcp: use a timeout when downloading graphite graphs

Fixes: http://tracker.ceph.com/issues/18597
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1011 from ceph/wip-kernel-kill-vsplitter

kernel: be more flexible about sha1 matching

kernel: be more flexible about sha1 matching

Some rpm scripts don't allow dashes in the Release field, so let's
accept both -g and _g.  Kill _vsplitter() as Calxeda is no more.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

clock: Fix clock's use of Remote.run()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Use clock by default (instead of clock.check)

We're seeing clocks desynchronized.  My theory is that this might
be because ntp can take five minutes or more to actually sync the
clocks, and clock.check doesn't do any setting of the clocks, just
reporting.  clock, OTOH, stops ntpd and does an ntpdate, and then
restarts ntpd, which should kickstart it with a much-closer-to-correct
time.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1009 from ceph/wip-archive

Avoid a race condition with job archive creation

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix a conflict with openstack requirements

Signed-off-by: Zack Cerza <zack@redhat.com>

worker: Create job archive directories

... not just run archive directories. This is to resolve a race
condition between the job creating its archive directory and the worker
symlinking its log into that directory.

Signed-off-by: Zack Cerza <zack@redhat.com>

run: Don't fail if the archive dir exists

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1006 from ceph/wip-cd-debug

Remove high level of debug logging which is not required

Remove high level of debug logging which is not required

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1005 from ceph/wip-prune-compress

prune: Compress teuthology.log

Reviewed-by: Dan Mick <dmick@redhat.com>

prune: Compress teuthology.log

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Use pkill -KILL to unconditionally wipe out hadoop processes

In CentOS 7, the command "ps -ef | grep 'java.*hadoop' | grep -v grep | awk
'{print $2}' | xargs kill -9" produces undesirable output when no matching
processes exist.

Note: the "-f" option to pkill mimics the semantics of "ps -ef". For example,
"ps -ef | grep 'java.*hadoop'" will match a process called "sh java343hadoop",
while pkill will match that process only with the -f option:

$ ps -ef | grep "java.*hadoop" | grep -v grep
root     16165  4101  0 11:57 pts/1    00:00:00 sh java343hadoop
$ pkill -KILL "java.*hadoop"
$
$ ps -ef | grep "java.*hadoop" | grep -v grep
root     16165  4101  0 11:57 pts/1    00:00:00 sh java343hadoop
$ pkill -f -KILL "java.*hadoop"
[1]+  Killed                  sh java343hadoop

Fixes: http://tracker.ceph.com/issues/17981
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #982 from ceph/wip-rgw-multisite

swift: added "--cluster" to rgw-admin command for multisite support

swift: added "--cluster" to rgw-admin command for multisite support

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #1003 from dmick/master

spawn_sol_log: use sys.executable to find bin/python

spawn_sol_log: use sys.executable to find bin/python

An activated virtualenv has PATH set to find Python with /usr/bin/env.
A binary run from virtualenv/bin doesn't set PATH.  Use sys.executable
to handle both invocation methods.

Fixes: http://tracker.ceph.com/issues/17986
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #995 from ceph/wip-ceph-ansible2

ceph-ansible task changes for tests to work with latest ceph-ansible master branch

cleanup temporary files at the end of the test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

patch from ansible module instead of ceph_ansible

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix client keyring permissions

Fix client keyring permission for workunits

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

unit test updates

unit test updates after subclass changes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

check for ceph health function

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

setup env and execute playbook

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rh specific playbook execution

rh specific playbook execution as per docs

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

add required functions after subclass change

add functions required after subclass change from ceph.ansible to normal
task

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

limit number of osd's based on roles

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use ceph-client role

use ceph-client role for clients

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use Task subclass instead of Ansible

make it a subclass of Task as its more appropriate for this test and is
not dependent on ansible task.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1000 from ceph/wip-sp-tag-magic

ShamanProject._tag_to_sha1(): retry w/ ceph.git

ShamanProject._tag_to_sha1(): retry w/ ceph.git

For upgrade tests that are otherwise using ceph-ci.git, we need to also
look in ceph.git to lookup released tags.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #999 from ceph/wip-ci

config: default to ceph-ci

TestShamanProject: mock _get_package_sha1()

Signed-off-by: Zack Cerza <zack@redhat.com>

test_email_addr() mock util.fetch_repos()

Signed-off-by: Zack Cerza <zack@redhat.com>

config: default to ceph-ci.git

Developers will normally be pushing branches to ceph-ci and testing against
those.  Testings branches in the canonical ceph.git will be less common
(mostly the cron jobs).

Signed-off-by: Sage Weil <sage@redhat.com>

config: use ceph git url for suite

e.g., if the user specifies --ceph-repo, we will use this for the suite,
too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #998 from ceph/wip-suite-repo-default

Reflect the move from ceph-qa-suite.git -> ceph.git

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #986 from ceph/wip-rgw-valgrind

valgrind: suppression for md_config_t string leak

Reviewed-by: Sage Weil <sage@redhat.com>

Default suite_relpath to 'qa'

To reflect the move from ceph-qa-suite.git -> ceph.git

Signed-off-by: Zack Cerza <zack@redhat.com>

Default suite repo to ceph.git

To reflect the move from ceph-qa-suite.git -> ceph.git

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #997 from ceph/wip-alt-ceph-repo

suite: Add --ceph-repo option

run: Use alternate ceph repo if specified

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Add --ceph-repo option

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: xenial msync whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #996 from ceph/wip-worker-reload-config

worker: reload teuthology config before each job

Reviewed-by: Dan Mick <dmick@redhat.com>

worker: reload teuthology config before each job

The new suite_repo setting was persisting between jobs, which is...
undesirable.

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: suppression for md_config_t string leak

Fixes: http://tracker.ceph.com/issues/17924
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #990 from ceph/wip-suite-repo

Allow using an alternate suite repo

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #994 from ceph/wip-shaman-default

Default use_shaman to True

Default use_shaman to True

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind.supp: xenial libstdc++ leak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #993 from ceph/wip-rerun-0match

suite: Correct rerun behavior when no jobs match