teuthology-suite: Drop default for --ceph

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1020 from ceph/wip-waitpid

Tell gevent not to patch os.waitpid()

Reviewed-by: Dan Mick <dmick@redhat.com>

Tell gevent not to patch os.waitpid()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1017 from ceph/wip-manhole

Use manhole to provide a way to debug hung jobs

Reviewed-by: Dan Mick <dmick@redhat.com>

Use manhole to provide a way to debug hung jobs

https://pypi.python.org/pypi/manhole

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1014 from jcsp/wip-18594

pcp: use a timeout when downloading graphite graphs

pcp: use a timeout when downloading graphite graphs

Fixes: http://tracker.ceph.com/issues/18597
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1011 from ceph/wip-kernel-kill-vsplitter

kernel: be more flexible about sha1 matching

kernel: be more flexible about sha1 matching

Some rpm scripts don't allow dashes in the Release field, so let's
accept both -g and _g.  Kill _vsplitter() as Calxeda is no more.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1009 from ceph/wip-archive

Avoid a race condition with job archive creation

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix a conflict with openstack requirements

Signed-off-by: Zack Cerza <zack@redhat.com>

worker: Create job archive directories

... not just run archive directories. This is to resolve a race
condition between the job creating its archive directory and the worker
symlinking its log into that directory.

Signed-off-by: Zack Cerza <zack@redhat.com>

run: Don't fail if the archive dir exists

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1006 from ceph/wip-cd-debug

Remove high level of debug logging which is not required

Remove high level of debug logging which is not required

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1005 from ceph/wip-prune-compress

prune: Compress teuthology.log

Reviewed-by: Dan Mick <dmick@redhat.com>

prune: Compress teuthology.log

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #982 from ceph/wip-rgw-multisite

swift: added "--cluster" to rgw-admin command for multisite support

swift: added "--cluster" to rgw-admin command for multisite support

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #1003 from dmick/master

spawn_sol_log: use sys.executable to find bin/python

spawn_sol_log: use sys.executable to find bin/python

An activated virtualenv has PATH set to find Python with /usr/bin/env.
A binary run from virtualenv/bin doesn't set PATH.  Use sys.executable
to handle both invocation methods.

Fixes: http://tracker.ceph.com/issues/17986
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #995 from ceph/wip-ceph-ansible2

ceph-ansible task changes for tests to work with latest ceph-ansible master branch

cleanup temporary files at the end of the test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

patch from ansible module instead of ceph_ansible

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix client keyring permissions

Fix client keyring permission for workunits

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

unit test updates

unit test updates after subclass changes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

check for ceph health function

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

setup env and execute playbook

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rh specific playbook execution

rh specific playbook execution as per docs

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

add required functions after subclass change

add functions required after subclass change from ceph.ansible to normal
task

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

limit number of osd's based on roles

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use ceph-client role

use ceph-client role for clients

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use Task subclass instead of Ansible

make it a subclass of Task as its more appropriate for this test and is
not dependent on ansible task.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1000 from ceph/wip-sp-tag-magic

ShamanProject._tag_to_sha1(): retry w/ ceph.git

ShamanProject._tag_to_sha1(): retry w/ ceph.git

For upgrade tests that are otherwise using ceph-ci.git, we need to also
look in ceph.git to lookup released tags.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #999 from ceph/wip-ci

config: default to ceph-ci

TestShamanProject: mock _get_package_sha1()

Signed-off-by: Zack Cerza <zack@redhat.com>

test_email_addr() mock util.fetch_repos()

Signed-off-by: Zack Cerza <zack@redhat.com>

config: default to ceph-ci.git

Developers will normally be pushing branches to ceph-ci and testing against
those.  Testings branches in the canonical ceph.git will be less common
(mostly the cron jobs).

Signed-off-by: Sage Weil <sage@redhat.com>

config: use ceph git url for suite

e.g., if the user specifies --ceph-repo, we will use this for the suite,
too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #998 from ceph/wip-suite-repo-default

Reflect the move from ceph-qa-suite.git -> ceph.git

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #986 from ceph/wip-rgw-valgrind

valgrind: suppression for md_config_t string leak

Reviewed-by: Sage Weil <sage@redhat.com>

Default suite_relpath to 'qa'

To reflect the move from ceph-qa-suite.git -> ceph.git

Signed-off-by: Zack Cerza <zack@redhat.com>

Default suite repo to ceph.git

To reflect the move from ceph-qa-suite.git -> ceph.git

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #997 from ceph/wip-alt-ceph-repo

suite: Add --ceph-repo option

run: Use alternate ceph repo if specified

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Add --ceph-repo option

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: xenial msync whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #996 from ceph/wip-worker-reload-config

worker: reload teuthology config before each job

Reviewed-by: Dan Mick <dmick@redhat.com>

worker: reload teuthology config before each job

The new suite_repo setting was persisting between jobs, which is...
undesirable.

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: suppression for md_config_t string leak

Fixes: http://tracker.ceph.com/issues/17924
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #990 from ceph/wip-suite-repo

Allow using an alternate suite repo

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #994 from ceph/wip-shaman-default

Default use_shaman to True

Default use_shaman to True

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind.supp: xenial libstdc++ leak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #993 from ceph/wip-rerun-0match

suite: Correct rerun behavior when no jobs match

suite: Correct rerun behavior when no jobs match

Also remove a stray print statement

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #992 from ceph/wip-kernel-reboot-xenial

kernel: fix reboot into the new kernel on xenial

Merge pull request #987 from ceph/wip-misdirected-ops

ceph.conf: osd debug  misdirected ops = true

Merge pull request #969 from jcsp/wip-generate-caps

misc: remove generate_caps

worker: Use suite_relpath if set

Signed-off-by: Zack Cerza <zack@redhat.com>

run: Use suite_relpath if set

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: add --suite-relpath option

Signed-off-by: Zack Cerza <zack@redhat.com>

worker: Use 'suite_repo' in job config if present

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: add --suite-repo option

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils: Use shallow clones of git repos

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils: Give cloned repos more unique names

Signed-off-by: Zack Cerza <zack@redhat.com>

kernel: fix reboot into the new kernel on xenial

shutdown is a symlink to systemctl on xenial, and on "shutdown -r now"
network gets shut down before ssh has a chance to close gracefully.
There have been multiple complaints about this and some distantly
related fixes have made it into recent systemds [1].

This doesn't seem to happen on centos, probably because of different
systemd versions or sshd configuration, so this band-aid is for ubuntu
only.  "systemctl --no-block reboot" is nicer, but not available on
trusty.

[1] https://github.com/systemd/systemd/issues/2390

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #991 from athanatos/wip-rerun

matrix.py: actually generate sorted descriptons

matrix.py: actually generate sorted descriptons

The existing code compares strings and tuples which, based on what seems
to be happening, is at best counterintuitive and at worst
non-deterministic.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #988 from ceph/wip-shaman-repo

ShamanProject: don't write broken repos

Reviewed-by: Dan Mick <dmick@redhat.com>

misc: avoid a circular import with .packaging

Import DEFAULT_OS_VERSION from .orchestra.opsys, where it is originally
defined.

Signed-off-by: Zack Cerza <zack@redhat.com>

ShamanProject: don't write broken repos

We saw several cases where repo file contents were actually small HTML
files containing "502 Bad Gateway" errors. This will cause ShamanProject
to raise an exception if that happens - and not write broken repo files.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #940 from ceph/wip-rh-changes

rh specific changes

ceph.conf: osd debug  misdirected ops = true

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #985 from ceph/wip-shaman-kernel

kernel task support for shaman

task.kernel: Support fetching kernels from Shaman

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ShamanProject.scm_version

This new property returns .['extra]['version'] from the Shaman query
result.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove rh prefix from function names

remove rh-prefix from funciton names and
fix review comments

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

unit tests for install function

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

uninstall based on packages defined in downstream.yaml

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

function to install debian downstream packages

Install debian downstream packages based on packages
defined in rh/downstream.yaml defined in suites dir

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

function to setup debian downstream repo

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use downstream.yaml to install packages and version check

remove hardcoded checks instead use downstream.yaml
defined in suites dir rh/downstream.yaml
for packages and version checks

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use rh/downstream.yaml to define packages

Remove hardcoded checks, use rh/downstream.yaml
from suite dir to define packages to install

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #984 from ceph/wip-dash-m

lock: -m alias for --machine-type

lock: -m alias for --machine-type

Just like suite.py.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #983 from ceph/wip-local-kernel-regex

kernel: update get_sha1_from_pkg_name() regex

kernel: update get_sha1_from_pkg_name() regex

ceph-build kernel packages will be named -ceph-g${SHA1:0:12}.  As the
first step towards using ceph-build, change the local package regex.
With this, I'm able to download post ceph/ceph-build#540 packages and
install them as a local deb/rpm.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #980 from jcsp/wip-python-rgw

teuthology/nuke: update for python-rgw

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #981 from ceph/wip-unpin-ansible

Revert "Pin ansible to <2.2.0"

teuthology/nuke: update for python-rgw

Signed-off-by: John Spray <john.spray@redhat.com>

Revert "Pin ansible to <2.2.0"

This reverts commit f2576e7a719a129a3c8be603bd9993d2b1ec94f2.

We finally heeded all the deprecation warnings.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #979 from tchaikov/wip-remove-libcephfs

teuthology: update nuke for libcephfs2

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

teuthology: update nuke for libcephfs2

try to remove libcephfs2 as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #974 from ceph/wip-install-subpackage

Make the install task a subpackage

Reviewed-by: Dan Mick <dmick@redhat.com>

ShamanProject: add repo installation and removal

Signed-off-by: Zack Cerza <zack@redhat.com>

Move repo installation to GitbuilderProject

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: bdev debug aio = true

detect aio stalls

Signed-off-by: Sage Weil <sage@redhat.com>

Add NoRemoteError

Signed-off-by: Zack Cerza <zack@redhat.com>