Fix slow worker startup

We can selectively skip fetching and/or bootstrapping if either have
been completed successfully in the last 60s

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix regression in schedule failure emails

Emails weren't getting sent when builds were broken

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #904 from dmick/wip-version

teuthology: log version number

Merge pull request #907 from ceph/wip-16640

task/packages: install libcephfs1-dev

task/packages: install libcephfs1-dev

The way we currently build the cephfs java
bindings depends on the unversioned .so file
which is in the -dev package, not the
main one.

Fixes: http://tracker.ceph.com/issues/16640
Signed-off-by: John Spray <john.spray@redhat.com>

teuthology: log version number

Round 2, this time with testing

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #902 from dmick/wip-schedule-suite-test

Add tests for schedule_suite(); fix some schedule_fail arguments

Merge pull request #906 from dmick/wip-email

Resume paying attention to --email

Resume paying attention to --email

Change in 61350db caused email arg to be lost.  Fix, add test

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #903 from ceph/revert-901-wip-version

Revert "Add teuthology git version query/logging"

Revert "Add teuthology git version query/logging"

Add tests for suite.run.schedule_suite

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #901 from dmick/wip-version

Add teuthology git version query/logging

Add teuthology git version query/logging

Tested exception handling by renaming /usr/bin/git

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Fix arguments to schedule_fail() again

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #890 from ceph/wip-fixlistindex-wusui

Fix resolve_role_list in daemon.py

Merge pull request #897 from ceph/wip-provision-subpackage

Make teuthology.provision a subpackage

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #864 from jcsp/wip-nocolor

task/ansible: disable color output

Merge pull request #898 from ceph/wip-gbp-dmick

Add 'tag' and 'ref' to GitbuilderProject's init, warn if more than one set, and tests

Fix resolve_role_list in daemon.py

resolve_role_list did not handle cases where roles specified contained wild-cards or
were not specified.  ceph.role.a would end up being resolved as ceph.rolea.

Fixes: tracker issue 16053
Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #894 from ceph/wip-newest

Complete --newest feature, add some cleanups

Use 'conf' instead of 'fn' for the processed arg YamlConfig

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.main: use processed args consistently

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.run.collect_jobs(): deepcopy base_config

deep_merge, despite the name, doesn't do fully deep merging;
it recurses through the whole object, but does not deepcopy
component objects at the lowest level.  This meant that
self.base_config could be corrupted by the second deep_merge
(if base_config contained keys that were augmented by
parsed_yaml, which was often the case, for things like overrides).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: move config tmpfile management to Run

base_config changes over the course of the run, and
so cannot be written to a tmpfile until it's completely
finalized.  Move the create/write/rm out of main() and into
schedule_suite().

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject tests: also test for the right log msg

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject: if multiple ref-selectors present, warn

...about which one is actually being used

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject: allow setting tag or ref in config dict

Precedence: ref, tag, branch, sha1.  (this is debatable.)

Also change log.debug() default-to-master to log.warning()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #899 from ceph/wip-ansible-2

Move to ansible 2

Reviewed-by: Dan Mick <dmick@redhat.com>

Use ansible >= 2.0

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #895 from ceph/wip-no-pyc

Stop generating .pyc files

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #896 from ceph/wip-test-reqs

Add test requirements

Reviewed-by: Dan Mick <dmick@redhat.com>

Add test requirements

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove leftover .pyc files

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't write pyc files

The .pyc files that python generates can be problematic if we move
modules around; teuthology processes end up picking up the old .pyc
files instead of finding the new .py files

Signed-off-by: Zack Cerza <zack@redhat.com>

Move openstack-specific code to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst-specific code to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.provision a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #893 from ceph/wip-gbp

GitbuilderProject: respect branch in config

Reviewed-by: Dan Mick <dmick@redhat.com>

GitbuilderProject: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject test: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #892 from ceph/wip-10481-new

Install -release RPMs directly

Reviewed-by: Dan Mick <dmick@redhat.com>

Install -release RPMs directly

This avoids having to dance around with wget and rm -f.
Fixes: 10481
http://tracker.ceph.com/issues/10481

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: make threadlocalptr whitelist more generic

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #888 from dmick/master

suite.py: must pass at least once through the backtrack loop

suite.py: must pass at least once through the backtrack loop

Changing the default of newest to '0' meant that the combined
loop which handles both backtrack and non-backtrack conditions
was never executing, leading to undefined variables.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #887 from dmick/master

suite.py: defaulting newest to 10 means it's always enabled

suite.py: defaulting newest to 10 means it's always enabled

Even if --newest isn't supplied, docopts will create it and
set it to the default.  Change the default to 0.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #886 from dmick/master

suite: --newest fixes: failure to rebuild base_config on sha1 backtrack, spurious log msg

suite/run.py: rebuild base config on sha1 backtrack

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: refactor to reuse config substitution code

In --newest mode we must recreate the base config, because
the ceph sha1 has been substituted several times already.
Factor the substitution code out of create_initial_config so
that it can be called from the backtracking code (which requires
storing kernel_dict and config_input on the instance).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: don't warn about backtrack failure if not --newest

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #885 from dmick/master

suite/run.py: fix order of args to schedule_fail()

suite/run.py: fix order of args to schedule_fail()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #884 from dmick/master

teuthology-suite: Spelling error in web API + minor test improvement

teuthology/task/valgrind: more rocksdb suppressions

Signed-off-by: Sage Weil <sage@redhat.com>

Come to an agreement about how to spell "committish"

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_find_git_parent: Compress the Mock setup a bit

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #883 from dmick/wip-newest

Add teuthology-suite --newest

Add test for util.find_git_parent()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

teuthology-suite: Implement --newest

If --newest is supplied, missing packages for one of the
required distro/versions are not fatal; instead, use a web
service to backtrack to older sha1s in an attempt to find the
newest commonly-built version.  Limit backtracking to
'--newest' commits (default 10).

See http://github.com/ceph/gitserver for the web service source.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

util.py: use passed package_versions dict even if empty

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Factor "collect jobs" and "schedule jobs" from schedule_suite

packaging.GitbuilderProject: allow more spec from config

To use GbP as a build-checking resource, allow more values
to come from config rather than hardcoding or deriving.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #849 from ceph/wip-fix-overrides

Fix ceph-deploy 'branch' override

Merge pull request #882 from ceph/wip-rocksdb-leak

valgrind: add supporession for rocksdb thread local leaks

Reviewed-by: Samuel Just <sjust@redhat.com>

Fix ceph-deploy 'branch' override

Default ceph-deploy override template defines devcommit which makes
'branch' overrides in the qa-suite useless since it only picksup one option
Remove the devcommit overrides from the template and use that as
default in case no other 'branch' override exist.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #880 from ceph/wip-docs

Docs overhaul

Several clarifications and removals

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: add supporession for rocksdb thread local leaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #881 from ceph/wip-downburst-wait

Pass --wait to downburst

Reviewed-by: Dan Mick <dmick@redhat.com>

Pass --wait to downburst

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst section to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix reference to lab_setup

Signed-off-by: Zack Cerza <zack@redhat.com>

Move detailed test config info to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Add intro for testers

Signed-off-by: Zack Cerza <zack@redhat.com>

Add new README.rst specifically for GitHub

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename 'contents' to 'index'

Signed-off-by: Zack Cerza <zack@redhat.com>

Move OpenStack backend section to its own file

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #879 from ceph/wip-16335

Fix regression in --subset

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #878 from ceph/wip-16302

Install python 2.7 on ubuntu downburst VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix regression in --subset

We were processing the --subset arg well after it was necessary :-/
http://tracker.ceph.com/issues/16335
Fixes: 16335
Signed-off-by: Zack Cerza <zack@redhat.com>

Install python 2.7 on ubuntu downburst VMs

http://tracker.ceph.com/issues/16302
Fixes: 16302
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #877 from ceph/wip-suite-fix

get_gitbuilder_hash: use default distro if needed

get_gitbuilder_hash: use default distro if needed

http://tracker.ceph.com/issues/16328
Fixes: 16328
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #872 from ceph/wip-suite-gbp

Suite refactor

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix indenting

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove duplicate deep_copy call and no-op assert

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unnecessary hyphen

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #876 from batrick/bash-syntax-error

nuke.py: remove trailing fi token

nuke.py: remove trailing fi token

This caused a syntax error in one of the teuthology runs:

    2016-06-10T02:30:39.291 INFO:teuthology.nuke:clearing kernel mount from all nodes
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run:Running command with timeout 60
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run.mira069:Running: "grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi"
    2016-06-10T02:30:39.363 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: syntax error near unexpected token `fi'
    2016-06-10T02:30:39.364 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: `grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi'

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Move README.rst to docs/

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove generated API docs on 'make clean'

Signed-off-by: Zack Cerza <zack@redhat.com>

Explain the return value of m_time

Signed-off-by: Zack Cerza <zack@redhat.com>

Add generated openstack API docs

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #873 from ceph/wip-pcp-archive

pcp: Use a smarter check for archive presence

Reviewed-by: Dan Mick <dmick@redhat.com>

pcp: Use a smarter check for archive presence

If ctx.archive was set, but to None, we were failing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.suite a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Refactor parts of suite into new Run class

Including smaller, more purposeful methods. This is for readability,
testability and to facilitate making future changes.

Signed-off-by: Zack Cerza <zack@redhat.com>

Speed up test_wait_*

They weren't patching sleep(). Now they do :)

Signed-off-by: Zack Cerza <zack@redhat.com>