Merge pull request #864 from jcsp/wip-nocolor

task/ansible: disable color output

Merge pull request #898 from ceph/wip-gbp-dmick

Add 'tag' and 'ref' to GitbuilderProject's init, warn if more than one set, and tests

Merge pull request #894 from ceph/wip-newest

Complete --newest feature, add some cleanups

Use 'conf' instead of 'fn' for the processed arg YamlConfig

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.main: use processed args consistently

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.run.collect_jobs(): deepcopy base_config

deep_merge, despite the name, doesn't do fully deep merging;
it recurses through the whole object, but does not deepcopy
component objects at the lowest level.  This meant that
self.base_config could be corrupted by the second deep_merge
(if base_config contained keys that were augmented by
parsed_yaml, which was often the case, for things like overrides).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: move config tmpfile management to Run

base_config changes over the course of the run, and
so cannot be written to a tmpfile until it's completely
finalized.  Move the create/write/rm out of main() and into
schedule_suite().

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject tests: also test for the right log msg

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject: if multiple ref-selectors present, warn

...about which one is actually being used

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject: allow setting tag or ref in config dict

Precedence: ref, tag, branch, sha1.  (this is debatable.)

Also change log.debug() default-to-master to log.warning()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #899 from ceph/wip-ansible-2

Move to ansible 2

Reviewed-by: Dan Mick <dmick@redhat.com>

Use ansible >= 2.0

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #895 from ceph/wip-no-pyc

Stop generating .pyc files

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #896 from ceph/wip-test-reqs

Add test requirements

Reviewed-by: Dan Mick <dmick@redhat.com>

Add test requirements

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove leftover .pyc files

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't write pyc files

The .pyc files that python generates can be problematic if we move
modules around; teuthology processes end up picking up the old .pyc
files instead of finding the new .py files

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #893 from ceph/wip-gbp

GitbuilderProject: respect branch in config

Reviewed-by: Dan Mick <dmick@redhat.com>

GitbuilderProject: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject test: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #892 from ceph/wip-10481-new

Install -release RPMs directly

Reviewed-by: Dan Mick <dmick@redhat.com>

Install -release RPMs directly

This avoids having to dance around with wget and rm -f.
Fixes: 10481
http://tracker.ceph.com/issues/10481

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: make threadlocalptr whitelist more generic

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #888 from dmick/master

suite.py: must pass at least once through the backtrack loop

suite.py: must pass at least once through the backtrack loop

Changing the default of newest to '0' meant that the combined
loop which handles both backtrack and non-backtrack conditions
was never executing, leading to undefined variables.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #887 from dmick/master

suite.py: defaulting newest to 10 means it's always enabled

suite.py: defaulting newest to 10 means it's always enabled

Even if --newest isn't supplied, docopts will create it and
set it to the default.  Change the default to 0.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #886 from dmick/master

suite: --newest fixes: failure to rebuild base_config on sha1 backtrack, spurious log msg

suite/run.py: rebuild base config on sha1 backtrack

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: refactor to reuse config substitution code

In --newest mode we must recreate the base config, because
the ceph sha1 has been substituted several times already.
Factor the substitution code out of create_initial_config so
that it can be called from the backtracking code (which requires
storing kernel_dict and config_input on the instance).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: don't warn about backtrack failure if not --newest

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #885 from dmick/master

suite/run.py: fix order of args to schedule_fail()

suite/run.py: fix order of args to schedule_fail()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #884 from dmick/master

teuthology-suite: Spelling error in web API + minor test improvement

teuthology/task/valgrind: more rocksdb suppressions

Signed-off-by: Sage Weil <sage@redhat.com>

Come to an agreement about how to spell "committish"

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_find_git_parent: Compress the Mock setup a bit

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #883 from dmick/wip-newest

Add teuthology-suite --newest

Add test for util.find_git_parent()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

teuthology-suite: Implement --newest

If --newest is supplied, missing packages for one of the
required distro/versions are not fatal; instead, use a web
service to backtrack to older sha1s in an attempt to find the
newest commonly-built version.  Limit backtracking to
'--newest' commits (default 10).

See http://github.com/ceph/gitserver for the web service source.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

util.py: use passed package_versions dict even if empty

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Factor "collect jobs" and "schedule jobs" from schedule_suite

packaging.GitbuilderProject: allow more spec from config

To use GbP as a build-checking resource, allow more values
to come from config rather than hardcoding or deriving.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #849 from ceph/wip-fix-overrides

Fix ceph-deploy 'branch' override

Merge pull request #882 from ceph/wip-rocksdb-leak

valgrind: add supporession for rocksdb thread local leaks

Reviewed-by: Samuel Just <sjust@redhat.com>

Fix ceph-deploy 'branch' override

Default ceph-deploy override template defines devcommit which makes
'branch' overrides in the qa-suite useless since it only picksup one option
Remove the devcommit overrides from the template and use that as
default in case no other 'branch' override exist.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #880 from ceph/wip-docs

Docs overhaul

Several clarifications and removals

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: add supporession for rocksdb thread local leaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #881 from ceph/wip-downburst-wait

Pass --wait to downburst

Reviewed-by: Dan Mick <dmick@redhat.com>

Pass --wait to downburst

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst section to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix reference to lab_setup

Signed-off-by: Zack Cerza <zack@redhat.com>

Move detailed test config info to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Add intro for testers

Signed-off-by: Zack Cerza <zack@redhat.com>

Add new README.rst specifically for GitHub

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename 'contents' to 'index'

Signed-off-by: Zack Cerza <zack@redhat.com>

Move OpenStack backend section to its own file

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #879 from ceph/wip-16335

Fix regression in --subset

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #878 from ceph/wip-16302

Install python 2.7 on ubuntu downburst VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix regression in --subset

We were processing the --subset arg well after it was necessary :-/
http://tracker.ceph.com/issues/16335
Fixes: 16335
Signed-off-by: Zack Cerza <zack@redhat.com>

Install python 2.7 on ubuntu downburst VMs

http://tracker.ceph.com/issues/16302
Fixes: 16302
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #877 from ceph/wip-suite-fix

get_gitbuilder_hash: use default distro if needed

get_gitbuilder_hash: use default distro if needed

http://tracker.ceph.com/issues/16328
Fixes: 16328
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #872 from ceph/wip-suite-gbp

Suite refactor

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix indenting

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove duplicate deep_copy call and no-op assert

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unnecessary hyphen

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #876 from batrick/bash-syntax-error

nuke.py: remove trailing fi token

nuke.py: remove trailing fi token

This caused a syntax error in one of the teuthology runs:

    2016-06-10T02:30:39.291 INFO:teuthology.nuke:clearing kernel mount from all nodes
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run:Running command with timeout 60
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run.mira069:Running: "grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi"
    2016-06-10T02:30:39.363 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: syntax error near unexpected token `fi'
    2016-06-10T02:30:39.364 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: `grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi'

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Move README.rst to docs/

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove generated API docs on 'make clean'

Signed-off-by: Zack Cerza <zack@redhat.com>

Explain the return value of m_time

Signed-off-by: Zack Cerza <zack@redhat.com>

Add generated openstack API docs

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #873 from ceph/wip-pcp-archive

pcp: Use a smarter check for archive presence

Reviewed-by: Dan Mick <dmick@redhat.com>

pcp: Use a smarter check for archive presence

If ctx.archive was set, but to None, we were failing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.suite a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Refactor parts of suite into new Run class

Including smaller, more purposeful methods. This is for readability,
testability and to facilitate making future changes.

Signed-off-by: Zack Cerza <zack@redhat.com>

Speed up test_wait_*

They weren't patching sleep(). Now they do :)

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unused get_gitbuilder_url()

Its functionality is covered by config.baseurl_template anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Use GitbuilderProject for more methods

Signed-off-by: Zack Cerza <zack@redhat.com>

Add OS.__eq__()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Don't hardcode arch

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Use opsys and packaging modules

Starting with get_distro_defaults()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Drop GitbuilderProject._get_codename()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Fix GBP._get_distro() for fedora

Gitbuilders use URLs like:
http://gitbuilder.ceph.com/ceph-rpm-fedora20-x86_64-basic/ref/hammer/sha1
Not:
http://gitbuilder.ceph.com/ceph-rpm-fc20-x86_64-basic/ref/hammer/sha1

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Don't raise if a codename isn't found

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Add more codenames

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Mark three methods as static/class

... methods

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: use OS's new codename discovery

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Automatically determine version or codename

... using a lookup table to map one to the other

Signed-off-by: Zack Cerza <zack@redhat.com>

Move constants to opsys module

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #868 from ceph/wip-4gb-vm

Bump downburst VMs to 4GB

Bump downburst VMs to 4GB

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #867 from ceph/wip-16049

pcp: Don't fail a job if graph downloading fails

Reviewed-by: Dan Mick <dmick@redhat.com>

pcp: Don't fail a job if graph downloading fails

http://tracker.ceph.com/issues/16049
Fixes: 16049
Signed-off-by: Zack Cerza <zack@redhat.com>

Fix abbreviated TestPCPTask.test_end()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #866 from ceph/wip-suite-not-found

Give a better error message for "suite not found"

test_describe_tests: remove 'manual test stubs'

Use a combined strategy: for test classes, use setup
to manage the list of patchers, also starting them
in setup because all the class-based tests use the same
fake fs data.

For the file-scope test functions, use @patch to manage them,
because the filesystem data changes with each test.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_suite: recode to use patch.start()

This lets us do all the setup/teardown in an organized way,
in class instance methods, so the test cases don't have to
deal with the mock arguments; all they do is pass the
fake_fs to start_patchers to get the right fake_fs data for
their particular test.

Signed-off-by: Dan Mick <dan.mick@redhat.com>