Move openstack-specific code to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst-specific code to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.provision a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #893 from ceph/wip-gbp

GitbuilderProject: respect branch in config

Reviewed-by: Dan Mick <dmick@redhat.com>

GitbuilderProject: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject test: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #892 from ceph/wip-10481-new

Install -release RPMs directly

Reviewed-by: Dan Mick <dmick@redhat.com>

Install -release RPMs directly

This avoids having to dance around with wget and rm -f.
Fixes: 10481
http://tracker.ceph.com/issues/10481

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: make threadlocalptr whitelist more generic

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #888 from dmick/master

suite.py: must pass at least once through the backtrack loop

suite.py: must pass at least once through the backtrack loop

Changing the default of newest to '0' meant that the combined
loop which handles both backtrack and non-backtrack conditions
was never executing, leading to undefined variables.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #887 from dmick/master

suite.py: defaulting newest to 10 means it's always enabled

suite.py: defaulting newest to 10 means it's always enabled

Even if --newest isn't supplied, docopts will create it and
set it to the default.  Change the default to 0.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #886 from dmick/master

suite: --newest fixes: failure to rebuild base_config on sha1 backtrack, spurious log msg

suite/run.py: rebuild base config on sha1 backtrack

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: refactor to reuse config substitution code

In --newest mode we must recreate the base config, because
the ceph sha1 has been substituted several times already.
Factor the substitution code out of create_initial_config so
that it can be called from the backtracking code (which requires
storing kernel_dict and config_input on the instance).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: don't warn about backtrack failure if not --newest

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #885 from dmick/master

suite/run.py: fix order of args to schedule_fail()

suite/run.py: fix order of args to schedule_fail()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #884 from dmick/master

teuthology-suite: Spelling error in web API + minor test improvement

teuthology/task/valgrind: more rocksdb suppressions

Signed-off-by: Sage Weil <sage@redhat.com>

Come to an agreement about how to spell "committish"

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_find_git_parent: Compress the Mock setup a bit

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #883 from dmick/wip-newest

Add teuthology-suite --newest

Add test for util.find_git_parent()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

teuthology-suite: Implement --newest

If --newest is supplied, missing packages for one of the
required distro/versions are not fatal; instead, use a web
service to backtrack to older sha1s in an attempt to find the
newest commonly-built version.  Limit backtracking to
'--newest' commits (default 10).

See http://github.com/ceph/gitserver for the web service source.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

util.py: use passed package_versions dict even if empty

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Factor "collect jobs" and "schedule jobs" from schedule_suite

packaging.GitbuilderProject: allow more spec from config

To use GbP as a build-checking resource, allow more values
to come from config rather than hardcoding or deriving.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #849 from ceph/wip-fix-overrides

Fix ceph-deploy 'branch' override

Merge pull request #882 from ceph/wip-rocksdb-leak

valgrind: add supporession for rocksdb thread local leaks

Reviewed-by: Samuel Just <sjust@redhat.com>

Fix ceph-deploy 'branch' override

Default ceph-deploy override template defines devcommit which makes
'branch' overrides in the qa-suite useless since it only picksup one option
Remove the devcommit overrides from the template and use that as
default in case no other 'branch' override exist.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #880 from ceph/wip-docs

Docs overhaul

Several clarifications and removals

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: add supporession for rocksdb thread local leaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #881 from ceph/wip-downburst-wait

Pass --wait to downburst

Reviewed-by: Dan Mick <dmick@redhat.com>

Pass --wait to downburst

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst section to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix reference to lab_setup

Signed-off-by: Zack Cerza <zack@redhat.com>

Move detailed test config info to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Add intro for testers

Signed-off-by: Zack Cerza <zack@redhat.com>

Add new README.rst specifically for GitHub

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename 'contents' to 'index'

Signed-off-by: Zack Cerza <zack@redhat.com>

Move OpenStack backend section to its own file

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #879 from ceph/wip-16335

Fix regression in --subset

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #878 from ceph/wip-16302

Install python 2.7 on ubuntu downburst VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix regression in --subset

We were processing the --subset arg well after it was necessary :-/
http://tracker.ceph.com/issues/16335
Fixes: 16335
Signed-off-by: Zack Cerza <zack@redhat.com>

Install python 2.7 on ubuntu downburst VMs

http://tracker.ceph.com/issues/16302
Fixes: 16302
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #877 from ceph/wip-suite-fix

get_gitbuilder_hash: use default distro if needed

get_gitbuilder_hash: use default distro if needed

http://tracker.ceph.com/issues/16328
Fixes: 16328
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #872 from ceph/wip-suite-gbp

Suite refactor

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix indenting

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove duplicate deep_copy call and no-op assert

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unnecessary hyphen

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #876 from batrick/bash-syntax-error

nuke.py: remove trailing fi token

nuke.py: remove trailing fi token

This caused a syntax error in one of the teuthology runs:

    2016-06-10T02:30:39.291 INFO:teuthology.nuke:clearing kernel mount from all nodes
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run:Running command with timeout 60
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run.mira069:Running: "grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi"
    2016-06-10T02:30:39.363 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: syntax error near unexpected token `fi'
    2016-06-10T02:30:39.364 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: `grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi'

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Move README.rst to docs/

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove generated API docs on 'make clean'

Signed-off-by: Zack Cerza <zack@redhat.com>

Explain the return value of m_time

Signed-off-by: Zack Cerza <zack@redhat.com>

Add generated openstack API docs

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #873 from ceph/wip-pcp-archive

pcp: Use a smarter check for archive presence

Reviewed-by: Dan Mick <dmick@redhat.com>

pcp: Use a smarter check for archive presence

If ctx.archive was set, but to None, we were failing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.suite a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Refactor parts of suite into new Run class

Including smaller, more purposeful methods. This is for readability,
testability and to facilitate making future changes.

Signed-off-by: Zack Cerza <zack@redhat.com>

Speed up test_wait_*

They weren't patching sleep(). Now they do :)

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unused get_gitbuilder_url()

Its functionality is covered by config.baseurl_template anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Use GitbuilderProject for more methods

Signed-off-by: Zack Cerza <zack@redhat.com>

Add OS.__eq__()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Don't hardcode arch

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Use opsys and packaging modules

Starting with get_distro_defaults()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Drop GitbuilderProject._get_codename()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Fix GBP._get_distro() for fedora

Gitbuilders use URLs like:
http://gitbuilder.ceph.com/ceph-rpm-fedora20-x86_64-basic/ref/hammer/sha1
Not:
http://gitbuilder.ceph.com/ceph-rpm-fc20-x86_64-basic/ref/hammer/sha1

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Don't raise if a codename isn't found

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Add more codenames

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Mark three methods as static/class

... methods

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: use OS's new codename discovery

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Automatically determine version or codename

... using a lookup table to map one to the other

Signed-off-by: Zack Cerza <zack@redhat.com>

Move constants to opsys module

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #868 from ceph/wip-4gb-vm

Bump downburst VMs to 4GB

Bump downburst VMs to 4GB

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #867 from ceph/wip-16049

pcp: Don't fail a job if graph downloading fails

Reviewed-by: Dan Mick <dmick@redhat.com>

pcp: Don't fail a job if graph downloading fails

http://tracker.ceph.com/issues/16049
Fixes: 16049
Signed-off-by: Zack Cerza <zack@redhat.com>

Fix abbreviated TestPCPTask.test_end()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #866 from ceph/wip-suite-not-found

Give a better error message for "suite not found"

test_describe_tests: remove 'manual test stubs'

Use a combined strategy: for test classes, use setup
to manage the list of patchers, also starting them
in setup because all the class-based tests use the same
fake fs data.

For the file-scope test functions, use @patch to manage them,
because the filesystem data changes with each test.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_suite: recode to use patch.start()

This lets us do all the setup/teardown in an organized way,
in class instance methods, so the test cases don't have to
deal with the mock arguments; all they do is pass the
fake_fs to start_patchers to get the right fake_fs data for
their particular test.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: update tests to handle new 'exists' fs operator

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.py: tell you that the suite you asked for doesn't exist

I know it's not in fashion to issue useful error messages, but...

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #865 from ceph/wip-13925

Prevent workers from becoming zombies when repo operations fail

Reviewed-by: Dan Mick <dmick@redhat.com>

When bootstrap/fetch fails, mark the job as dead

Note that this will only happen *after* all retries have failed.

http://tracker.ceph.com/issues/13925
Fixes: 13925
Signed-off-by: Zack Cerza <zack@redhat.com>

Move MaxWhileTries to the exceptions module

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a test for issue 13925

It will fail - the next commit fixes it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a test for the worker's main loop

Signed-off-by: Zack Cerza <zack@redhat.com>

Move job preparation to new prep_job method

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #863 from dmick/wip-suite-sha1

suite.py: allow specifying ceph sha1

test_suite: add new ceph_sha1 param, tests for same

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: add '--sha1/S' to specify Ceph sha1 directly

Check that sha1 exists in the git repo if supplied.  Avoid
checking that branch exists if sha1 is given (but keep it
because other things use branchname...job names,
pulpito, etc.)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Add a unit test for a nonexistent ceph branch

This test checks whether scheduling fails when an invalid ceph branch is
specified

Signed-off-by: Zack Cerza <zack@redhat.com>

Add SkipJob

Signed-off-by: Zack Cerza <zack@redhat.com>

Store job.jid as job_id

This is a trivial commit to prepare for a refactor

Signed-off-by: Zack Cerza <zack@redhat.com>