valgrind: add supporession for rocksdb thread local leaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #881 from ceph/wip-downburst-wait

Pass --wait to downburst

Reviewed-by: Dan Mick <dmick@redhat.com>

Pass --wait to downburst

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #879 from ceph/wip-16335

Fix regression in --subset

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #878 from ceph/wip-16302

Install python 2.7 on ubuntu downburst VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix regression in --subset

We were processing the --subset arg well after it was necessary :-/
http://tracker.ceph.com/issues/16335
Fixes: 16335
Signed-off-by: Zack Cerza <zack@redhat.com>

Install python 2.7 on ubuntu downburst VMs

http://tracker.ceph.com/issues/16302
Fixes: 16302
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #877 from ceph/wip-suite-fix

get_gitbuilder_hash: use default distro if needed

get_gitbuilder_hash: use default distro if needed

http://tracker.ceph.com/issues/16328
Fixes: 16328
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #872 from ceph/wip-suite-gbp

Suite refactor

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix indenting

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove duplicate deep_copy call and no-op assert

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unnecessary hyphen

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #876 from batrick/bash-syntax-error

nuke.py: remove trailing fi token

nuke.py: remove trailing fi token

This caused a syntax error in one of the teuthology runs:

    2016-06-10T02:30:39.291 INFO:teuthology.nuke:clearing kernel mount from all nodes
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run:Running command with timeout 60
    2016-06-10T02:30:39.292 INFO:teuthology.orchestra.run.mira069:Running: "grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi"
    2016-06-10T02:30:39.363 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: syntax error near unexpected token `fi'
    2016-06-10T02:30:39.364 INFO:teuthology.orchestra.run.mira069.stderr:bash: -c: line 0: `grep ceph /etc/mtab | grep -o 'on /.* type' | grep -o '/.* ' | xargs -r sudo umount -f ; fi'

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Explain the return value of m_time

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #873 from ceph/wip-pcp-archive

pcp: Use a smarter check for archive presence

Reviewed-by: Dan Mick <dmick@redhat.com>

pcp: Use a smarter check for archive presence

If ctx.archive was set, but to None, we were failing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.suite a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Refactor parts of suite into new Run class

Including smaller, more purposeful methods. This is for readability,
testability and to facilitate making future changes.

Signed-off-by: Zack Cerza <zack@redhat.com>

Speed up test_wait_*

They weren't patching sleep(). Now they do :)

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove unused get_gitbuilder_url()

Its functionality is covered by config.baseurl_template anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Use GitbuilderProject for more methods

Signed-off-by: Zack Cerza <zack@redhat.com>

Add OS.__eq__()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Don't hardcode arch

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Use opsys and packaging modules

Starting with get_distro_defaults()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Drop GitbuilderProject._get_codename()

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Fix GBP._get_distro() for fedora

Gitbuilders use URLs like:
http://gitbuilder.ceph.com/ceph-rpm-fedora20-x86_64-basic/ref/hammer/sha1
Not:
http://gitbuilder.ceph.com/ceph-rpm-fc20-x86_64-basic/ref/hammer/sha1

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Don't raise if a codename isn't found

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Add more codenames

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: Mark three methods as static/class

... methods

Signed-off-by: Zack Cerza <zack@redhat.com>

packaging: use OS's new codename discovery

Signed-off-by: Zack Cerza <zack@redhat.com>

OS: Automatically determine version or codename

... using a lookup table to map one to the other

Signed-off-by: Zack Cerza <zack@redhat.com>

Move constants to opsys module

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #868 from ceph/wip-4gb-vm

Bump downburst VMs to 4GB

Bump downburst VMs to 4GB

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #867 from ceph/wip-16049

pcp: Don't fail a job if graph downloading fails

Reviewed-by: Dan Mick <dmick@redhat.com>

pcp: Don't fail a job if graph downloading fails

http://tracker.ceph.com/issues/16049
Fixes: 16049
Signed-off-by: Zack Cerza <zack@redhat.com>

Fix abbreviated TestPCPTask.test_end()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #866 from ceph/wip-suite-not-found

Give a better error message for "suite not found"

test_describe_tests: remove 'manual test stubs'

Use a combined strategy: for test classes, use setup
to manage the list of patchers, also starting them
in setup because all the class-based tests use the same
fake fs data.

For the file-scope test functions, use @patch to manage them,
because the filesystem data changes with each test.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_suite: recode to use patch.start()

This lets us do all the setup/teardown in an organized way,
in class instance methods, so the test cases don't have to
deal with the mock arguments; all they do is pass the
fake_fs to start_patchers to get the right fake_fs data for
their particular test.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: update tests to handle new 'exists' fs operator

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.py: tell you that the suite you asked for doesn't exist

I know it's not in fashion to issue useful error messages, but...

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #865 from ceph/wip-13925

Prevent workers from becoming zombies when repo operations fail

Reviewed-by: Dan Mick <dmick@redhat.com>

When bootstrap/fetch fails, mark the job as dead

Note that this will only happen *after* all retries have failed.

http://tracker.ceph.com/issues/13925
Fixes: 13925
Signed-off-by: Zack Cerza <zack@redhat.com>

Move MaxWhileTries to the exceptions module

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a test for issue 13925

It will fail - the next commit fixes it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a test for the worker's main loop

Signed-off-by: Zack Cerza <zack@redhat.com>

Move job preparation to new prep_job method

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #863 from dmick/wip-suite-sha1

suite.py: allow specifying ceph sha1

test_suite: add new ceph_sha1 param, tests for same

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: add '--sha1/S' to specify Ceph sha1 directly

Check that sha1 exists in the git repo if supplied.  Avoid
checking that branch exists if sha1 is given (but keep it
because other things use branchname...job names,
pulpito, etc.)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Add a unit test for a nonexistent ceph branch

This test checks whether scheduling fails when an invalid ceph branch is
specified

Signed-off-by: Zack Cerza <zack@redhat.com>

Add SkipJob

Signed-off-by: Zack Cerza <zack@redhat.com>

Store job.jid as job_id

This is a trivial commit to prepare for a refactor

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #860 from ceph/wip-pcp

Add PCP task

Reviewed-by: Dan Mick <dmick@redhat.com>

Add PCP task

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #862 from ceph/wip-15427

Remote: log file sizes before transferring

Remote._sftp_get_file(): Log file size

This is so that jobs transferring large files don't get mistaken for
being hung.

Fixes: http://tracker.ceph.com/issues/15427
Signed-off-by: Zack Cerza <zack@redhat.com>

Add Remote._format_size()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add Remote._sftp_get_size()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add missing unit test for Remote._sftp_open_file

Signed-off-by: Zack Cerza <zack@redhat.com>

Rewrite unit tests for orchestra.remote

They were using Fudge. Rather than write more Fudge tests, I decided to
rewrite the module to use Mock like almost everything else in teuthology
does.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #861 from smithfarm/wip-opensuse-codename

packaging.py: make _get_distro() support openSUSE

Merge pull request #859 from ceph/wip-15862

task/hadoop: fix for multi cluster is_type() change

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

packaging.py: make _get_distro() support openSUSE

Add an explicit "elif" block for opensuse - without this, it falls through into
the else clause and dies because it cannot determine a codename.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Add run.Raw.__eq__()

Signed-off-by: Zack Cerza <zack@redhat.com>

task/hadoop: fix for multi cluster is_type() change

This is the only non-ceph task that uses custom roles, and it happens
to put a '.' as a delimiter between hadoop and the rest of the
role. Since the multi-cluster branch, is_type() is ceph-specific and
interprets this as a cluster named hadoop.

To avoid needing to change several branches in ceph-qa-suite, just use
a local is_hadoop_type() function instead of relying on the now
ceph-specific misc.is_type().

Fixes: http://tracker.ceph.com/issues/15862
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #841 from ceph/wip-multi-cluster

support multiple ceph clusters

Merge pull request #855 from dmick/wip-dependencies

bootstrap: Use mariadb-devel on RHEL for mysql-devel; remove mysqlclient dependency

mysqlclient is not needed; MySQLdb-python is for teuthology-coverage

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Use the appropriate package name for mysql/mariadb

Community package download isn't necessary; RHEL ships
mariadb and its -devel package

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #854 from ceph/wip-15711

Always mention name and sha1 when scheduling fails

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Always mention name and sha1 when scheduling fails

http://tracker.ceph.com/issues/15711
Fixes: 15711
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #832 from ceph/wip-fix-nuke

 fix nuke

Merge pull request #851 from ceph/wip-prune-mon

prune: also remove mon data in jobs' 'data' subdir

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #853 from ceph/wip-pcp2

task/selinux: another pcp whitelist

Reviewed-by: Boris Ranto <branto@redhat.com>

task/selinux: another pcp whitelist

SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1462234212.274:85266): avc: denied { read } for pid=1984 comm="pmcd" name="pmlogger_daily.pid" dev="tmpfs" ino=1474542 scontext=system_u:system_r:pcp_pmcd_t:s0 tcontext=system_u:object_r:cron_var_run_t:s0 tclass=file']

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #852 from ceph/wip-pcp-selinux

task.selinux: add PCP's pmlogger to the whitelist

task.selinux: add PCP's pmlogger to the whitelist

Signed-off-by: Zack Cerza <zack@redhat.com>

Prune mon data along with remote logs

Signed-off-by: Zack Cerza <zack@redhat.com>

Move subdir removal logic into separate function

Signed-off-by: Zack Cerza <zack@redhat.com>

Add timeout to commands

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix nuke, redo some cleanup functions

Redo some functions, remove proc waits, Reorder functions
for better cleanup.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Remove ceph and its dependent packages

Remove ceph and its dependent packages using force option

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use timeout feature of paramiko

use timeout feature of paramiko when executing commands

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #836 from ceph/wip-rh-bootstrap

Changes for bootstrap to work on RHEL

Merge pull request #850 from ceph/wip-paramiko

Unpin paramiko

Reviewed-by: Dan Mick <dmick@redhat.com>

Unpin paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #848 from ceph/wip-filter-wusui

Restrict filter to not scan too far up a path.

Restrict filter to not scan too far up a path.
Directory names closer to the root than /suites are not used by the filter.
Fixes: http://tracker.ceph.com/issues/15470
Signed-off-by: Warren Usui <wusui@redhat.com>

Changes for bootstrap to work on RHEL

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #846 from ceph/wip-babel

Work around pip dependency issue w/ nova and Babel

Reviewed-by: Dan Mick <dmick@redhat.com>

Work around pip dependency issue w/ nova and Babel

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #845 from ceph/wip-vault-pass

ansible.CephLab: touch ~/.vault_pass.txt

Reviewed-by: Dan Mick <dmick@redhat.com>

ansible.CephLab: touch ~/.vault_pass.txt

To avoid ansible bailing with an OSError. Normally we don't even need
the vault password for cephlab.yml.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #843 from ceph/wip-gethash

teuthology-suite: Don't allow scheduling when packages are not present

Reviewed-by: Dan Mick <dmick@redhat.com>

Simplify message counting filtered jobs

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't attempt to schedule when missing packages

By default, anyway. It's possible to set
config.suite_allow_missing_packages to continue stick with the old
behavior.

Signed-off-by: Zack Cerza <zack@redhat.com>