Instead of StreamHandler, rework copy_file_to()

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop use of xreadlines()

Signed-off-by: Zack Cerza <zack@redhat.com>

Move some stream management into RemoteProcess

For better organization and less copypasta.

Signed-off-by: Zack Cerza <zack@redhat.com>

Log stderr and stdout as long as they aren't pipes

Signed-off-by: Zack Cerza <zack@redhat.com>

Rewrite orchestra.run tests using mock

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #912 from smithfarm/wip-drop-ceph-devel

packages.yaml: drop ceph-devel from rpm section

packages.yaml: drop ceph-devel from rpm section

Upgrade tests on https://github.com/ceph/ceph/9744 (which drops the ceph-devel
package after a long period of deprecation) are currently failing
because teuthology tries to install ceph-devel, even though it is not needed
for any tests.

See also https://github.com/ceph/ceph-qa-suite/pull/1068

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #905 from lande1234/wip-lan-testing

fix wrong words

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #910 from ceph/wip-worker-startup

Stop workers on demand

Dan Mick <dmick@redhat.com>

Merge pull request #911 from ceph/wip-queue-status

teuthology-queue: add --status

teuthology-queue: add --status

Prints queue status, e.g.:

    $ teuthology-queue -m smithi --status
    {'paused': False, 'count': 2900, 'name': 'smithi'}

Signed-off-by: Zack Cerza <zack@redhat.com>

Stop workers on demand

Generalize need_restart()

So that we can also use it to tell the workers to stop

Signed-off-by: Zack Cerza <zack@redhat.com>

Revert "task/packages: install libcephfs1-dev"

This reverts commit 48be9e89df559dce13731fcdf59153181ffd2646.

Fixes: http://tracker.ceph.com/issues/16703
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #909 from ceph/wip-worker-startup

Fix slow worker startup

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix slow worker startup

We can selectively skip fetching and/or bootstrapping if either have
been completed successfully in the last 60s

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #908 from ceph/wip-sched-fail-ceph-ver

Fix regression in schedule failure emails

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix regression in schedule failure emails

Emails weren't getting sent when builds were broken

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #904 from dmick/wip-version

teuthology: log version number

Merge pull request #907 from ceph/wip-16640

task/packages: install libcephfs1-dev

task/packages: install libcephfs1-dev

The way we currently build the cephfs java
bindings depends on the unversioned .so file
which is in the -dev package, not the
main one.

Fixes: http://tracker.ceph.com/issues/16640
Signed-off-by: John Spray <john.spray@redhat.com>

teuthology: log version number

Round 2, this time with testing

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #902 from dmick/wip-schedule-suite-test

Add tests for schedule_suite(); fix some schedule_fail arguments

Merge pull request #906 from dmick/wip-email

Resume paying attention to --email

Resume paying attention to --email

Change in 61350db caused email arg to be lost.  Fix, add test

Signed-off-by: Dan Mick <dan.mick@redhat.com>

fix wrong words

Signed-off-by: lande1234 <lan.de3@zte.com.cn>

Merge pull request #903 from ceph/revert-901-wip-version

Revert "Add teuthology git version query/logging"

Revert "Add teuthology git version query/logging"

Add tests for suite.run.schedule_suite

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #901 from dmick/wip-version

Add teuthology git version query/logging

Add teuthology git version query/logging

Tested exception handling by renaming /usr/bin/git

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Fix arguments to schedule_fail() again

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #890 from ceph/wip-fixlistindex-wusui

Fix resolve_role_list in daemon.py

Merge pull request #897 from ceph/wip-provision-subpackage

Make teuthology.provision a subpackage

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #864 from jcsp/wip-nocolor

task/ansible: disable color output

Merge pull request #898 from ceph/wip-gbp-dmick

Add 'tag' and 'ref' to GitbuilderProject's init, warn if more than one set, and tests

Fix resolve_role_list in daemon.py

resolve_role_list did not handle cases where roles specified contained wild-cards or
were not specified.  ceph.role.a would end up being resolved as ceph.rolea.

Fixes: tracker issue 16053
Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #894 from ceph/wip-newest

Complete --newest feature, add some cleanups

Use 'conf' instead of 'fn' for the processed arg YamlConfig

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.main: use processed args consistently

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.run.collect_jobs(): deepcopy base_config

deep_merge, despite the name, doesn't do fully deep merging;
it recurses through the whole object, but does not deepcopy
component objects at the lowest level.  This meant that
self.base_config could be corrupted by the second deep_merge
(if base_config contained keys that were augmented by
parsed_yaml, which was often the case, for things like overrides).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: move config tmpfile management to Run

base_config changes over the course of the run, and
so cannot be written to a tmpfile until it's completely
finalized.  Move the create/write/rm out of main() and into
schedule_suite().

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject tests: also test for the right log msg

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject: if multiple ref-selectors present, warn

...about which one is actually being used

Signed-off-by: Dan Mick <dan.mick@redhat.com>

GitbuilderProject: allow setting tag or ref in config dict

Precedence: ref, tag, branch, sha1.  (this is debatable.)

Also change log.debug() default-to-master to log.warning()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #899 from ceph/wip-ansible-2

Move to ansible 2

Reviewed-by: Dan Mick <dmick@redhat.com>

Use ansible >= 2.0

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #895 from ceph/wip-no-pyc

Stop generating .pyc files

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #896 from ceph/wip-test-reqs

Add test requirements

Reviewed-by: Dan Mick <dmick@redhat.com>

Add test requirements

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove leftover .pyc files

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't write pyc files

The .pyc files that python generates can be problematic if we move
modules around; teuthology processes end up picking up the old .pyc
files instead of finding the new .py files

Signed-off-by: Zack Cerza <zack@redhat.com>

Move openstack-specific code to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst-specific code to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.provision a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #893 from ceph/wip-gbp

GitbuilderProject: respect branch in config

Reviewed-by: Dan Mick <dmick@redhat.com>

GitbuilderProject: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject test: respect branch in config

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #892 from ceph/wip-10481-new

Install -release RPMs directly

Reviewed-by: Dan Mick <dmick@redhat.com>

Install -release RPMs directly

This avoids having to dance around with wget and rm -f.
Fixes: 10481
http://tracker.ceph.com/issues/10481

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: make threadlocalptr whitelist more generic

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #888 from dmick/master

suite.py: must pass at least once through the backtrack loop

suite.py: must pass at least once through the backtrack loop

Changing the default of newest to '0' meant that the combined
loop which handles both backtrack and non-backtrack conditions
was never executing, leading to undefined variables.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #887 from dmick/master

suite.py: defaulting newest to 10 means it's always enabled

suite.py: defaulting newest to 10 means it's always enabled

Even if --newest isn't supplied, docopts will create it and
set it to the default.  Change the default to 0.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #886 from dmick/master

suite: --newest fixes: failure to rebuild base_config on sha1 backtrack, spurious log msg

suite/run.py: rebuild base config on sha1 backtrack

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: refactor to reuse config substitution code

In --newest mode we must recreate the base config, because
the ceph sha1 has been substituted several times already.
Factor the substitution code out of create_initial_config so
that it can be called from the backtracking code (which requires
storing kernel_dict and config_input on the instance).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/run.py: don't warn about backtrack failure if not --newest

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #885 from dmick/master

suite/run.py: fix order of args to schedule_fail()

suite/run.py: fix order of args to schedule_fail()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #884 from dmick/master

teuthology-suite: Spelling error in web API + minor test improvement

teuthology/task/valgrind: more rocksdb suppressions

Signed-off-by: Sage Weil <sage@redhat.com>

Come to an agreement about how to spell "committish"

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_find_git_parent: Compress the Mock setup a bit

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #883 from dmick/wip-newest

Add teuthology-suite --newest

Add test for util.find_git_parent()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

teuthology-suite: Implement --newest

If --newest is supplied, missing packages for one of the
required distro/versions are not fatal; instead, use a web
service to backtrack to older sha1s in an attempt to find the
newest commonly-built version.  Limit backtracking to
'--newest' commits (default 10).

See http://github.com/ceph/gitserver for the web service source.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

util.py: use passed package_versions dict even if empty

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Factor "collect jobs" and "schedule jobs" from schedule_suite

packaging.GitbuilderProject: allow more spec from config

To use GbP as a build-checking resource, allow more values
to come from config rather than hardcoding or deriving.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #849 from ceph/wip-fix-overrides

Fix ceph-deploy 'branch' override

Merge pull request #882 from ceph/wip-rocksdb-leak

valgrind: add supporession for rocksdb thread local leaks

Reviewed-by: Samuel Just <sjust@redhat.com>

Fix ceph-deploy 'branch' override

Default ceph-deploy override template defines devcommit which makes
'branch' overrides in the qa-suite useless since it only picksup one option
Remove the devcommit overrides from the template and use that as
default in case no other 'branch' override exist.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #880 from ceph/wip-docs

Docs overhaul

Several clarifications and removals

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind: add supporession for rocksdb thread local leaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #881 from ceph/wip-downburst-wait

Pass --wait to downburst

Reviewed-by: Dan Mick <dmick@redhat.com>

Pass --wait to downburst

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst section to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix reference to lab_setup

Signed-off-by: Zack Cerza <zack@redhat.com>

Move detailed test config info to its own doc

Signed-off-by: Zack Cerza <zack@redhat.com>

Add intro for testers

Signed-off-by: Zack Cerza <zack@redhat.com>

Add new README.rst specifically for GitHub

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename 'contents' to 'index'

Signed-off-by: Zack Cerza <zack@redhat.com>

Move OpenStack backend section to its own file

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #879 from ceph/wip-16335

Fix regression in --subset

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #878 from ceph/wip-16302

Install python 2.7 on ubuntu downburst VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix regression in --subset

We were processing the --subset arg well after it was necessary :-/
http://tracker.ceph.com/issues/16335
Fixes: 16335
Signed-off-by: Zack Cerza <zack@redhat.com>

Install python 2.7 on ubuntu downburst VMs

http://tracker.ceph.com/issues/16302
Fixes: 16302
Signed-off-by: Zack Cerza <zack@redhat.com>