Add an integration test to expose a logging bug

This is related to: http://tracker.ceph.com/issues/17102

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #943 from ceph/wip-downburst-pkgs

Install required packages when downbursting VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Install wget on downbursted VMs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Install redhat-lsb-core on downbursted centos VMs for `lsb_release`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Install git on downbursted VMs during cloud-init

Fixes: http://tracker.ceph.com/issues/17154
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #945 from ceph/wip-kernel-hier-config

kernel: don't use GitbuilderProject's _get_config_value_for_remote()

kernel: don't use GitbuilderProject's _get_config_value_for_remote()

Commit a01f1e169074 ("kernel: use packaging.GitbuilderProject to talk
with gitbuilder.") switched the kernel task to GitbuilderProject, which
parses the config in an attempt to pick the "best" config value for
a given remote.  However, the kernel task does its own parsing and
supports more than GitbuilderProject currently handles, e.g.:

  kernel:
    client:
      branch: testing

The above got broken by passing the entire 'kernel:' stanza to
GitbuilderProject: _get_uri_reference() chokes on "client" and defaults
to master branch.  It also effectively disables 'overrides:' support,
used by krbd:unmap suite.  Bring these back by passing in only the role
config in task().

In download_kernel(), we already have the sha1 and just need to get the
url, so construct a {'sha1': <sha>} dict on the fly.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #944 from ceph/wip-console-creds

Console: rename has_credentials

Merge pull request #942 from dmick/wip-token

openstack: Cache auth tokens, add unit test fixes

Console: rename has_credentials

... to has_ipmi_credentials

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #858 from smithfarm/wip-openstack-opensuse

Get dummy suite to run on openSUSE Leap in OpenStack

test_openstack: misc.sh() is now called with -q

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Fix nuke openstack tests (patch OpenStack().run instead of misc.sh)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #937 from ceph/wip-console-log

Log IPMI console output for bare metal hosts

Reviewed-by: Dan Mick <dmick@redhat.com>

Use the new console_log class

Signed-off-by: Zack Cerza <zack@redhat.com>

Add console_log task

Signed-off-by: Zack Cerza <zack@redhat.com>

Patch teardown in some test methods

Specifically, test_setup_called() and test_begin_called()

Signed-off-by: Zack Cerza <zack@redhat.com>

Check ipmi credentials in a separate method

And also call it from spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add PhysicalConsole.spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Move ipmi credential storage to PhysicalConsole

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole._pexpect_spawn(): pass logfile arg

... directly when calling pexpect.spawn()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #938 from ceph/wip-17084

Kill valgrind.bin processes.

Reviewed-by: Dan Mick <dmick@danceorelse.org>

Merge pull request #939 from smithfarm/wip-17075

Revert "packages.yaml: reflect python-ceph package split"

Revert "packages.yaml: reflect python-ceph package split"

This reverts commit 4543fb64cedffc97c816feff7d4a69cd0456ab15.

Fixes: http://tracker.ceph.com/issues/17075
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Kill valgrind.bin processes

http://tracker.ceph.com/issues/17084
Fixes: 17084
Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: rename _exec() to _pexpect_spawn()

Signed-off-by: Zack Cerza <zack@redhat.com>

openstack: cache auth tokens

Fixes: http://tracker.ceph.com/issues/16893
Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add get_provider() to match set_provider()

So that set_provider does not need to be called explicitly before using
the .provider data member.

Signed-off-by: Loic Dachary <loic@dachary.org>

PhysicalConsole: split out command generation

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #936 from ceph/wip-fix-nuke-imports

Fix imports of nuke actions from other modules

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix imports of nuke actions from other modules

Specifically, ceph-qa-suite.git/tasks/cephfs/filesystem.py apparently
needs the clear_firewall action.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #935 from ceph/wip-nuke-improvements

Nuke improvements, and a bonus refactor

Reviewed-by: Dan Mick <dmick@redhat.com>

Check locks before doing anything else

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole.power_cycle(): use timeout=300

Signed-off-by: Zack Cerza <zack@redhat.com>

If console access fails, try SSH

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ConsoleError, and use it

... instead of just a RuntimeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix reference to ipmi domain

Signed-off-by: Zack Cerza <zack@redhat.com>

Move most of nuke's actions into their own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Add Remote.console

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.nuke a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

check_console(): attempt a power cycle

If the host is unresponsive and its power is on, reboot it.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole._wait_for_login(): reduce attempts

Instead of trying six times by default, try twice.

Signed-off-by: Zack Cerza <zack@redhat.com>

Move console checking to its own function

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove redundant timeout assignment

Signed-off-by: Zack Cerza <zack@redhat.com>

Move console classes to their own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Cleanup some timeout logic

Signed-off-by: Zack Cerza <zack@redhat.com>

Make console checking more readable

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #933 from tchaikov/wip-17035

valgrind: add suppression for libunwind in libtcmalloc.so.4.2.6

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

valgrind: add suppression for libunwind in libtcmalloc.so.4.2.6

Fixes: http://tracker.ceph.com/issues/17035
Signed-off-by: Kefu Chai <kchai@redhat.com>

PhysicalConsole._wait_for_login() raise on failure

Signed-off-by: Zack Cerza <zack@redhat.com>

getRemoteConsole(): look up IPMI credentials

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: remove packages before reboot

Just in case daemons coming up after reboot is problematic.
Related: http://tracker.ceph.com/issues/17027

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #930 from ceph/wip-openstack-regression

 Work around regression in openstacksdk 0.9.2

Reviewed-by: Dan Mick <dmick@redhat.com>

 Work around regression in openstacksdk 0.9.2

... that caused `openstack ip floating list -f json` to fail and return
"Connection failure that may be retried."

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #922 from ceph/wip-install-py3

install: Allow installing python3 packages universally

Merge pull request #927 from dachary/wip-16972-quiet

openstack: do not display openstack client warnings

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #928 from ceph/revert-925-wip-ovh-json

Revert "misc.sh(): Don't capture stderr by default"

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "misc.sh(): Don't capture stderr by default"

openstack: do not display openstack client warnings

Because they are concatenated with the STDOUT by the sh function despite
being output on STDERR and that breaks json parsing.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #926 from ceph/wip-beanstalkd-socketerror

worker: Don't die on a beanstalkc SocketError

Reviewed-by: Dan Mick <dmick@redhat.com>

Don't die on a beanstalkc SocketError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #925 from ceph/wip-ovh-json

misc.sh(): Don't capture stderr by default

Reviewed-by: Dan Mick <dmick@redhat.com>

misc.sh(): Don't capture stderr by default

If not capturing it, log it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #924 from smithfarm/wip-lock-centos

teuthology-lock: add centos 7.2 to downburst default table

Merge pull request #923 from ceph/wip-14794

valgrind: wildcard boost supression

Merge pull request #916 from smithfarm/wip-python-split

packages.yaml: reflect python-ceph package split

teuthology-lock: drop openSUSE 12.2 from downburst default table

AFAIK Downburst does not provide any openSUSE images.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

teuthology-lock: add centos 7.2 to downburst default table

When downburst is unavailable, teuthology-lock reverts to default_table and
fails because the only supported CentOS version, 7.2., is missing.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

valgrind: wildcard boost supression

...so that it will match mangled names.

Fixes: http://tracker.ceph.com/issues/14794
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #921 from ceph/wip-ansible-dead

CephLab: playbook failures result in status 'dead'

CephLab: playbook failures result in status 'dead'

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #914 from ceph/wip-orchestra-logging

orchestra.run logging

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #920 from ceph/wip-worker-sentinel

Fix an obvious bug in sentinel()

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix an obvious bug in sentinel()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #917 from ceph/wip-dv-broken

Fix regression when using teuthology-suite with --kernel

Reviewed-by: Dan Mick <dmick@redhat.com>

choose_kernel(): fix call to get_gitbuilder_hash

http://tracker.ceph.com/issues/16838
Fixes: 16838
Signed-off-by: Zack Cerza <zack@redhat.com>

Remove defaults for get_gitbuilder_hash args

http://tracker.ceph.com/issues/16838
Fixes: 16838
Signed-off-by: Zack Cerza <zack@redhat.com>

Add reproducer for teuthology-suite regression

http://tracker.ceph.com/issues/16838

Signed-off-by: Zack Cerza <zack@redhat.com>

Update tests to reflect internal changes

Signed-off-by: Zack Cerza <zack@redhat.com>

Instead of StreamHandler, rework copy_file_to()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #913 from dmick/wip-os-version

teuthology-suite: add --D/--distroversion

Drop use of xreadlines()

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject: fix/improve warning about multiple refs

warn() tries to warn you when you overspecify a commit (by
giving more than one of ref, tag, branch, or sha1).

1) fix it to actually work (it was always warning)
2) Improve the warning to indicate not only which is taking
precedence but what was supplied for the others.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: Allow distroversion to contain either codename or version

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite/tests: add distroversion

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite: Add distroversion

Allow -D/--distroversion to select version of distro to schedule.
Arg can be either numeric version or codename.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Add OS class method 'version_codename' to infer one from the other

Signed-off-by: Dan Mick <dan.mick@redhat.com>

packages.yaml: reflect python-ceph package split

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #915 from dachary/wip-14799-valgrind

valgrind: add suppression for tcmalloc in libboost_thread-mt.so.1.53.0

valgrind: add suppression for tcmalloc in libboost_thread-mt.so.1.53.0

Use regular expression to catch:

tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long)

Fixes: http://tracker.ceph.com/issues/14799
Signed-off-by: Loic Dachary <loic@dachary.org>

Move some stream management into RemoteProcess

For better organization and less copypasta.

Signed-off-by: Zack Cerza <zack@redhat.com>

Log stderr and stdout as long as they aren't pipes

Signed-off-by: Zack Cerza <zack@redhat.com>

Rewrite orchestra.run tests using mock

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #912 from smithfarm/wip-drop-ceph-devel

packages.yaml: drop ceph-devel from rpm section

packages.yaml: drop ceph-devel from rpm section

Upgrade tests on https://github.com/ceph/ceph/9744 (which drops the ceph-devel
package after a long period of deprecation) are currently failing
because teuthology tries to install ceph-devel, even though it is not needed
for any tests.

See also https://github.com/ceph/ceph-qa-suite/pull/1068

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #905 from lande1234/wip-lan-testing

fix wrong words

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #910 from ceph/wip-worker-startup

Stop workers on demand

Dan Mick <dmick@redhat.com>

Merge pull request #911 from ceph/wip-queue-status

teuthology-queue: add --status

teuthology-queue: add --status

Prints queue status, e.g.:

    $ teuthology-queue -m smithi --status
    {'paused': False, 'count': 2900, 'name': 'smithi'}

Signed-off-by: Zack Cerza <zack@redhat.com>

Stop workers on demand