Set ctx.cluster even when a job has no roles

This is to allow for teuthology to run self-tests that don't use
resources.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #952 from ceph/wip-17211

suite.Run: include suite_sha1 in job config

Reviewed-by: Dan Mick <dmick@redhat.com>

suite.Run: include suite_sha1 in job config

http://tracker.ceph.com/issues/17211
Fixes: #17211
Signed-off-by: Zack Cerza <zack@redhat.com>

Linter fixes

Merge pull request #950 from jcsp/wip-cn-lst-err

orchestra.run: fix generating ConnectionLostError

orchestra.run: fix generating ConnectionLostError

Previously you could get AttributeError if
a node went away, because it tried to call
getpeername after get_transport() had
returned None.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #941 from ceph/wip-conserver

Support console sessions using conserver

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #947 from ceph/wip-17102

Fix logging regression

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

copy_to_log(): Drop use of readlines()

http://tracker.ceph.com/issues/17102
Fixes: 17102
Calling readlines() on the source object causes the teuthology thread to
block on the remote command's execution. Instead, iterate over the
source object as advised at:
https://docs.python.org/2.7/library/stdtypes.html#file.xreadlines

This interface is also supported by StringIO and cStringIO.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a global switch to disable conserver

Signed-off-by: Zack Cerza <zack@redhat.com>

If conserver fails, fall back to ipmitool

Signed-off-by: Zack Cerza <zack@redhat.com>

Make IPMI arguments to PhysicalConsole optional

Since we now look them up in the global configuration anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a test for has_ipmi_credentials

Signed-off-by: Zack Cerza <zack@redhat.com>

console_log: Be aware of conserver

Signed-off-by: Zack Cerza <zack@redhat.com>

spawn_sol_log(): don't use the shell

Shell usage is keeping the spawned command from terminating properly.

Signed-off-by: Zack Cerza <zack@redhat.com>

Use read-only conserver sessions when appropriate

Signed-off-by: Zack Cerza <zack@redhat.com>

_exit_session(): also work with conserver-client

Signed-off-by: Zack Cerza <zack@redhat.com>

Add tests for spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Get console sessions using _get_console()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add an integration test to expose a logging bug

This is related to: http://tracker.ceph.com/issues/17102

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #943 from ceph/wip-downburst-pkgs

Install required packages when downbursting VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Install wget on downbursted VMs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Install redhat-lsb-core on downbursted centos VMs for `lsb_release`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Install git on downbursted VMs during cloud-init

Fixes: http://tracker.ceph.com/issues/17154
Signed-off-by: David Galloway <dgallowa@redhat.com>

Rename _check_credentials(); move calls

It's now _check_ipmi_credentials() and is only called by _ipmi_command()

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename _pexpect_spawn() to _pexpect_spawn_ipmi()

A new _pexpect_spawn() is added, to be used for general cases. It is
used by _pexpect_spawn_ipmi().

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename _build_command() to _ipmi_command()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add _console_command()

This provides a way to generate a command that establishes a SOL console
session that, depending on whether conserver settings are present, will
use that or fall back to ipmitool.

Signed-off-by: Zack Cerza <zack@redhat.com>

Make PhysicalConsole aware of conserver settings

Signed-off-by: Zack Cerza <zack@redhat.com>

Add settings for conserver

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: drop map() in __init__()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add some tests for the console module

... better late than never?

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix a circular import issue

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #945 from ceph/wip-kernel-hier-config

kernel: don't use GitbuilderProject's _get_config_value_for_remote()

kernel: don't use GitbuilderProject's _get_config_value_for_remote()

Commit a01f1e169074 ("kernel: use packaging.GitbuilderProject to talk
with gitbuilder.") switched the kernel task to GitbuilderProject, which
parses the config in an attempt to pick the "best" config value for
a given remote.  However, the kernel task does its own parsing and
supports more than GitbuilderProject currently handles, e.g.:

  kernel:
    client:
      branch: testing

The above got broken by passing the entire 'kernel:' stanza to
GitbuilderProject: _get_uri_reference() chokes on "client" and defaults
to master branch.  It also effectively disables 'overrides:' support,
used by krbd:unmap suite.  Bring these back by passing in only the role
config in task().

In download_kernel(), we already have the sha1 and just need to get the
url, so construct a {'sha1': <sha>} dict on the fly.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #944 from ceph/wip-console-creds

Console: rename has_credentials

Merge pull request #942 from dmick/wip-token

openstack: Cache auth tokens, add unit test fixes

Console: rename has_credentials

... to has_ipmi_credentials

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #858 from smithfarm/wip-openstack-opensuse

Get dummy suite to run on openSUSE Leap in OpenStack

test_openstack: misc.sh() is now called with -q

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Fix nuke openstack tests (patch OpenStack().run instead of misc.sh)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #937 from ceph/wip-console-log

Log IPMI console output for bare metal hosts

Reviewed-by: Dan Mick <dmick@redhat.com>

Use the new console_log class

Signed-off-by: Zack Cerza <zack@redhat.com>

Add console_log task

Signed-off-by: Zack Cerza <zack@redhat.com>

Patch teardown in some test methods

Specifically, test_setup_called() and test_begin_called()

Signed-off-by: Zack Cerza <zack@redhat.com>

Check ipmi credentials in a separate method

And also call it from spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add PhysicalConsole.spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Move ipmi credential storage to PhysicalConsole

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole._pexpect_spawn(): pass logfile arg

... directly when calling pexpect.spawn()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #938 from ceph/wip-17084

Kill valgrind.bin processes.

Reviewed-by: Dan Mick <dmick@danceorelse.org>

Merge pull request #939 from smithfarm/wip-17075

Revert "packages.yaml: reflect python-ceph package split"

Revert "packages.yaml: reflect python-ceph package split"

This reverts commit 4543fb64cedffc97c816feff7d4a69cd0456ab15.

Fixes: http://tracker.ceph.com/issues/17075
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Kill valgrind.bin processes

http://tracker.ceph.com/issues/17084
Fixes: 17084
Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: rename _exec() to _pexpect_spawn()

Signed-off-by: Zack Cerza <zack@redhat.com>

openstack: cache auth tokens

Fixes: http://tracker.ceph.com/issues/16893
Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add get_provider() to match set_provider()

So that set_provider does not need to be called explicitly before using
the .provider data member.

Signed-off-by: Loic Dachary <loic@dachary.org>

PhysicalConsole: split out command generation

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #936 from ceph/wip-fix-nuke-imports

Fix imports of nuke actions from other modules

Reviewed-by: Dan Mick <dmick@redhat.com>

Fix imports of nuke actions from other modules

Specifically, ceph-qa-suite.git/tasks/cephfs/filesystem.py apparently
needs the clear_firewall action.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #935 from ceph/wip-nuke-improvements

Nuke improvements, and a bonus refactor

Reviewed-by: Dan Mick <dmick@redhat.com>

Check locks before doing anything else

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole.power_cycle(): use timeout=300

Signed-off-by: Zack Cerza <zack@redhat.com>

If console access fails, try SSH

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ConsoleError, and use it

... instead of just a RuntimeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix reference to ipmi domain

Signed-off-by: Zack Cerza <zack@redhat.com>

Move most of nuke's actions into their own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Add Remote.console

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.nuke a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

check_console(): attempt a power cycle

If the host is unresponsive and its power is on, reboot it.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole._wait_for_login(): reduce attempts

Instead of trying six times by default, try twice.

Signed-off-by: Zack Cerza <zack@redhat.com>

Move console checking to its own function

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove redundant timeout assignment

Signed-off-by: Zack Cerza <zack@redhat.com>

Move console classes to their own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Cleanup some timeout logic

Signed-off-by: Zack Cerza <zack@redhat.com>

Make console checking more readable

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #933 from tchaikov/wip-17035

valgrind: add suppression for libunwind in libtcmalloc.so.4.2.6

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

valgrind: add suppression for libunwind in libtcmalloc.so.4.2.6

Fixes: http://tracker.ceph.com/issues/17035
Signed-off-by: Kefu Chai <kchai@redhat.com>

PhysicalConsole._wait_for_login() raise on failure

Signed-off-by: Zack Cerza <zack@redhat.com>

getRemoteConsole(): look up IPMI credentials

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: remove packages before reboot

Just in case daemons coming up after reboot is problematic.
Related: http://tracker.ceph.com/issues/17027

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #930 from ceph/wip-openstack-regression

 Work around regression in openstacksdk 0.9.2

Reviewed-by: Dan Mick <dmick@redhat.com>

 Work around regression in openstacksdk 0.9.2

... that caused `openstack ip floating list -f json` to fail and return
"Connection failure that may be retried."

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #922 from ceph/wip-install-py3

install: Allow installing python3 packages universally

Merge pull request #927 from dachary/wip-16972-quiet

openstack: do not display openstack client warnings

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #928 from ceph/revert-925-wip-ovh-json

Revert "misc.sh(): Don't capture stderr by default"

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "misc.sh(): Don't capture stderr by default"

openstack: do not display openstack client warnings

Because they are concatenated with the STDOUT by the sh function despite
being output on STDERR and that breaks json parsing.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #926 from ceph/wip-beanstalkd-socketerror

worker: Don't die on a beanstalkc SocketError

Reviewed-by: Dan Mick <dmick@redhat.com>

Don't die on a beanstalkc SocketError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #925 from ceph/wip-ovh-json

misc.sh(): Don't capture stderr by default

Reviewed-by: Dan Mick <dmick@redhat.com>

misc.sh(): Don't capture stderr by default

If not capturing it, log it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #924 from smithfarm/wip-lock-centos

teuthology-lock: add centos 7.2 to downburst default table

Merge pull request #923 from ceph/wip-14794

valgrind: wildcard boost supression

Merge pull request #916 from smithfarm/wip-python-split

packages.yaml: reflect python-ceph package split

teuthology-lock: drop openSUSE 12.2 from downburst default table

AFAIK Downburst does not provide any openSUSE images.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

teuthology-lock: add centos 7.2 to downburst default table

When downburst is unavailable, teuthology-lock reverts to default_table and
fails because the only supported CentOS version, 7.2., is missing.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

valgrind: wildcard boost supression

...so that it will match mangled names.

Fixes: http://tracker.ceph.com/issues/14794
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #921 from ceph/wip-ansible-dead

CephLab: playbook failures result in status 'dead'

CephLab: playbook failures result in status 'dead'

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #914 from ceph/wip-orchestra-logging

orchestra.run logging

Reviewed-by: Dan Mick <dmick@redhat.com>