Update the default archive_base path

None of our labs are using /var/lib/teuthworker/archive anymore. The
incorrect default could trip up users.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: debug monc = 1

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #963 from ceph/wip-rerun

Add rerun capability to teuthology-suite

Reviewed-by: Dan Mick <dmick@redhat.com>

suite: Add rerun capability

Using a new --rerun argument for teuthology-suite, optionally paired
with --rerun-statuses, we can now ask teuthology to re-schedule a
suite-run. The main use-case for the feature is to re-schedule only
those jobs from a given run which failed.

Fixes: #10945
http://tracker.ceph.com/issues/10945

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ResultsReporter.get_run()

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: parse filter args into lists sooner

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #951 from ceph/wip-shaman-2

Support fetching packages from shaman instead of gitbuilder

Reviewed-by: Dan Mick <dmick@redhat.com>

ceph.conf: debug auth on mon

Tracking down an intermitted EPERM from mon... I think related to
rotating keys.

See http://tracker.ceph.com/issues/17495

Signed-off-by: Sage Weil <sage@redhat.com>

GitbuilderProject: remove 'v' from version values

Some version values have a leading 'v', e.g.:
http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/v10.2.0/version
is 'v10.2.0', whereas:
http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel/version
is '10.2.3-0'.

Signed-off-by: Zack Cerza <zack@redhat.com>

ShamanProject: correct ref precedence

Signed-off-by: Zack Cerza <zack@redhat.com>

More properly test ref precedence for Shaman

Signed-off-by: Zack Cerza <zack@redhat.com>

ShamanProject: split out search URI generation

Signed-off-by: Zack Cerza <zack@redhat.com>

Move logic from _get_uri_reference...

Into new _choose_reference() method. This is because uri_reference is
irrelevant to ShamanProject, but we still want to use the same rules for
deciding whether to search for ref/tag/branch/sha1 when more than one is
specified.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add tests for GitbuilderProject._get_uri_reference

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix typo in GitbuilderProject._get_uri_reference()

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject: always set codename

When a GitbuilderProject (and also ShamanProject) was being init-ed from
a Remote object, we weren't setting the codename.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.install: use builder.codename for deb repos

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject: don't mangle rpm versions

Up until now, given a package like:
"ceph-test-11.0.0-2135.g8403c95.x86_64", we were returning "11.0.0",
which doesn't allow us to properly verify installed package versions.
Start returning the entire string provided by gitbuilder so that we can
ensure we match the value returned by packaging.get_package_version().

For .deb packages, we have always used string like:
"11.0.0-2135-g8403c95-1xenial"

Signed-off-by: Zack Cerza <zack@redhat.com>

get_package_version(): return V-R for rpm

Even though strictly speaking "version" implies just the %{VERSION}
field, that is hardly ever useful on its own.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal: use get_builder_project()

Signed-off-by: Zack Cerza <zack@redhat.com>

run: Allow a job to override config.use_shaman

Signed-off-by: Zack Cerza <zack@redhat.com>

suite tests: use GitbuilderProject and not ShamanProject

Signed-off-by: Zack Cerza <zack@redhat.com>

task.install: use get_builder_project()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add and use get_builder_project()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ShamanProject

Signed-off-by: Zack Cerza <zack@redhat.com>

Move much of TestGitbuilderProject into base class

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #960 from ceph/wip-17391

teuthology-nuke: lazy-unmount osd mount points

Reviewed-by: Dan Mick <dmick@redhat.com>

remove_osd_mounts(): do lazy unmounts

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Add unlock_firmware_repo()

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: move logging calls, and fix linter warnings

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: remove ceph data later

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind.supp: tweak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #406 from ceph/wip-journal-zero

ceph.conf: journal_zero_on_create = true

ceph.conf: journal_zero_on_create = true

Make sure we test this code path.

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind.supp: another rocksdb leak

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind.supp: a rocksdb Env leak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #958 from ceph/wip-reweight-config

ceph.conf: set min rweight min bytes per osd = 10

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.conf: set min rweight min bytes per osd = 10

This lets the reweight-by-utilization command work even after mons are
restarted (ceph_manager is currently setting this via injectargs, but that
is not persistent).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #953 from rjfd/wip-opensuse-pr

OpenSUSE Leap support in teuthology

install: use --capability with zypper {install,remove}

Otherwise we pick up python-ceph 0.80.9 from the distro repo instead of
a newer python-ceph-compat from the test repo.

References: https://bugzilla.opensuse.org/show_bug.cgi?id=997751
Signed-off-by: Nathan Cutler <ncutler@suse.com>

install: opensuse: support for zypper package manager

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #949 from dachary/wip-openstack-token-endpoint

openstack: use OS_AUTH_TYPE=token_endpoint when possible

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #956 from ceph/wip-nodeless

Set ctx.cluster even when a job has no roles

Reviewed-by: Dan Mick <dmick@redhat.com>

Set ctx.cluster even when a job has no roles

This is to allow for teuthology to run self-tests that don't use
resources.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #955 from ceph/wip-revert-nodeless

Revert "Set ctx.cluster even when a job has no roles"

Revert "Set ctx.cluster even when a job has no roles"

This reverts commit ffd14be3bd79baec5d9cafd8972d592085e0e099.

Merge pull request #954 from ceph/wip-nodeless

Set ctx.cluster even when a job has no roles

Reviewed-by: Dan Mick <dmick@redhat.com>

Set ctx.cluster even when a job has no roles

This is to allow for teuthology to run self-tests that don't use
resources.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #952 from ceph/wip-17211

suite.Run: include suite_sha1 in job config

Reviewed-by: Dan Mick <dmick@redhat.com>

suite.Run: include suite_sha1 in job config

http://tracker.ceph.com/issues/17211
Fixes: #17211
Signed-off-by: Zack Cerza <zack@redhat.com>

Linter fixes

openstack: use OS_AUTH_TYPE=token_endpoint when possible

It requires setting the OS_URL depending on the command and avoids
calling the auth endpoint entirely.

Fixes: http://tracker.ceph.com/issues/16893
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #950 from jcsp/wip-cn-lst-err

orchestra.run: fix generating ConnectionLostError

orchestra.run: fix generating ConnectionLostError

Previously you could get AttributeError if
a node went away, because it tried to call
getpeername after get_transport() had
returned None.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #941 from ceph/wip-conserver

Support console sessions using conserver

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #947 from ceph/wip-17102

Fix logging regression

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

copy_to_log(): Drop use of readlines()

http://tracker.ceph.com/issues/17102
Fixes: 17102
Calling readlines() on the source object causes the teuthology thread to
block on the remote command's execution. Instead, iterate over the
source object as advised at:
https://docs.python.org/2.7/library/stdtypes.html#file.xreadlines

This interface is also supported by StringIO and cStringIO.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a global switch to disable conserver

Signed-off-by: Zack Cerza <zack@redhat.com>

If conserver fails, fall back to ipmitool

Signed-off-by: Zack Cerza <zack@redhat.com>

Make IPMI arguments to PhysicalConsole optional

Since we now look them up in the global configuration anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a test for has_ipmi_credentials

Signed-off-by: Zack Cerza <zack@redhat.com>

console_log: Be aware of conserver

Signed-off-by: Zack Cerza <zack@redhat.com>

spawn_sol_log(): don't use the shell

Shell usage is keeping the spawned command from terminating properly.

Signed-off-by: Zack Cerza <zack@redhat.com>

Use read-only conserver sessions when appropriate

Signed-off-by: Zack Cerza <zack@redhat.com>

_exit_session(): also work with conserver-client

Signed-off-by: Zack Cerza <zack@redhat.com>

Add tests for spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Get console sessions using _get_console()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add an integration test to expose a logging bug

This is related to: http://tracker.ceph.com/issues/17102

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #943 from ceph/wip-downburst-pkgs

Install required packages when downbursting VMs

Reviewed-by: Dan Mick <dmick@redhat.com>

Install wget on downbursted VMs

Signed-off-by: David Galloway <dgallowa@redhat.com>

Install redhat-lsb-core on downbursted centos VMs for `lsb_release`

Signed-off-by: David Galloway <dgallowa@redhat.com>

Install git on downbursted VMs during cloud-init

Fixes: http://tracker.ceph.com/issues/17154
Signed-off-by: David Galloway <dgallowa@redhat.com>

Rename _check_credentials(); move calls

It's now _check_ipmi_credentials() and is only called by _ipmi_command()

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename _pexpect_spawn() to _pexpect_spawn_ipmi()

A new _pexpect_spawn() is added, to be used for general cases. It is
used by _pexpect_spawn_ipmi().

Signed-off-by: Zack Cerza <zack@redhat.com>

Rename _build_command() to _ipmi_command()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add _console_command()

This provides a way to generate a command that establishes a SOL console
session that, depending on whether conserver settings are present, will
use that or fall back to ipmitool.

Signed-off-by: Zack Cerza <zack@redhat.com>

Make PhysicalConsole aware of conserver settings

Signed-off-by: Zack Cerza <zack@redhat.com>

Add settings for conserver

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: drop map() in __init__()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add some tests for the console module

... better late than never?

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix a circular import issue

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #945 from ceph/wip-kernel-hier-config

kernel: don't use GitbuilderProject's _get_config_value_for_remote()

kernel: don't use GitbuilderProject's _get_config_value_for_remote()

Commit a01f1e169074 ("kernel: use packaging.GitbuilderProject to talk
with gitbuilder.") switched the kernel task to GitbuilderProject, which
parses the config in an attempt to pick the "best" config value for
a given remote.  However, the kernel task does its own parsing and
supports more than GitbuilderProject currently handles, e.g.:

  kernel:
    client:
      branch: testing

The above got broken by passing the entire 'kernel:' stanza to
GitbuilderProject: _get_uri_reference() chokes on "client" and defaults
to master branch.  It also effectively disables 'overrides:' support,
used by krbd:unmap suite.  Bring these back by passing in only the role
config in task().

In download_kernel(), we already have the sha1 and just need to get the
url, so construct a {'sha1': <sha>} dict on the fly.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #944 from ceph/wip-console-creds

Console: rename has_credentials

Merge pull request #942 from dmick/wip-token

openstack: Cache auth tokens, add unit test fixes

Console: rename has_credentials

... to has_ipmi_credentials

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #858 from smithfarm/wip-openstack-opensuse

Get dummy suite to run on openSUSE Leap in OpenStack

test_openstack: misc.sh() is now called with -q

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Fix nuke openstack tests (patch OpenStack().run instead of misc.sh)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #937 from ceph/wip-console-log

Log IPMI console output for bare metal hosts

Reviewed-by: Dan Mick <dmick@redhat.com>

Use the new console_log class

Signed-off-by: Zack Cerza <zack@redhat.com>

Add console_log task

Signed-off-by: Zack Cerza <zack@redhat.com>

Patch teardown in some test methods

Specifically, test_setup_called() and test_begin_called()

Signed-off-by: Zack Cerza <zack@redhat.com>

Check ipmi credentials in a separate method

And also call it from spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add PhysicalConsole.spawn_sol_log()

Signed-off-by: Zack Cerza <zack@redhat.com>

Move ipmi credential storage to PhysicalConsole

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole._pexpect_spawn(): pass logfile arg

... directly when calling pexpect.spawn()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #938 from ceph/wip-17084

Kill valgrind.bin processes.

Reviewed-by: Dan Mick <dmick@danceorelse.org>

Merge pull request #939 from smithfarm/wip-17075

Revert "packages.yaml: reflect python-ceph package split"

Revert "packages.yaml: reflect python-ceph package split"

This reverts commit 4543fb64cedffc97c816feff7d4a69cd0456ab15.

Fixes: http://tracker.ceph.com/issues/17075
Signed-off-by: Nathan Cutler <ncutler@suse.com>