openstack: install docker in the teuthology instance

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: install ceph-workbench

Signed-off-by: Loic Dachary <loic@dachary.org>
Conflicts:
teuthology/openstack/__init__.py

Merge pull request #1327 from kshtsk/misc-drop-stringio-import

misc: drop StringIO import to fix flake8

Reviewed-by: Nathan Cutler <ncutler@suse.com>

misc: drop StringIO import to fix flake8

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1318 from kshtsk/wip-misc-use-remote-sh

misc: use remote.sh instead of remote.run

Merge PR #1322 into master

* refs/pull/1322/head:
add note that ceph.conf.template should no longer be modified

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1323 from kshtsk/wip-misc-drop-legacy-code-in-skeleton_config

misc: drop testdir replacement in skeleton_config

Merge pull request #1325 from rjfd/wip-fix-import-loop

lock/ops: remove import to break import loop

lock/ops: remove import to break import loop

Fixes: https://tracker.ceph.com/issues/42256
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #1324 from badone/wip-remove-package-overrides

install/rpm: Remove _package_overrides

Reviewed-by: Kefu Chai <kchai@redhat.com>

install/rpm: Remove _package_overrides

The _package_overrides function is currently not required.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

misc: drop testdir replacement in skeleton_config

The str.format is not used anymore for ceph.conf.template

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1321 from kshtsk/wip-ceph-cm-ansible-url

config: add option for ceph cm ansible git url

Merge pull request #1312 from ryg-/wip-42081-set-os

lock: update target os-type and os-version after provisioning

add note that ceph.conf.template should no longer be modified

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

config: add option for ceph cm ansible git url

This is to be able to supply a custom location without changing
the ceph git base url.

Signed-off-by: Roi Dayan <roid@mellanox.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/config.py
teuthology/suite.py
teuthology/test/test_suite.py

Merge pull request #1320 from kshtsk/wip-fog-python3

fog: get rid of StringIO usage for python3 portability

fog: get rid of StringIO usage for python3 portability

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1317 from kshtsk/wip-python3-use-isinstance

suite/matrix: use isinstance for python3 compatibility

misc: use remote.sh instead of remote.run

To ease porting and compatibility in order to avoid
usage of cStringIO.StringIO it is better to use
remote.sh method which supposed to return captured
value as a string.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

run_tasks: use isinstance() instead of type()

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology/exit: use isinstance for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: use isinstance for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock: remove os_version and os_type for node before reprovisiong

os_version and os_type don't reflect actual os version
when node is under provisioning and/or provisiong failed.
The fix cleanup fileds before provisioning.

Signed-off-by: Roman Grigoryev <roman.grigorev@suse.de>

lock: update target os-type and os-version after provisioning

After provisioning node via "teuthology-lock" (via both
"--lock" and "--lock-many") users cannot see actual
versioni and type  of provisioned OS in pulpito.
It is pretty helpful to see that version if we need to
do some manual work.

Signed-off-by: Roman Grigoryev <roman.grigorev@suse.de>

Merge pull request #1315 from kshtsk/wip-future-print

use python3 compatible print

Merge pull request #1307 from kshtsk/beanstalkc3

beanstalk: switch to py3 compatible package

test_build_matrix: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

provision/cloud: use python3 compatible syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

parallel: use python3 compatible syntax for print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

packaging: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

contextutil: python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: fix print syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/cli: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

schedule: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ls: use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

results: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: fix tab inconstent indents

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

beanstalk: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

kill: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fix tab inconsistent indent

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe_tests: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: avoid using print for writing files

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1314 from kshtsk/wip-no-iteritems

Get rid of iteritems and xrange

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

python3: get rid of iteritems

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ansible: use py3 compatible octals

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: get rid of xrange

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1309 from jtlayton/fixes

Minor bugfixes for teuthology

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1313 from kshtsk/wip-open-instead-file

python3: use open() function instead file() class

python3: use open() function instead file() class

Since python3 does not have file() we should stop using it.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1310 from kshtsk/wip-all-ipv4-octets-in-target-names

openstack: encode instance name with the full IP

Remove BaseException.message deprecation warning

DeprecationWarning: BaseException.message has been deprecated as of Python 2.6

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Use yaml.safe_load to suppress deprecation warning

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: encode instance name with the full IP

Using only the last two numbers is problematic when mixing multiple
OpenStack clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/provision.py

Merge pull request #847 from kmroz/wip-docs

docs: add SUSE to list of bootstrapable operating systems

Merge pull request #1092 from gohighsec/master

docs: use kernel version distro instead of jewel in the example of ru…

describe_tests: skip over hidden directories

The ceph/qa directory has a bunch of .qa symlinks under it that cause
teuthology-describe-tests to hit symlink loops. Just skip any dentry
with a name that starts with '.'.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

bootstrap: force usage of python2 in virtualenv command

None of this code seems to be python3 ready, so force the virtualenv
command in the bootstrap script to use python2 on a box where both are
available.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

Merge pull request #1302 from rishabh-d-dave/rewrite-remote-mktemp

orchestra/remote: rewrite mktemp to use coreutils mktemp

Merge pull request #1303 from rhcs-dashboard/wip-minimal-reqs

setup.py: move mysqlclient/coverage to optional deps

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

beanstalk: switch to py3 compatible package

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1300 from smithfarm/wip-install-no-purge

install: drop purge_data and related code

install: drop purge_data and related code

purge_data was used to remove any Ceph daemon metadata remaining after the
teardown phase of the install task runs. In particular, the ceph task was
known to leave OSD metadata behind in the OSD directories under /var/lib/ceph.

While the install task causes the ceph-osd package to be installed and creates
directories under /var/lib/ceph, it is the ceph task that actually
populates these directories. Thus, their "unpopulation" should be
implemented in the ceph task, and not here in the install task.

(If the directories were empty after the ceph task teardown phase, they would be
removed when the install task removes the ceph-osd package from the system. But
the package management tool refuses to remove them when they contain files not
owned by a package.)

Not running purge_data during the teardown phase of the install task makes it
easier to implement alternative deployment tasks (i.e. tests that deploy Ceph
clusters, but do not call ceph.py).

In the final analysis, however, this code is no longer needed now that all test
nodes (virtual and bare metal) are reimaged between runs.

This commit does not move purge_data into the ceph task because the author felt
that the ceph task should take more care to clean up after itself. For example,
nowadays there are ceph CLI commands for removing OSDs.

Aside: this commit also drops two lines of dead code that flake8 was
complaining about.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1305 from kshtsk/wip-safe-load

suite/run: use yaml.safe_load

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suite/run: use yaml.safe_load

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1304 from batrick/selinux-smartd

selinux: ignore smartd

Reviewed-by: Kefu Chai <kchai@redhat.com>

selinux: ignore smartd

The failures [1]:

    2019-08-22T22:11:34.747 INFO:teuthology.orchestra.run.smithi195:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\)'
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7593): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.811 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8009): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1

[1] /ceph/teuthology-archive/pdonnell-2019-08-22_20:06:33-fs-wip-pdonnell-testing-20190822.163036-distro-basic-smithi/4242071/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.py: move myqslclient to optional deps

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

orchestra/remote: rewrite mktemp to use coreutils mktemp

Use mktemp instead of Python's tempfile.mktemp since coreutils is more
easily available.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1301 from rishabh-d-dave/remote-add-mkdtemp

orchestra/remote: add a method to create temporary directory

orchestra/remote: add a method to create temporary directory

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1299 from kshtsk/wip-kernel-none

suite/run: add support 'none' for kernel branch parameter

suite/run: add support 'none' for kernel branch parameter

This can be useful when one wants to disable kernel hash

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1282 from kshtsk/wip-suse-disable-ansible

initial_tasks: make it possible to disable ansible

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #1298 into master

* refs/pull/1298/head:
teuthology/orchestra/daemon: fix the log type
teuthology/orchestra/daemon: Check if stdin is valid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/orchestra/daemon: fix the log type

Signed-off-by: Jos Collin <jcollin@redhat.com>

teuthology/orchestra/daemon: Check if stdin is valid

As there is self.proc.stdin.close() in stop(), there are chances
of write failing with IOError("File is closed").

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #1296 into master

* refs/pull/1296/head:
teuthology/console: allow deferring login after reset

Reviewed-by: David Galloway <dgallowa@redhat.com>

teuthology/console: allow deferring login after reset

Fixes: http://tracker.ceph.com/issues/37681
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #1295 into master

* refs/pull/1295/head:
orchestra/daemon: Fix proc AttributeError: 'NoneType'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

orchestra/daemon: Fix proc AttributeError: 'NoneType'

Fixes the chance of calling signal() on a  daemon which is not running.

Signed-off-by: Jos Collin <jcollin@redhat.com>

initial_tasks: make it possible to disable ansible

Introduce ceph_cm_ansible option in order to ommit 'ansible.cephlab'
task, so alternative way of target node provisioning can be selected

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1289 from kshtsk/fix-testexiter

imports: make imports absolute

Merge pull request #1293 from kshtsk/add-param-to-config-load

config: add parameter for config load

Merge pull request #1294 from ceph/update-deps

Update dependencies to address security issues

pip-compile -P ansible

pip-compile -P paramiko

pip-compile -P pyopenssl

pip-compile -P cryptography

pip-compile -P requests

pip-compile -P pyyaml

pip-compile -P jinja2

py3: make imports python2/3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

config: add parameter for config load

This patch makes it possible to load config file directly
from the given path or data object, for testing purposes
and allows to remove silly notification about missing
teuthology.yaml file.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1287 from kshtsk/wip-suse-libcloud-userdata-cfg

libcloud: read userdata from config

libcloud: add test for userdata read from config

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1291 from batrick/i39554

orchestra/run: add omit_sudo arg used by vstart_runner

Reviewed-by: Greg Farnum <gfarnum@redhat.com>