install/rpm: wipe all zypper repos before adding repos-under-test

Our tests should be designed to not rely on any pre-existing repos. If there are
any pre-existing repos, these can potentially skew test results by causing
unexpected packages/package versions to be installed.

Therefore, wipe any pre-existing repos before adding the repos-under-test.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1309 from jtlayton/fixes

Minor bugfixes for teuthology

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1313 from kshtsk/wip-open-instead-file

python3: use open() function instead file() class

python3: use open() function instead file() class

Since python3 does not have file() we should stop using it.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1310 from kshtsk/wip-all-ipv4-octets-in-target-names

openstack: encode instance name with the full IP

Remove BaseException.message deprecation warning

DeprecationWarning: BaseException.message has been deprecated as of Python 2.6

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Use yaml.safe_load to suppress deprecation warning

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: encode instance name with the full IP

Using only the last two numbers is problematic when mixing multiple
OpenStack clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/provision.py

Merge pull request #847 from kmroz/wip-docs

docs: add SUSE to list of bootstrapable operating systems

Merge pull request #1092 from gohighsec/master

docs: use kernel version distro instead of jewel in the example of ru…

describe_tests: skip over hidden directories

The ceph/qa directory has a bunch of .qa symlinks under it that cause
teuthology-describe-tests to hit symlink loops. Just skip any dentry
with a name that starts with '.'.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

bootstrap: force usage of python2 in virtualenv command

None of this code seems to be python3 ready, so force the virtualenv
command in the bootstrap script to use python2 on a box where both are
available.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

Merge pull request #1302 from rishabh-d-dave/rewrite-remote-mktemp

orchestra/remote: rewrite mktemp to use coreutils mktemp

Merge pull request #1303 from rhcs-dashboard/wip-minimal-reqs

setup.py: move mysqlclient/coverage to optional deps

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1300 from smithfarm/wip-install-no-purge

install: drop purge_data and related code

install: drop purge_data and related code

purge_data was used to remove any Ceph daemon metadata remaining after the
teardown phase of the install task runs. In particular, the ceph task was
known to leave OSD metadata behind in the OSD directories under /var/lib/ceph.

While the install task causes the ceph-osd package to be installed and creates
directories under /var/lib/ceph, it is the ceph task that actually
populates these directories. Thus, their "unpopulation" should be
implemented in the ceph task, and not here in the install task.

(If the directories were empty after the ceph task teardown phase, they would be
removed when the install task removes the ceph-osd package from the system. But
the package management tool refuses to remove them when they contain files not
owned by a package.)

Not running purge_data during the teardown phase of the install task makes it
easier to implement alternative deployment tasks (i.e. tests that deploy Ceph
clusters, but do not call ceph.py).

In the final analysis, however, this code is no longer needed now that all test
nodes (virtual and bare metal) are reimaged between runs.

This commit does not move purge_data into the ceph task because the author felt
that the ceph task should take more care to clean up after itself. For example,
nowadays there are ceph CLI commands for removing OSDs.

Aside: this commit also drops two lines of dead code that flake8 was
complaining about.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1305 from kshtsk/wip-safe-load

suite/run: use yaml.safe_load

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suite/run: use yaml.safe_load

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1304 from batrick/selinux-smartd

selinux: ignore smartd

Reviewed-by: Kefu Chai <kchai@redhat.com>

selinux: ignore smartd

The failures [1]:

    2019-08-22T22:11:34.747 INFO:teuthology.orchestra.run.smithi195:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\)'
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7593): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.811 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8009): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1

[1] /ceph/teuthology-archive/pdonnell-2019-08-22_20:06:33-fs-wip-pdonnell-testing-20190822.163036-distro-basic-smithi/4242071/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.py: move myqslclient to optional deps

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

orchestra/remote: rewrite mktemp to use coreutils mktemp

Use mktemp instead of Python's tempfile.mktemp since coreutils is more
easily available.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1301 from rishabh-d-dave/remote-add-mkdtemp

orchestra/remote: add a method to create temporary directory

orchestra/remote: add a method to create temporary directory

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1299 from kshtsk/wip-kernel-none

suite/run: add support 'none' for kernel branch parameter

suite/run: add support 'none' for kernel branch parameter

This can be useful when one wants to disable kernel hash

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1282 from kshtsk/wip-suse-disable-ansible

initial_tasks: make it possible to disable ansible

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #1298 into master

* refs/pull/1298/head:
teuthology/orchestra/daemon: fix the log type
teuthology/orchestra/daemon: Check if stdin is valid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/orchestra/daemon: fix the log type

Signed-off-by: Jos Collin <jcollin@redhat.com>

teuthology/orchestra/daemon: Check if stdin is valid

As there is self.proc.stdin.close() in stop(), there are chances
of write failing with IOError("File is closed").

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #1296 into master

* refs/pull/1296/head:
teuthology/console: allow deferring login after reset

Reviewed-by: David Galloway <dgallowa@redhat.com>

teuthology/console: allow deferring login after reset

Fixes: http://tracker.ceph.com/issues/37681
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #1295 into master

* refs/pull/1295/head:
orchestra/daemon: Fix proc AttributeError: 'NoneType'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

orchestra/daemon: Fix proc AttributeError: 'NoneType'

Fixes the chance of calling signal() on a  daemon which is not running.

Signed-off-by: Jos Collin <jcollin@redhat.com>

initial_tasks: make it possible to disable ansible

Introduce ceph_cm_ansible option in order to ommit 'ansible.cephlab'
task, so alternative way of target node provisioning can be selected

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1289 from kshtsk/fix-testexiter

imports: make imports absolute

Merge pull request #1293 from kshtsk/add-param-to-config-load

config: add parameter for config load

Merge pull request #1294 from ceph/update-deps

Update dependencies to address security issues

pip-compile -P ansible

pip-compile -P paramiko

pip-compile -P pyopenssl

pip-compile -P cryptography

pip-compile -P requests

pip-compile -P pyyaml

pip-compile -P jinja2

py3: make imports python2/3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

config: add parameter for config load

This patch makes it possible to load config file directly
from the given path or data object, for testing purposes
and allows to remove silly notification about missing
teuthology.yaml file.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1287 from kshtsk/wip-suse-libcloud-userdata-cfg

libcloud: read userdata from config

libcloud: add test for userdata read from config

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1291 from batrick/i39554

orchestra/run: add omit_sudo arg used by vstart_runner

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

orchestra/run: add omit_sudo arg used by vstart_runner

Introduced-by: ceph/ceph.git:3e0a1361f78452d79c283f70b5dc2b865d62cb61
Fixes: https://tracker.ceph.com/issues/39554
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1281 from kshtsk/wip-suse-fix-install-rpms

install/rpm: add install_ceph_packages and repos_only options

libcloud: read userdata from config

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

install/rpm: add install_ceph_packages and repos_only options

This commit adds two new boolean options to the install task:

install_ceph_packages (defaults to "True") - controls whether or not ceph
packages are installed. Note: the "librados2" and "ceph-test" packages get
installed even if this option is set to False.

repos_only (defaults to "False") - when this option is set to "True",
the install task merely adds repos and does nothing else.

The new options currently work only on RPM systems: on DEB systems they are
ignored.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1288 from kshtsk/suse-chrony

task/clock: add chrony support

task/clock: add chrony support

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1290 from kshtsk/fix-tox-mock

tox: set mock version to 2.0.0

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test/test_exit: set mock version to 2.0.0

After upgrade mock from 2.0.0 to 3.0.3 the TestExiter started failing
with incorrect call_count.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1279 from kshtsk/wip-suse-fixes-1

suse related fixes

Merge pull request #1280 from kshtsk/wip-suse-fix-reconnect

fix reconnect

Merge pull request #1286 from ceph/wip-yuval-add-pubsub-deps

rgw/pubsub: add dependencies needed for pubsub tests

Merge pull request #1274 from ceph/wip-move-valgrind_supp

valgrind.supp: move the whitelist file to /qa dir of ceph.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1285 from gekios/wip-ceph-dev-env-fix

openstack: always pass branch and repo parameters

openstack: always pass branch and repo parameters

We need to always pass the parameters below in order to make it work using
environment variables:
--ceph
--ceph-repo
--suite-repo
--suite-branch
--teuthology-branch

Signed-off-by: gkyratsas <gkyratsas@suse.com>
Conflicts:
teuthology/openstack/__init__.py

test/opsys add opensuse 15.1 case

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

syslog.py: whitelist innocent tcmu-runner log message

When running the new iSCSI smoke test, tcmu-runner emits a syslog
message that was not whitelisted:

2019-03-27T10:00:17.292154+00:00 target192168000056 tcmu-runner[37366]: 2019-03-27 10:00:17.291 37366 [INFO] load_our_module:537: Inserted module 'target_core_user'

Since the message is at severity level INFO, it was causing teuthology
to raise a red flag.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

task: internal/syslog: expand salt exclusion

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/task/internal/syslog.py

task: syslog: salt INFO messages generate false negatives

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/task/internal/syslog.py

opsys: Drop opensuse-leap from DISTRO_CODENAME_MAP

Drop 'opensuse-leap' from DISTRO_CODENAME_MAP in favor of 'opensuse'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Add new openSUSE Leap ID for os-release

misc: add check to delete_file

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

misc: get_system_type: openSUSE means rpm

2018-03-03T23:59:57.524 INFO:teuthology.orchestra.run.target149202163177:Running: 'sudo lsb_release -is'
2018-03-03T23:59:57.600 INFO:teuthology.orchestra.run.target149202163177.stdout:openSUSE
2018-03-03T23:59:57.601 DEBUG:teuthology.misc:System to be installed: openSUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

misc.py: Handle SLE-12-SP2 in get_system_type()

In SLE-12-SP1, "lsb_release -is" returns "SUSE LINUX". In SLE-12-SP2, it
returns "SUSE".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

opsys: opensuse id related fix

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

misc.py: saner write_file() and sudo_write_file()

Using python to do this might have made sense in 2011, but it doesn't now.

Our hand is forced here, because systems are no longer guaranteed to
have an executable called "python".

For background details on why this is happening, see
https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3#.2Fusr.2Fbin.2Fpython

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: enable testing on openSUSE Leap 15.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: support openSUSE Leap 15.0

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1275 from ceph/wip-default-os

update default OS versions

update default OS versions

centos/rhel 7.4 -> 7.6
ubuntu 16.04 -> 18.04

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind.supp: move the whitelist file to /qa dir of ceph.git

This change drops valgrind.supp whitelist file from teuthology
repo and alters the install task to acquire it from /qa dir of
ceph-qa-suite repository.
It's expected the commit will have sibiling in ceph.git moving
the proper valgrind.supp there.

The rationale behind the entire change is duplication of these
files between two repos, their inconsistencies and confusion.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #1284 into master

* refs/pull/1284/head:
selinux: ignore updatedb failures

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

selinux: ignore updatedb failures

This is with RHEL 7.6.

The failure [1]:

    2019-04-17T21:06:33.388 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi088.front.sepia.ceph.com: ['type=AVC msg=audit(1555531592.378:4451): avc:  denied  { rename } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { read write open } for  pid=10476 comm="updatedb" path="/var/lib/mlocate/mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { create } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { add_name } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1555531591.622:4447): avc:  denied  { write } for  pid=10476 comm="updatedb" path="/var/lib/mlocate/mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531592.377:4450): avc:  denied  { setattr } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { write } for  pid=10476 comm="updatedb" name="mlocate" dev="sda1" ino=76978 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1']

[1] /ceph/teuthology-archive/pdonnell-2019-04-17_06:02:40-fs-wip-pdonnell-testing-20190417.032809-distro-basic-smithi/3856822/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1278 from kshtsk/wip-log-userdata

cloud/openstack: log userdata for debug purpose

orchestra: fix no connection after reboot

This resolves the issue when remote lost connection
if the command was a reboot of the host and there was
only one unsuccessful try to reconnect and as a result
was loosing transport and got stuck with message like:

WARNING:tasks.deepsea.orch:No connection established yet..

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

Fix code to allow reconncting to a node after the connection was lost.

This patch introduces a 'context' parameter for
teuthology.orchestra.remote's run(..) method which
works around an issue with paramiko.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

cloud/openstack: log userdata for debug purpose

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1276 from tchaikov/wip-python34-to-python36

task/install: normalize python3 packages to python36 ones

Reviewed-by: Boris Ranto <branto@redhat.com>

task/install: normalize python3 packages to python36 ones

EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/pubsub: add dependencies needed for pubsub tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #1271 from kshtsk/wip-openstack-arch

add openstack arch

Merge pull request #1270 from kshtsk/wip-allow_sizes_and_networks

libcloud: add allow_sizes and allow_networks

Merge pull request #1272 from kshtsk/wip-ssh-interface

add ssh_interface property

openstack: enable openSUSE Leap 42.2

Fixes: #29
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: drop some obsolete images

Centos <7.2 is obsolete, Ubuntu <14 is obsolete

(Note that the openstack-integration tests download all of these images
one-by-one, so this is a time-saving measure.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: update Debian 8.0 image URL to current

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: enable CentOS 7.3

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: add Leap 42.2 cloud image URL comment

Just a commented-out URL for now, until we figure out how to handle .tbz
extension

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: debian now is 8.7.0

Signed-off-by: Loic Dachary <ldachary@redhat.com>

openstack: use 16.04 for teuthology VM

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: debian jessie is now 8.5.0

Signed-off-by: Loic Dachary <loic@dachary.org>