add note that ceph.conf.template should no longer be modified

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1320 from kshtsk/wip-fog-python3

fog: get rid of StringIO usage for python3 portability

fog: get rid of StringIO usage for python3 portability

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1317 from kshtsk/wip-python3-use-isinstance

suite/matrix: use isinstance for python3 compatibility

run_tasks: use isinstance() instead of type()

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology/exit: use isinstance for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: use isinstance for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1315 from kshtsk/wip-future-print

use python3 compatible print

Merge pull request #1307 from kshtsk/beanstalkc3

beanstalk: switch to py3 compatible package

test_build_matrix: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

provision/cloud: use python3 compatible syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

parallel: use python3 compatible syntax for print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

packaging: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

contextutil: python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: fix print syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/cli: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

schedule: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ls: use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

results: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: fix tab inconstent indents

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

beanstalk: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

kill: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fix tab inconsistent indent

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe_tests: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: avoid using print for writing files

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1314 from kshtsk/wip-no-iteritems

Get rid of iteritems and xrange

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

python3: get rid of iteritems

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ansible: use py3 compatible octals

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: get rid of xrange

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1309 from jtlayton/fixes

Minor bugfixes for teuthology

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1313 from kshtsk/wip-open-instead-file

python3: use open() function instead file() class

python3: use open() function instead file() class

Since python3 does not have file() we should stop using it.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1310 from kshtsk/wip-all-ipv4-octets-in-target-names

openstack: encode instance name with the full IP

Remove BaseException.message deprecation warning

DeprecationWarning: BaseException.message has been deprecated as of Python 2.6

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Use yaml.safe_load to suppress deprecation warning

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: encode instance name with the full IP

Using only the last two numbers is problematic when mixing multiple
OpenStack clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/provision.py

Merge pull request #847 from kmroz/wip-docs

docs: add SUSE to list of bootstrapable operating systems

Merge pull request #1092 from gohighsec/master

docs: use kernel version distro instead of jewel in the example of ru…

describe_tests: skip over hidden directories

The ceph/qa directory has a bunch of .qa symlinks under it that cause
teuthology-describe-tests to hit symlink loops. Just skip any dentry
with a name that starts with '.'.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

bootstrap: force usage of python2 in virtualenv command

None of this code seems to be python3 ready, so force the virtualenv
command in the bootstrap script to use python2 on a box where both are
available.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

Merge pull request #1302 from rishabh-d-dave/rewrite-remote-mktemp

orchestra/remote: rewrite mktemp to use coreutils mktemp

Merge pull request #1303 from rhcs-dashboard/wip-minimal-reqs

setup.py: move mysqlclient/coverage to optional deps

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

beanstalk: switch to py3 compatible package

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1300 from smithfarm/wip-install-no-purge

install: drop purge_data and related code

install: drop purge_data and related code

purge_data was used to remove any Ceph daemon metadata remaining after the
teardown phase of the install task runs. In particular, the ceph task was
known to leave OSD metadata behind in the OSD directories under /var/lib/ceph.

While the install task causes the ceph-osd package to be installed and creates
directories under /var/lib/ceph, it is the ceph task that actually
populates these directories. Thus, their "unpopulation" should be
implemented in the ceph task, and not here in the install task.

(If the directories were empty after the ceph task teardown phase, they would be
removed when the install task removes the ceph-osd package from the system. But
the package management tool refuses to remove them when they contain files not
owned by a package.)

Not running purge_data during the teardown phase of the install task makes it
easier to implement alternative deployment tasks (i.e. tests that deploy Ceph
clusters, but do not call ceph.py).

In the final analysis, however, this code is no longer needed now that all test
nodes (virtual and bare metal) are reimaged between runs.

This commit does not move purge_data into the ceph task because the author felt
that the ceph task should take more care to clean up after itself. For example,
nowadays there are ceph CLI commands for removing OSDs.

Aside: this commit also drops two lines of dead code that flake8 was
complaining about.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1305 from kshtsk/wip-safe-load

suite/run: use yaml.safe_load

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suite/run: use yaml.safe_load

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1304 from batrick/selinux-smartd

selinux: ignore smartd

Reviewed-by: Kefu Chai <kchai@redhat.com>

selinux: ignore smartd

The failures [1]:

    2019-08-22T22:11:34.747 INFO:teuthology.orchestra.run.smithi195:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\)'
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7593): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.811 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8009): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1

[1] /ceph/teuthology-archive/pdonnell-2019-08-22_20:06:33-fs-wip-pdonnell-testing-20190822.163036-distro-basic-smithi/4242071/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.py: move myqslclient to optional deps

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

orchestra/remote: rewrite mktemp to use coreutils mktemp

Use mktemp instead of Python's tempfile.mktemp since coreutils is more
easily available.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1301 from rishabh-d-dave/remote-add-mkdtemp

orchestra/remote: add a method to create temporary directory

orchestra/remote: add a method to create temporary directory

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1299 from kshtsk/wip-kernel-none

suite/run: add support 'none' for kernel branch parameter

suite/run: add support 'none' for kernel branch parameter

This can be useful when one wants to disable kernel hash

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1282 from kshtsk/wip-suse-disable-ansible

initial_tasks: make it possible to disable ansible

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #1298 into master

* refs/pull/1298/head:
teuthology/orchestra/daemon: fix the log type
teuthology/orchestra/daemon: Check if stdin is valid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/orchestra/daemon: fix the log type

Signed-off-by: Jos Collin <jcollin@redhat.com>

teuthology/orchestra/daemon: Check if stdin is valid

As there is self.proc.stdin.close() in stop(), there are chances
of write failing with IOError("File is closed").

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #1296 into master

* refs/pull/1296/head:
teuthology/console: allow deferring login after reset

Reviewed-by: David Galloway <dgallowa@redhat.com>

teuthology/console: allow deferring login after reset

Fixes: http://tracker.ceph.com/issues/37681
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #1295 into master

* refs/pull/1295/head:
orchestra/daemon: Fix proc AttributeError: 'NoneType'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

orchestra/daemon: Fix proc AttributeError: 'NoneType'

Fixes the chance of calling signal() on a  daemon which is not running.

Signed-off-by: Jos Collin <jcollin@redhat.com>

initial_tasks: make it possible to disable ansible

Introduce ceph_cm_ansible option in order to ommit 'ansible.cephlab'
task, so alternative way of target node provisioning can be selected

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1289 from kshtsk/fix-testexiter

imports: make imports absolute

Merge pull request #1293 from kshtsk/add-param-to-config-load

config: add parameter for config load

Merge pull request #1294 from ceph/update-deps

Update dependencies to address security issues

pip-compile -P ansible

pip-compile -P paramiko

pip-compile -P pyopenssl

pip-compile -P cryptography

pip-compile -P requests

pip-compile -P pyyaml

pip-compile -P jinja2

py3: make imports python2/3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

config: add parameter for config load

This patch makes it possible to load config file directly
from the given path or data object, for testing purposes
and allows to remove silly notification about missing
teuthology.yaml file.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1287 from kshtsk/wip-suse-libcloud-userdata-cfg

libcloud: read userdata from config

libcloud: add test for userdata read from config

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1291 from batrick/i39554

orchestra/run: add omit_sudo arg used by vstart_runner

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

orchestra/run: add omit_sudo arg used by vstart_runner

Introduced-by: ceph/ceph.git:3e0a1361f78452d79c283f70b5dc2b865d62cb61
Fixes: https://tracker.ceph.com/issues/39554
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1281 from kshtsk/wip-suse-fix-install-rpms

install/rpm: add install_ceph_packages and repos_only options

libcloud: read userdata from config

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

install/rpm: add install_ceph_packages and repos_only options

This commit adds two new boolean options to the install task:

install_ceph_packages (defaults to "True") - controls whether or not ceph
packages are installed. Note: the "librados2" and "ceph-test" packages get
installed even if this option is set to False.

repos_only (defaults to "False") - when this option is set to "True",
the install task merely adds repos and does nothing else.

The new options currently work only on RPM systems: on DEB systems they are
ignored.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1288 from kshtsk/suse-chrony

task/clock: add chrony support

task/clock: add chrony support

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1290 from kshtsk/fix-tox-mock

tox: set mock version to 2.0.0

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test/test_exit: set mock version to 2.0.0

After upgrade mock from 2.0.0 to 3.0.3 the TestExiter started failing
with incorrect call_count.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1279 from kshtsk/wip-suse-fixes-1

suse related fixes

Merge pull request #1280 from kshtsk/wip-suse-fix-reconnect

fix reconnect

Merge pull request #1286 from ceph/wip-yuval-add-pubsub-deps

rgw/pubsub: add dependencies needed for pubsub tests

Merge pull request #1274 from ceph/wip-move-valgrind_supp

valgrind.supp: move the whitelist file to /qa dir of ceph.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1285 from gekios/wip-ceph-dev-env-fix

openstack: always pass branch and repo parameters

openstack: always pass branch and repo parameters

We need to always pass the parameters below in order to make it work using
environment variables:
--ceph
--ceph-repo
--suite-repo
--suite-branch
--teuthology-branch

Signed-off-by: gkyratsas <gkyratsas@suse.com>
Conflicts:
teuthology/openstack/__init__.py

test/opsys add opensuse 15.1 case

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

syslog.py: whitelist innocent tcmu-runner log message

When running the new iSCSI smoke test, tcmu-runner emits a syslog
message that was not whitelisted:

2019-03-27T10:00:17.292154+00:00 target192168000056 tcmu-runner[37366]: 2019-03-27 10:00:17.291 37366 [INFO] load_our_module:537: Inserted module 'target_core_user'

Since the message is at severity level INFO, it was causing teuthology
to raise a red flag.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

task: internal/syslog: expand salt exclusion

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/task/internal/syslog.py

task: syslog: salt INFO messages generate false negatives

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/task/internal/syslog.py