openstack: allow setting of teuthology VM params from command line

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: add new OVH flavors to filter list

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: user-data: add filesystems:ceph:mimic OBS repo

Temporary measure to make it possible for DeepSea to deploy Ceph
in openSUSE Leap 15.0

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: user-data: remove system librados2 librbd1

The pre-installed system librados2 and librbd1 packages may block us
from installing packages-under-test.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: user-data: tweak packages on openSUSE Leap 15.0

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: use install-deps.sh from local clone if available

Fixes: https://github.com/SUSE/teuthology/issues/53
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack/buildpackages: tweaked opensuse 42.2 user-data

Fixes: https://github.com/SUSE/teuthology/issues/54
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: add opensuse-42.2 userdata

Symlink to 42.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: conditionally fetch tags from ceph/ceph.git

A new parameter, canonical_tags, is added to .teuthology.yaml defaulting to
true.

A new command-line option "--no-canonical-tags" is added to teuthology-openstack
(when given, canonical_tags is set to false in .teuthology.yaml; otherwise it
defaults to true).

The buildpackages task is modified to take the appropriate action based on the
value of canonical_tags taken from .teuthology.yaml. Also, an info message is
displayed reminding the user that the command-line parameter only takes place
when the teuthology VM is created. Like --simultaneous-jobs, the parameter has
no effect in subsequent runs against an existing teuthology environment.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
scripts/openstack.py

buildpackages: use sle instead of sles for SUSE Linux Enterprise

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tasks: import buildpackages.py from ceph/ceph master

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: add centos-7.3-user-data.txt

Tests are failing with:

2017-02-18T11:07:34.742 DEBUG:teuthology.misc:timeout 30m openstack server create --image 'teuthology-centos-7.3-x86_64'  --flavor hg-60-ssd --key-name teuthology --security-group teuthology --property ownedby=149.202.169.26 --user-data centos-7.3-user-data.txt --wait ceph-rpm-centos7-x86_64-basic-f65754e312f11823c5bb7e9fe24655f3fd68bcd2
2017-02-18T11:07:37.227 DEBUG:teuthology.misc:Can't open 'centos-7.3-user-data.txt': [Errno 2] No such file or directory: 'centos-7.3-user-data.txt'

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: fix make-rpm.sh for kraken + SUSE

tl;dr:

SUSE builds in the OBS do not use the make-dist script to generate the
tarball.  This commit overrides a SUSE downstream patch that was breaking
make-dist.

Full analysis:

make-rpm.sh breaks for kraken when run on SLE-12-SP2:

+ rpmbuild -ba --define '_unpackaged_files_terminate_build 0' --define '_topdir /tmp/release/sles/WORKDIR' ceph.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.F2hJfx
+ umask 022
+ cd /tmp/release/sles/WORKDIR/BUILD
+ cd /tmp/release/sles/WORKDIR/BUILD
+ rm -rf ceph-11.1.0
+ /usr/bin/bzip2 -dc /tmp/release/sles/WORKDIR/SOURCES/ceph-11.1.0-7423-g8f37fcc.tar.bz2
+ /bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd ceph-11.1.0
/var/tmp/rpm-tmp.F2hJfx: line 35: cd: ceph-11.1.0: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.F2hJfx (%prep)

The buildpackages task uses "make-dist" to generate the tarball. In order for
the tarball so generated to be buildable by rpmbuild, the spec file must
contain the following lines:

Version:        @VERSION@
Release:        @RPM_RELEASE@%{?dist}
...
%prep
%autosetup -p1 -n @TARBALL_BASENAME@

SUSE builds in OBS/IBS use a modified spec file and uses a different mechanism
to generate the tarball. (This worked well up until jewel but for kraken and
beyond we may need to reconsider.) In the SUSE downstream version of the spec
file, the lines look like this:

Version:        @VERSION@
Release:        @RPM_RELEASE@%{?dist}
...
%prep
%if 0%{?rhel}
%autosetup -p1 -n @TARBALL_BASENAME@
%endif
%if ! 0%{?rhel}
%autosetup -p1
%endif

The absence of "-n @TARBALL_BASENAME@" causes the tarball basename to be
"ceph-$VERSION" (e.g. ceph-12.0.0) instead of the full
"ceph-$VERSION-$RPM_RELEASE" (e.g. ceph-12.0.0-3534.g000000), yet rpmbuild
expects the latter because the "Release:" field of the spec file is populated.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: populate version file with RPM version

Until now, the "version" file contained a string like 10.2.5-6244-g9027e75

The RPM version obtained by doing "rpm -q $PKG --qf %{VERSION}-%{RELEASE}"
looks like this: 10.2.5-6244.g9027e75

When teuthology compares these two strings, they don't match even though
the version numbers are effectively the same.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: add user-data for newer SUSE versions

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: apply suse modifications

From https://github.com/SUSE/ceph-qa-suite/tree/wip-opensuse-jewel
because it is known to work. Omit the branch specification.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: import from ceph/master

It is exactly the same in all branches. It really belongs in teuthology
and was placed in ceph-qa-suite because it required a more flexible
development cycle.

Signed-off-by: Loic Dachary <loic@dachary.org>

setup-openstack.sh: populate all subnets

It used to be that OVH only returned a single subnet. Now it is returning two
or three and we need to populate them all.

Fixes: https://bugzilla.suse.com/show_bug.cgi?id=1023263
Signed-off-by: Loic Dachary <loic@dachary.org>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: cloudlab support

The only image available and working there at the moment is 14.04.1
which require a few tweaks to be setup properly.

https://groups.google.com/forum/#!topic/cloudlab-users/EN6YASWXwZU

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: setup ceph-workbench credentials

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: install docker in the teuthology instance

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: install ceph-workbench

Signed-off-by: Loic Dachary <loic@dachary.org>
Conflicts:
teuthology/openstack/__init__.py

Merge pull request #1327 from kshtsk/misc-drop-stringio-import

misc: drop StringIO import to fix flake8

Reviewed-by: Nathan Cutler <ncutler@suse.com>

misc: drop StringIO import to fix flake8

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1318 from kshtsk/wip-misc-use-remote-sh

misc: use remote.sh instead of remote.run

Merge PR #1322 into master

* refs/pull/1322/head:
add note that ceph.conf.template should no longer be modified

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1323 from kshtsk/wip-misc-drop-legacy-code-in-skeleton_config

misc: drop testdir replacement in skeleton_config

Merge pull request #1325 from rjfd/wip-fix-import-loop

lock/ops: remove import to break import loop

lock/ops: remove import to break import loop

Fixes: https://tracker.ceph.com/issues/42256
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #1324 from badone/wip-remove-package-overrides

install/rpm: Remove _package_overrides

Reviewed-by: Kefu Chai <kchai@redhat.com>

install/rpm: Remove _package_overrides

The _package_overrides function is currently not required.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

misc: drop testdir replacement in skeleton_config

The str.format is not used anymore for ceph.conf.template

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1321 from kshtsk/wip-ceph-cm-ansible-url

config: add option for ceph cm ansible git url

Merge pull request #1312 from ryg-/wip-42081-set-os

lock: update target os-type and os-version after provisioning

add note that ceph.conf.template should no longer be modified

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

config: add option for ceph cm ansible git url

This is to be able to supply a custom location without changing
the ceph git base url.

Signed-off-by: Roi Dayan <roid@mellanox.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/config.py
teuthology/suite.py
teuthology/test/test_suite.py

Merge pull request #1320 from kshtsk/wip-fog-python3

fog: get rid of StringIO usage for python3 portability

fog: get rid of StringIO usage for python3 portability

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1317 from kshtsk/wip-python3-use-isinstance

suite/matrix: use isinstance for python3 compatibility

misc: use remote.sh instead of remote.run

To ease porting and compatibility in order to avoid
usage of cStringIO.StringIO it is better to use
remote.sh method which supposed to return captured
value as a string.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

run_tasks: use isinstance() instead of type()

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology/exit: use isinstance for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: use isinstance for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock: remove os_version and os_type for node before reprovisiong

os_version and os_type don't reflect actual os version
when node is under provisioning and/or provisiong failed.
The fix cleanup fileds before provisioning.

Signed-off-by: Roman Grigoryev <roman.grigorev@suse.de>

lock: update target os-type and os-version after provisioning

After provisioning node via "teuthology-lock" (via both
"--lock" and "--lock-many") users cannot see actual
versioni and type  of provisioned OS in pulpito.
It is pretty helpful to see that version if we need to
do some manual work.

Signed-off-by: Roman Grigoryev <roman.grigorev@suse.de>

Merge pull request #1315 from kshtsk/wip-future-print

use python3 compatible print

Merge pull request #1307 from kshtsk/beanstalkc3

beanstalk: switch to py3 compatible package

test_build_matrix: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

provision/cloud: use python3 compatible syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

parallel: use python3 compatible syntax for print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

packaging: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

contextutil: python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: fix print syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/cli: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

schedule: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ls: use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

results: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: fix tab inconstent indents

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

beanstalk: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

kill: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fix tab inconsistent indent

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Use future print for python3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe_tests: use python3 compatible print

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: avoid using print for writing files

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1314 from kshtsk/wip-no-iteritems

Get rid of iteritems and xrange

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

python3: get rid of iteritems

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ansible: use py3 compatible octals

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: get rid of xrange

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1309 from jtlayton/fixes

Minor bugfixes for teuthology

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1313 from kshtsk/wip-open-instead-file

python3: use open() function instead file() class

python3: use open() function instead file() class

Since python3 does not have file() we should stop using it.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1310 from kshtsk/wip-all-ipv4-octets-in-target-names

openstack: encode instance name with the full IP

Remove BaseException.message deprecation warning

DeprecationWarning: BaseException.message has been deprecated as of Python 2.6

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Use yaml.safe_load to suppress deprecation warning

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: encode instance name with the full IP

Using only the last two numbers is problematic when mixing multiple
OpenStack clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/provision.py

Merge pull request #847 from kmroz/wip-docs

docs: add SUSE to list of bootstrapable operating systems

Merge pull request #1092 from gohighsec/master

docs: use kernel version distro instead of jewel in the example of ru…

describe_tests: skip over hidden directories

The ceph/qa directory has a bunch of .qa symlinks under it that cause
teuthology-describe-tests to hit symlink loops. Just skip any dentry
with a name that starts with '.'.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

bootstrap: force usage of python2 in virtualenv command

None of this code seems to be python3 ready, so force the virtualenv
command in the bootstrap script to use python2 on a box where both are
available.

Signed-off-by: Jeff Layton <jlayton@kernel.org>

Merge pull request #1302 from rishabh-d-dave/rewrite-remote-mktemp

orchestra/remote: rewrite mktemp to use coreutils mktemp

Merge pull request #1303 from rhcs-dashboard/wip-minimal-reqs

setup.py: move mysqlclient/coverage to optional deps

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

beanstalk: switch to py3 compatible package

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1300 from smithfarm/wip-install-no-purge

install: drop purge_data and related code

install: drop purge_data and related code

purge_data was used to remove any Ceph daemon metadata remaining after the
teardown phase of the install task runs. In particular, the ceph task was
known to leave OSD metadata behind in the OSD directories under /var/lib/ceph.

While the install task causes the ceph-osd package to be installed and creates
directories under /var/lib/ceph, it is the ceph task that actually
populates these directories. Thus, their "unpopulation" should be
implemented in the ceph task, and not here in the install task.

(If the directories were empty after the ceph task teardown phase, they would be
removed when the install task removes the ceph-osd package from the system. But
the package management tool refuses to remove them when they contain files not
owned by a package.)

Not running purge_data during the teardown phase of the install task makes it
easier to implement alternative deployment tasks (i.e. tests that deploy Ceph
clusters, but do not call ceph.py).

In the final analysis, however, this code is no longer needed now that all test
nodes (virtual and bare metal) are reimaged between runs.

This commit does not move purge_data into the ceph task because the author felt
that the ceph task should take more care to clean up after itself. For example,
nowadays there are ceph CLI commands for removing OSDs.

Aside: this commit also drops two lines of dead code that flake8 was
complaining about.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1305 from kshtsk/wip-safe-load

suite/run: use yaml.safe_load

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suite/run: use yaml.safe_load

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1304 from batrick/selinux-smartd

selinux: ignore smartd

Reviewed-by: Kefu Chai <kchai@redhat.com>

selinux: ignore smartd

The failures [1]:

    2019-08-22T22:11:34.747 INFO:teuthology.orchestra.run.smithi195:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\)'
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7592): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566509255.666:7593): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { read } for  pid=8598 comm="smartd" name="nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.810 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8008): avc:  denied  { open } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1
    2019-08-22T22:11:34.811 INFO:teuthology.orchestra.run.smithi195.stdout:type=AVC msg=audit(1566511059.828:8009): avc:  denied  { ioctl } for  pid=8598 comm="smartd" path="/dev/nvme0" dev="devtmpfs" ino=13028 ioctlcmd=4e41 scontext=system_u:system_r:fsdaemon_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=chr_file permissive=1

[1] /ceph/teuthology-archive/pdonnell-2019-08-22_20:06:33-fs-wip-pdonnell-testing-20190822.163036-distro-basic-smithi/4242071/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.py: move myqslclient to optional deps

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

orchestra/remote: rewrite mktemp to use coreutils mktemp

Use mktemp instead of Python's tempfile.mktemp since coreutils is more
easily available.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1301 from rishabh-d-dave/remote-add-mkdtemp

orchestra/remote: add a method to create temporary directory

orchestra/remote: add a method to create temporary directory

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1299 from kshtsk/wip-kernel-none

suite/run: add support 'none' for kernel branch parameter

suite/run: add support 'none' for kernel branch parameter

This can be useful when one wants to disable kernel hash

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1282 from kshtsk/wip-suse-disable-ansible

initial_tasks: make it possible to disable ansible

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #1298 into master

* refs/pull/1298/head:
teuthology/orchestra/daemon: fix the log type
teuthology/orchestra/daemon: Check if stdin is valid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/orchestra/daemon: fix the log type

Signed-off-by: Jos Collin <jcollin@redhat.com>

teuthology/orchestra/daemon: Check if stdin is valid

As there is self.proc.stdin.close() in stop(), there are chances
of write failing with IOError("File is closed").

Signed-off-by: Jos Collin <jcollin@redhat.com>