openstack: enable testing on openSUSE Leap 15.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: support openSUSE Leap 15.0

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1275 from ceph/wip-default-os

update default OS versions

update default OS versions

centos/rhel 7.4 -> 7.6
ubuntu 16.04 -> 18.04

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #1284 into master

* refs/pull/1284/head:
selinux: ignore updatedb failures

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

selinux: ignore updatedb failures

This is with RHEL 7.6.

The failure [1]:

    2019-04-17T21:06:33.388 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi088.front.sepia.ceph.com: ['type=AVC msg=audit(1555531592.378:4451): avc:  denied  { rename } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { read write open } for  pid=10476 comm="updatedb" path="/var/lib/mlocate/mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { create } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { add_name } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1555531591.622:4447): avc:  denied  { write } for  pid=10476 comm="updatedb" path="/var/lib/mlocate/mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531592.377:4450): avc:  denied  { setattr } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { write } for  pid=10476 comm="updatedb" name="mlocate" dev="sda1" ino=76978 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1']

[1] /ceph/teuthology-archive/pdonnell-2019-04-17_06:02:40-fs-wip-pdonnell-testing-20190417.032809-distro-basic-smithi/3856822/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1278 from kshtsk/wip-log-userdata

cloud/openstack: log userdata for debug purpose

cloud/openstack: log userdata for debug purpose

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1276 from tchaikov/wip-python34-to-python36

task/install: normalize python3 packages to python36 ones

Reviewed-by: Boris Ranto <branto@redhat.com>

task/install: normalize python3 packages to python36 ones

EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1271 from kshtsk/wip-openstack-arch

add openstack arch

Merge pull request #1270 from kshtsk/wip-allow_sizes_and_networks

libcloud: add allow_sizes and allow_networks

Merge pull request #1272 from kshtsk/wip-ssh-interface

add ssh_interface property

openstack: enable openSUSE Leap 42.2

Fixes: #29
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: drop some obsolete images

Centos <7.2 is obsolete, Ubuntu <14 is obsolete

(Note that the openstack-integration tests download all of these images
one-by-one, so this is a time-saving measure.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: update Debian 8.0 image URL to current

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: enable CentOS 7.3

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: add Leap 42.2 cloud image URL comment

Just a commented-out URL for now, until we figure out how to handle .tbz
extension

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: debian now is 8.7.0

Signed-off-by: Loic Dachary <ldachary@redhat.com>

openstack: use 16.04 for teuthology VM

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: debian jessie is now 8.5.0

Signed-off-by: Loic Dachary <loic@dachary.org>

add ubuntu 16.04 xenial support

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: update flavor tests

With the latest flavor list from OVH

Signed-off-by: Loic Dachary <ldachary@redhat.com>

openstack: test must preserve the environment

Signed-off-by: Loic Dachary <ldachary@redhat.com>

openstack: ovh renamed flavors : s/eg/hg/ (part 3)

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: ovh renamed flavors : s/eg/hg/ (part 2)

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: ovh renamed flavors : s/eg/hg/

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add get_provider() to match set_provider()

So that set_provider does not need to be called explicitly before using
the .provider data member.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add aarch64 support

Signed-off-by: Loic Dachary <loic@dachary.org>

libcloud: add allow_sizes and allow_networks

In order to simplify selection of flavors and networks
there is added support for allow_sizes and allow_networks
records for each provider configuration.

Both supports either single regular expression or a list
of regular expressions.
Also exclude_sizes supports lists now as well.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

cloud/openstack: add ntp dependency for dummy

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

cloud/openstack: add ssh_interface property

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1268 from kshtsk/wip-default-suite-branch

teuthology-suite: no default value for suite branch

teuthology-suite: no default value for suite branch

By default suite branch must be equal to ceph one.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1267 from dmick/master

task/kernel: fix Depends: parsing for dependencies with "|"

task/kernel: fix Depends: parsing for dependencies with "|"

also make the "Checking kernel version...want" logmsg include quotes
for spotting problems more quickly

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1266 from kshtsk/wip-test-repo-from-file

Read test repo from file

teuthology-openstack: strip out unicode tags for interim test repo yaml

When json file loads from file for --test-repo it is converted
to u'' instead of bare str. Later yaml.dump includes unicode
tags in resulting yaml which makes teuthology-suite fail and
throw an exception.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

teuthology-openstack: update help for --test-repo argument

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

teuthology-openstack: support test repo read from file

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Add optional priority for --test-repo

Priority can be set adding optional '!' symbol with a number between
name and url. Internally there will be a 'priority' field added
next to 'name' and 'url' fields to the yaml definition.

Usage examples:
    --test-repo 'first-repo:http://hostname.domain/no-priority/repo'
    --test-repo 'second-repo!99:http://hostname.domain/default-priority/repo'
    --test-repo 'third-repo!1:http://hostname.domain/highest-priority/repo'
    --test-repo 'fourth-repo!2:http://hostname.domain/next-priority/repo'
    --test-repo 'fifth-repo!0:http://hostname.domain/zero-priority/repo'

Notes for zypper:
    The higher the number, the lower the priority.
    No priority is equivalent to 99 or 0 priorities.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1263 from kshtsk/wip-os-getenv

openstack: take defaults from environment

openstack: take defaults from environment

This patch introduces TEUTH_ based environment variables
which can be used to avoid bulky commands while overiding
default values for some of the teuthology arguments.
Removes mandatory of --ceph option for teuthology-suite,
and defaults to 'master'.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1265 from batrick/selinux-nagios

selinux: ignore nagios denial

Merge pull request #1264 from tchaikov/yaml-loads

ansible.py: catch yaml.YAMLError when yaml.safe_load()

Reviewed-by: David Zafman <dzafman@redhat.com>

selinux: ignore nagios denial

Got this in RHEL 7.6 testing.

Fixes: https://tracker.ceph.com/issues/38519
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ansible.py: catch yaml.YAMLError when yaml.safe_load()

yaml.safe_load() also throws yaml.scanner.ScannerError. so use
yaml.YAMLError to cover them all. see
https://pyyaml.org/wiki/PyYAMLDocumentation .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1259 from kshtsk/wip-mysqlclient

python-cov: update python mysql dependency

Merge pull request #1258 from kshtsk/wip-no-nuke

openstack: fix all volumes nuke

openstack: fix all volumes nuke

Do not allow teuthology-nuke to delete all volumes on openstack.

There is a bug on some openstack clients that --name option
does not work correctly for 'openstack volume list' in contrast
to 'openstack server list' and lists all available volumes
ignoring '--name' option at all.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1255 from kshtsk/wip-reconnect-timeout

orchestra: fix retry and socket timeout misconception

python-cov: update python mysql dependency

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

Merge pull request #1257 from kshtsk/fix-test-repo-utils

test_repo_utils: fix shared temporary directory

test_repo_utils: fix shared temporary directory

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

orchestra: fix retry and socket timeouts misconception

Fix retry and socket timeouts misconception for reconnect.
Add default sleep time as argument to the signature.

Signed-off-by: Kyr <kyrylo.shatskyy@gmail.com>

Merge pull request #1254 from kshtsk/wip-dump-console-log

Dump console log if ssh_keyscan failed

Merge pull request #1253 from ceph/wip-valgrind-whitelist

valgrind.supp: whitelist something in glibc

openstack: dump console log if ssh_keyscan failed

Dump server's console log if ssh_keyscan failed

Signed-off-by: Kyr <kyrylo.shatskyy@gmail.com>

misc: ssh_keyscan_wait should use _ssh_keyscan

After ssh_keyscan refactoring ssh_keyscan_wait should be using
the _ssh_keyscan instead.

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

misc: ssh_keyscan_wait should not raise any exceptions

The ssh_keyscan_wait raises exception MaxWhileTries while
it should return False.

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

Merge pull request #1256 from kshtsk/fix-f632

Fix flake8 pytest F632 errors

Fix flake8 pytest F632 errors

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

valgrind.supp: whitelist something in glibc

  <kind>Leak_PossiblyLost</kind>
  <xwhat>
    <text>7 bytes in 1 blocks are possibly lost in loss record 20 of 33,344</text>
    <leakedbytes>7</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA811C23</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>malloc</fn>
      <dir>/builddir/build/BUILD/valgrind-3.13.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>299</line>
    </frame>
    <frame>
      <ip>0xE709809</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>strdup</fn>
    </frame>
    <frame>
      <ip>0xFC49444</ip>
      <obj>/usr/lib64/libnl-3.so.200.23.0</obj>
      <fn>__trans_list_add</fn>
    </frame>
    <frame>
      <ip>0xF9E07E0</ip>
      <obj>/usr/lib64/libnl-route-3.so.200.23.0</obj>
    </frame>
    <frame>
      <ip>0x9BF7B02</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
      <fn>_dl_init</fn>
    </frame>
    <frame>
      <ip>0x9BE9069</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
    </frame>
    <frame>
      <ip>0x5</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CA2</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CAB</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CAE</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CB8</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CBD</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CC0</ip>
    </frame>
  </stack>
</error>

On centos 7.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1250 from tchaikov/wip-sphinx-reformat

task: reformat the docstring for sphinx

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1251 from kshtsk/wip-matrix-tostr

Fix matrix dump tostr() method

Fix matrix dump tostr() method

Addresses issue while trying to dump matrix object:

  AttributeError: Base instance has no attribute '__getitem__'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

task: reformat the docstring for sphinx

the code block should start with `::`, and should be indented.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1247 from kshtsk/wip-allow-runtime-logging-for-sh

orchestra: allow runtime logging for sh

orchestra: allow runtime logging for sh

When run.run is used with 'stdout' argument, the output of run command
is read to the given stream and logged with corresponding level only after
completion, which is different from the default behaviour. This 'hides'
useful information when, for example, the command is hung or stuck.
Aso it is usually handy when there are some data can be found
in the log for a prolonged command execution during runtime.

This patch addresses the issue.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1246 from tchaikov/wip-downgrade

task/install: assert on installed_version > downgrade_version

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: assert on installed_version > downgrade_version

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1245 from tchaikov/wip-downgrade-packages

task/install: populate "downgrade_packages" to nested_config

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: populate "downgrade_packages" to nested_config

- populate "downgrade_packages" to `nested_config` in `task()`.
  otherwise, the "downgrade_packages" is invisible to the `install()`
  function.
- also print logging message before downgrading packages.
- cleanup rpm._downgrade_packages(), no need to format `downgrade_cmd`.
  it's already formated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1244 from tchaikov/wip-install-downgrade-packages

task/install: add "downgrade_packages" option for "install" task

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: add "downgrade_packages" option for "install" task

in rados/thrash-old-clients, hammer or jewel packages is installed. but
yum does not allow downgrade a package by default, if a newer version is
already installed. in this case, librbd1 and librados2 are installed as
dependencies of qemu-kvm. their version is 1:10.2.5-4.el7 at the time of
writing in CentOS/RHEL 7.5. so if we want to install librbd1 or
librados2 from jewel, yum will simply consider the requirement is
already fulfilled and hence do nothing. if we want to install
ceph-radosgw from jewel, yum will fail, as ceph-radosgw depends on
librados2 and other Ceph packages of the same version from jewel. but
librbd1 and librados2 have been already installed. the error message
looks like:

Error: Package: 1:ceph-common-0.94.10-87.g116a558.el7.x86_64 (Ceph)
           Requires: librados2 = 1:0.94.10-87.g116a558.el7
           Installed: 1:librados2-10.2.5-4.el7.x86_64 (@base)
               librados2 = 1:10.2.5-4.el7
           Available: 1:librados2-0.94.10-87.g116a558.el7.x86_64 (Ceph)
               librados2 = 1:0.94.10-87.g116a558.el7

so we need to downgrade librbd1 and librados2 first.

in this change, "downgrade_package" option is added for "install" task,
so we can specify packages to be downgraded to given version, these
packages won't be installed twice if they are also specified by
"install" task elsewhere to be installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

task/install: define "project" before using it

no need to define `project` at the beginning of install(), we can
just use `config.get('project', 'ceph')`. better readability this way,
as we will reference `ceph` in the same expression.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1243 from tchaikov/wip-install-pkg-order

task/install: keep the order of pkgs to be installed

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

task/install: keep the order of pkgs to be installed

before this change, the package list to be installed are ordered by
the hash function used by set().

after this change, the order of packages are not changed. so we can
ensure the order of package installation is the identical to that
of package list in yaml.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1241 from ceph/wip-job-id

Remove unused 'machine_type' parameter from kill_job()

Merge pull request #1194 from adamwg/no-validate-refs

Add no-validate-refs option to teuthology-suite

Merge pull request #1239 from ceph/wip-addrvec

misc: optionally bind to v1+v2 addrs for mons

Merge pull request #1237 from ceph/wip-rhel8

opsys: Add RHEL8

Merge pull request #1238 from ceph/wip-instrument-import

run_tasks.py: log the right ImportError message

Remove unused 'machine_type' parameter from kill_job()

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

misc: optionally bind to v1+v2 addrs for mons

Signed-off-by: Sage Weil <sage@redhat.com>

run_tasks.py: allow _import to raise the right ImportError

It turns out it's possible for a file qa/tasks/foo.py to exist,
yet importing it still raises an ImportError because it references a
non-existent symbol.

In this case, teuthology was clobbering the real ImportError with its
own bogus text.

Fixes: http://tracker.ceph.com/issues/37370
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Add validate-sha1 option to teuthology-suite

Allow users to skip validation of SHA1s passed to -S when submitting
jobs via teuthology-suite. This is useful when testing against
non-GitHub repos, since teuthology can't validate refs for a plain git
repository.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

Merge PR #1234 into master

* refs/pull/1234/head:
only pass system errors to hub handler
small refactor of bash code

Reviewed-by: Kefu Chai <kchai@redhat.com>

opsys: Add RHEL8

Signed-off-by: David Galloway <dgallowa@redhat.com>

only pass system errors to hub handler

Fixes: http://tracker.ceph.com/issues/36731
Code suggested/contributed by Kefu.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

small refactor of bash code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1236 from ceph/wip-netaddr

ceph_ansible: Install netaddr

ceph_ansible: Install netaddr

See https://github.com/ceph/ceph-ansible/pull/3199

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1230 from tchaikov/wip-better-remote-run

@zmc @smithfarm gents, i've updated the change to address the concerns. if no objections, i'd like merge as it is in two days.

Merge pull request #1235 from ceph/wip-suppress-cpython

valgind: ignore all leaks relating to CPython code

valgind: ignore all leaks relating to CPython code

Yes, this is a big hammer, and we are ignoring a lot.  However, it is a
HUGE step forward to what we do now, which is not check for ceph-mgr
leaks at all.

By adding this suppress I found and fixed 3 separate ceph-mgr leaks.  This
will let us prevent others (in non-Py code) from being introduced.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1231 from kshtsk/wip-shortname-logging-for-ip

Fix ipv4 and ipv6 address logging for Remote.run

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1233 from kshtsk/wip-remote-sh

orchestra: add remote.sh commands analogous to misc.sh

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

orchestra: add remote.sh commands analogous to misc.sh

Adds a remote.sh similar to misc.sh, in fact a shortcut for remote.run,
but return output instead of proc
Example:
    my_name = Remote('127.0.0.1').sh('whoami')

Adds a remote.sh_file run a script as file on a remote with or without sudo
Example 1: Run python script
    Remote('127.0.0.1').sh_file("#!/usr/bin/env python3\n"
                                "import sys\n"
                                "print(sys.version_info)")
Example 2: Run script as root
    Remote('user@host.domain').sh_file("whoami", sudo=True,
                                       label="who-am-i-for-the-real")
Example 3: Run script as other user
    Remote('user@host.domain').sh_file("whoami", sudo='nobody', )

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #1228 from zenglg/master

Fix setenforce failed while selinux is disabled