libcloud: add test for userdata read from config

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

libcloud: read userdata from config

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1288 from kshtsk/suse-chrony

task/clock: add chrony support

task/clock: add chrony support

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1290 from kshtsk/fix-tox-mock

tox: set mock version to 2.0.0

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test/test_exit: set mock version to 2.0.0

After upgrade mock from 2.0.0 to 3.0.3 the TestExiter started failing
with incorrect call_count.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1279 from kshtsk/wip-suse-fixes-1

suse related fixes

Merge pull request #1280 from kshtsk/wip-suse-fix-reconnect

fix reconnect

Merge pull request #1286 from ceph/wip-yuval-add-pubsub-deps

rgw/pubsub: add dependencies needed for pubsub tests

Merge pull request #1274 from ceph/wip-move-valgrind_supp

valgrind.supp: move the whitelist file to /qa dir of ceph.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1285 from gekios/wip-ceph-dev-env-fix

openstack: always pass branch and repo parameters

openstack: always pass branch and repo parameters

We need to always pass the parameters below in order to make it work using
environment variables:
--ceph
--ceph-repo
--suite-repo
--suite-branch
--teuthology-branch

Signed-off-by: gkyratsas <gkyratsas@suse.com>
Conflicts:
teuthology/openstack/__init__.py

test/opsys add opensuse 15.1 case

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

syslog.py: whitelist innocent tcmu-runner log message

When running the new iSCSI smoke test, tcmu-runner emits a syslog
message that was not whitelisted:

2019-03-27T10:00:17.292154+00:00 target192168000056 tcmu-runner[37366]: 2019-03-27 10:00:17.291 37366 [INFO] load_our_module:537: Inserted module 'target_core_user'

Since the message is at severity level INFO, it was causing teuthology
to raise a red flag.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

task: internal/syslog: expand salt exclusion

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/task/internal/syslog.py

task: syslog: salt INFO messages generate false negatives

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/task/internal/syslog.py

opsys: Drop opensuse-leap from DISTRO_CODENAME_MAP

Drop 'opensuse-leap' from DISTRO_CODENAME_MAP in favor of 'opensuse'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Add new openSUSE Leap ID for os-release

misc: add check to delete_file

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

misc: get_system_type: openSUSE means rpm

2018-03-03T23:59:57.524 INFO:teuthology.orchestra.run.target149202163177:Running: 'sudo lsb_release -is'
2018-03-03T23:59:57.600 INFO:teuthology.orchestra.run.target149202163177.stdout:openSUSE
2018-03-03T23:59:57.601 DEBUG:teuthology.misc:System to be installed: openSUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

misc.py: Handle SLE-12-SP2 in get_system_type()

In SLE-12-SP1, "lsb_release -is" returns "SUSE LINUX". In SLE-12-SP2, it
returns "SUSE".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

opsys: opensuse id related fix

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

misc.py: saner write_file() and sudo_write_file()

Using python to do this might have made sense in 2011, but it doesn't now.

Our hand is forced here, because systems are no longer guaranteed to
have an executable called "python".

For background details on why this is happening, see
https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3#.2Fusr.2Fbin.2Fpython

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: enable testing on openSUSE Leap 15.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: support openSUSE Leap 15.0

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1275 from ceph/wip-default-os

update default OS versions

update default OS versions

centos/rhel 7.4 -> 7.6
ubuntu 16.04 -> 18.04

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind.supp: move the whitelist file to /qa dir of ceph.git

This change drops valgrind.supp whitelist file from teuthology
repo and alters the install task to acquire it from /qa dir of
ceph-qa-suite repository.
It's expected the commit will have sibiling in ceph.git moving
the proper valgrind.supp there.

The rationale behind the entire change is duplication of these
files between two repos, their inconsistencies and confusion.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #1284 into master

* refs/pull/1284/head:
selinux: ignore updatedb failures

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

selinux: ignore updatedb failures

This is with RHEL 7.6.

The failure [1]:

    2019-04-17T21:06:33.388 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi088.front.sepia.ceph.com: ['type=AVC msg=audit(1555531592.378:4451): avc:  denied  { rename } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { read write open } for  pid=10476 comm="updatedb" path="/var/lib/mlocate/mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { create } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { add_name } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1555531591.622:4447): avc:  denied  { write } for  pid=10476 comm="updatedb" path="/var/lib/mlocate/mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531592.377:4450): avc:  denied  { setattr } for  pid=10476 comm="updatedb" name="mlocate.db.eDrUCz" dev="sda1" ino=5477 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1555531589.705:4446): avc:  denied  { write } for  pid=10476 comm="updatedb" name="mlocate" dev="sda1" ino=76978 scontext=system_u:system_r:locate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=1']

[1] /ceph/teuthology-archive/pdonnell-2019-04-17_06:02:40-fs-wip-pdonnell-testing-20190417.032809-distro-basic-smithi/3856822/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1278 from kshtsk/wip-log-userdata

cloud/openstack: log userdata for debug purpose

orchestra: fix no connection after reboot

This resolves the issue when remote lost connection
if the command was a reboot of the host and there was
only one unsuccessful try to reconnect and as a result
was loosing transport and got stuck with message like:

WARNING:tasks.deepsea.orch:No connection established yet..

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

Fix code to allow reconncting to a node after the connection was lost.

This patch introduces a 'context' parameter for
teuthology.orchestra.remote's run(..) method which
works around an issue with paramiko.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

cloud/openstack: log userdata for debug purpose

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1276 from tchaikov/wip-python34-to-python36

task/install: normalize python3 packages to python36 ones

Reviewed-by: Boris Ranto <branto@redhat.com>

task/install: normalize python3 packages to python36 ones

EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/pubsub: add dependencies needed for pubsub tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #1271 from kshtsk/wip-openstack-arch

add openstack arch

Merge pull request #1270 from kshtsk/wip-allow_sizes_and_networks

libcloud: add allow_sizes and allow_networks

Merge pull request #1272 from kshtsk/wip-ssh-interface

add ssh_interface property

openstack: enable openSUSE Leap 42.2

Fixes: #29
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: drop some obsolete images

Centos <7.2 is obsolete, Ubuntu <14 is obsolete

(Note that the openstack-integration tests download all of these images
one-by-one, so this is a time-saving measure.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: update Debian 8.0 image URL to current

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: enable CentOS 7.3

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: add Leap 42.2 cloud image URL comment

Just a commented-out URL for now, until we figure out how to handle .tbz
extension

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: debian now is 8.7.0

Signed-off-by: Loic Dachary <ldachary@redhat.com>

openstack: use 16.04 for teuthology VM

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: debian jessie is now 8.5.0

Signed-off-by: Loic Dachary <loic@dachary.org>

add ubuntu 16.04 xenial support

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: update flavor tests

With the latest flavor list from OVH

Signed-off-by: Loic Dachary <ldachary@redhat.com>

openstack: test must preserve the environment

Signed-off-by: Loic Dachary <ldachary@redhat.com>

openstack: ovh renamed flavors : s/eg/hg/ (part 3)

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: ovh renamed flavors : s/eg/hg/ (part 2)

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: ovh renamed flavors : s/eg/hg/

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add get_provider() to match set_provider()

So that set_provider does not need to be called explicitly before using
the .provider data member.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: add aarch64 support

Signed-off-by: Loic Dachary <loic@dachary.org>

libcloud: add allow_sizes and allow_networks

In order to simplify selection of flavors and networks
there is added support for allow_sizes and allow_networks
records for each provider configuration.

Both supports either single regular expression or a list
of regular expressions.
Also exclude_sizes supports lists now as well.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

cloud/openstack: add ntp dependency for dummy

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

cloud/openstack: add ssh_interface property

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1268 from kshtsk/wip-default-suite-branch

teuthology-suite: no default value for suite branch

teuthology-suite: no default value for suite branch

By default suite branch must be equal to ceph one.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1267 from dmick/master

task/kernel: fix Depends: parsing for dependencies with "|"

task/kernel: fix Depends: parsing for dependencies with "|"

also make the "Checking kernel version...want" logmsg include quotes
for spotting problems more quickly

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1266 from kshtsk/wip-test-repo-from-file

Read test repo from file

teuthology-openstack: strip out unicode tags for interim test repo yaml

When json file loads from file for --test-repo it is converted
to u'' instead of bare str. Later yaml.dump includes unicode
tags in resulting yaml which makes teuthology-suite fail and
throw an exception.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

teuthology-openstack: update help for --test-repo argument

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

teuthology-openstack: support test repo read from file

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Add optional priority for --test-repo

Priority can be set adding optional '!' symbol with a number between
name and url. Internally there will be a 'priority' field added
next to 'name' and 'url' fields to the yaml definition.

Usage examples:
    --test-repo 'first-repo:http://hostname.domain/no-priority/repo'
    --test-repo 'second-repo!99:http://hostname.domain/default-priority/repo'
    --test-repo 'third-repo!1:http://hostname.domain/highest-priority/repo'
    --test-repo 'fourth-repo!2:http://hostname.domain/next-priority/repo'
    --test-repo 'fifth-repo!0:http://hostname.domain/zero-priority/repo'

Notes for zypper:
    The higher the number, the lower the priority.
    No priority is equivalent to 99 or 0 priorities.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1263 from kshtsk/wip-os-getenv

openstack: take defaults from environment

openstack: take defaults from environment

This patch introduces TEUTH_ based environment variables
which can be used to avoid bulky commands while overiding
default values for some of the teuthology arguments.
Removes mandatory of --ceph option for teuthology-suite,
and defaults to 'master'.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1265 from batrick/selinux-nagios

selinux: ignore nagios denial

Merge pull request #1264 from tchaikov/yaml-loads

ansible.py: catch yaml.YAMLError when yaml.safe_load()

Reviewed-by: David Zafman <dzafman@redhat.com>

selinux: ignore nagios denial

Got this in RHEL 7.6 testing.

Fixes: https://tracker.ceph.com/issues/38519
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ansible.py: catch yaml.YAMLError when yaml.safe_load()

yaml.safe_load() also throws yaml.scanner.ScannerError. so use
yaml.YAMLError to cover them all. see
https://pyyaml.org/wiki/PyYAMLDocumentation .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1259 from kshtsk/wip-mysqlclient

python-cov: update python mysql dependency

Merge pull request #1258 from kshtsk/wip-no-nuke

openstack: fix all volumes nuke

openstack: fix all volumes nuke

Do not allow teuthology-nuke to delete all volumes on openstack.

There is a bug on some openstack clients that --name option
does not work correctly for 'openstack volume list' in contrast
to 'openstack server list' and lists all available volumes
ignoring '--name' option at all.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1255 from kshtsk/wip-reconnect-timeout

orchestra: fix retry and socket timeout misconception

python-cov: update python mysql dependency

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

Merge pull request #1257 from kshtsk/fix-test-repo-utils

test_repo_utils: fix shared temporary directory

test_repo_utils: fix shared temporary directory

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

orchestra: fix retry and socket timeouts misconception

Fix retry and socket timeouts misconception for reconnect.
Add default sleep time as argument to the signature.

Signed-off-by: Kyr <kyrylo.shatskyy@gmail.com>

Merge pull request #1254 from kshtsk/wip-dump-console-log

Dump console log if ssh_keyscan failed

Merge pull request #1253 from ceph/wip-valgrind-whitelist

valgrind.supp: whitelist something in glibc

openstack: dump console log if ssh_keyscan failed

Dump server's console log if ssh_keyscan failed

Signed-off-by: Kyr <kyrylo.shatskyy@gmail.com>

misc: ssh_keyscan_wait should use _ssh_keyscan

After ssh_keyscan refactoring ssh_keyscan_wait should be using
the _ssh_keyscan instead.

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

misc: ssh_keyscan_wait should not raise any exceptions

The ssh_keyscan_wait raises exception MaxWhileTries while
it should return False.

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

Merge pull request #1256 from kshtsk/fix-f632

Fix flake8 pytest F632 errors

Fix flake8 pytest F632 errors

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

valgrind.supp: whitelist something in glibc

  <kind>Leak_PossiblyLost</kind>
  <xwhat>
    <text>7 bytes in 1 blocks are possibly lost in loss record 20 of 33,344</text>
    <leakedbytes>7</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA811C23</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>malloc</fn>
      <dir>/builddir/build/BUILD/valgrind-3.13.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>299</line>
    </frame>
    <frame>
      <ip>0xE709809</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>strdup</fn>
    </frame>
    <frame>
      <ip>0xFC49444</ip>
      <obj>/usr/lib64/libnl-3.so.200.23.0</obj>
      <fn>__trans_list_add</fn>
    </frame>
    <frame>
      <ip>0xF9E07E0</ip>
      <obj>/usr/lib64/libnl-route-3.so.200.23.0</obj>
    </frame>
    <frame>
      <ip>0x9BF7B02</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
      <fn>_dl_init</fn>
    </frame>
    <frame>
      <ip>0x9BE9069</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
    </frame>
    <frame>
      <ip>0x5</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CA2</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CAB</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CAE</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CB8</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CBD</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CC0</ip>
    </frame>
  </stack>
</error>

On centos 7.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1250 from tchaikov/wip-sphinx-reformat

task: reformat the docstring for sphinx

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1251 from kshtsk/wip-matrix-tostr

Fix matrix dump tostr() method

Fix matrix dump tostr() method

Addresses issue while trying to dump matrix object:

  AttributeError: Base instance has no attribute '__getitem__'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

task: reformat the docstring for sphinx

the code block should start with `::`, and should be indented.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1247 from kshtsk/wip-allow-runtime-logging-for-sh

orchestra: allow runtime logging for sh

orchestra: allow runtime logging for sh

When run.run is used with 'stdout' argument, the output of run command
is read to the given stream and logged with corresponding level only after
completion, which is different from the default behaviour. This 'hides'
useful information when, for example, the command is hung or stuck.
Aso it is usually handy when there are some data can be found
in the log for a prolonged command execution during runtime.

This patch addresses the issue.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1246 from tchaikov/wip-downgrade

task/install: assert on installed_version > downgrade_version

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: assert on installed_version > downgrade_version

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1245 from tchaikov/wip-downgrade-packages

task/install: populate "downgrade_packages" to nested_config

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: populate "downgrade_packages" to nested_config

- populate "downgrade_packages" to `nested_config` in `task()`.
  otherwise, the "downgrade_packages" is invisible to the `install()`
  function.
- also print logging message before downgrading packages.
- cleanup rpm._downgrade_packages(), no need to format `downgrade_cmd`.
  it's already formated.

Signed-off-by: Kefu Chai <kchai@redhat.com>