pep8 corrections and small fix to check if coverage is defined at ctx.config level

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

fix couple flake8 issues

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fixes ceph-ansible timeouts

Start firewalld before starting ceph-ansible task

Fix single cluster role name

Cluster name support removed

ceph-ansible task yaml will use the name 'cluster' in rgw multisite test suite,
purely to differentiate between role names provided to the clusters in the yaml.
But it will not be using the cluster names during configuration.

[ceph/teuthology]: disable gcov flags in ceph-coverage script

Signed-off-by: shylesh kumar <shylesh.mohan@gmail.com>

add new cli teuthology-reimage-fog to reimage FOG provisioned nodes without actually
locking the nodes.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

add a missing comma :(

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

disable firewall run in ceph-ansible playbook

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add client keys on client machines in case we are using ceph-ansible for cluster setup

Signed-off-by: Shylesh Kumar <shmohan@redhat.com>

dont use the stdout when running playbook, this may cause the prints to buffer.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Removed unused variable 'remote'

Updated docstring to describe haproxy roles

Adds haproxy support and other cleanup fixes

check if osd id is an integer, if so use it and discard any empty or non integer ids.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

fix couple more flake8 issues

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add osd_disk_info to ctx so that any task can access it like ctx.osd_disk_info

Signed-off-by: shylesh kumar <shylesh.mohan@gmail.com>

add ansible 2.6 repo

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

fix rh_install unit test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

exec failure fix

Signed-off-by: Shylesh Kumar <shylesh.mohan@gmail.com>

Fix log remote dir with dual clusters

missing nfss key  added in generate hosts file section

Fix compatibility with single cluster run

All fixes included for running dual cluster with ceph-ansible.

Fix unregister during final cleanup, due to stage cdn task that is not needed now.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

need to debug why check_status is being ignored, for now catch
the command failed error, log and ignore so that tests can pass.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

unsubscribe at the start of the test as well, if nodes are not freshly reimaged
then they may have subscription from old configuration

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

typo fix and ignore status if unregister fails

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Enable standard rhel repos as well after stage cdn configuration

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

subscribe rhel octo nodes using stage cdn from configuration

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add misc cleanup options before we unlock the system

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

osd addition fix in roles and cleanup shipped files

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ship_utilities import fix

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ship utilities file during ceph-ansible task

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

detect osd id's and use them in new roles as ansible can change the order
of osd's that are bootstrapped

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix roles after cluster is setup for mon/mgr/mds and also
setup daemons for use by ceph_manager

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fog provides fresh setup so there is no need to do any precleanup

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

fast fix for nvme test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Disable running cm-ansible by default, FOG images have already
run cm-ansible code and it would be redundant to run them again.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

dont poweroff nodes, leave it as is for adhoc debug if needed.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Octo lab has no VPS and its default FOG, so we dont
need any data[os_type] to be sent at all for queue requests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

skip specific os_type/os_version request when FOG reimage in place

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Enable FOG reimage by default for all machine types

the downside: you can't control from teuthology.yaml what
machines can be enabled for FOG

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

run git_ignore_ssl task after ansible setup

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add internal task to ignore ssl errors while
cloning from untrusted gitlab

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add option to schedule jobs using simple subset

Total number of jobs are divided into equal subsets based on
user options and can be scheduled using 1/10, 2/10 etc

If a suite produces 100 jobs, 1/10 would schedule first 10 jobs,
2/10 will schedule next set of 10 jobs and so on.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use update_latest_rh_kernel for redhat configs

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add option to retry for exec task.

retry and sleep_for_retry options can be used with exec task without
needing to use lengthy shell based while loops for similar
task. Also adds timeout option for commands that can be used
to return within specified time.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

when cm-ansible breaks!, backup to skip cm-ansible and run jobs

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

misc run-rh-task fixes during rhcs 3.0

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

debian repo is setup during internal task and no longer needed.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Some more cleanup before repo setup and skip subscription manager
for beta ISO tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1258 from kshtsk/wip-no-nuke

openstack: fix all volumes nuke

openstack: fix all volumes nuke

Do not allow teuthology-nuke to delete all volumes on openstack.

There is a bug on some openstack clients that --name option
does not work correctly for 'openstack volume list' in contrast
to 'openstack server list' and lists all available volumes
ignoring '--name' option at all.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1255 from kshtsk/wip-reconnect-timeout

orchestra: fix retry and socket timeout misconception

Merge pull request #1257 from kshtsk/fix-test-repo-utils

test_repo_utils: fix shared temporary directory

test_repo_utils: fix shared temporary directory

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

orchestra: fix retry and socket timeouts misconception

Fix retry and socket timeouts misconception for reconnect.
Add default sleep time as argument to the signature.

Signed-off-by: Kyr <kyrylo.shatskyy@gmail.com>

Merge pull request #1254 from kshtsk/wip-dump-console-log

Dump console log if ssh_keyscan failed

Merge pull request #1253 from ceph/wip-valgrind-whitelist

valgrind.supp: whitelist something in glibc

openstack: dump console log if ssh_keyscan failed

Dump server's console log if ssh_keyscan failed

Signed-off-by: Kyr <kyrylo.shatskyy@gmail.com>

misc: ssh_keyscan_wait should use _ssh_keyscan

After ssh_keyscan refactoring ssh_keyscan_wait should be using
the _ssh_keyscan instead.

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

misc: ssh_keyscan_wait should not raise any exceptions

The ssh_keyscan_wait raises exception MaxWhileTries while
it should return False.

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

Merge pull request #1256 from kshtsk/fix-f632

Fix flake8 pytest F632 errors

Fix flake8 pytest F632 errors

Signed-off-by: Kyr <kyrylo.shatskyy@suse.com>

valgrind.supp: whitelist something in glibc

  <kind>Leak_PossiblyLost</kind>
  <xwhat>
    <text>7 bytes in 1 blocks are possibly lost in loss record 20 of 33,344</text>
    <leakedbytes>7</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA811C23</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>malloc</fn>
      <dir>/builddir/build/BUILD/valgrind-3.13.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>299</line>
    </frame>
    <frame>
      <ip>0xE709809</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>strdup</fn>
    </frame>
    <frame>
      <ip>0xFC49444</ip>
      <obj>/usr/lib64/libnl-3.so.200.23.0</obj>
      <fn>__trans_list_add</fn>
    </frame>
    <frame>
      <ip>0xF9E07E0</ip>
      <obj>/usr/lib64/libnl-route-3.so.200.23.0</obj>
    </frame>
    <frame>
      <ip>0x9BF7B02</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
      <fn>_dl_init</fn>
    </frame>
    <frame>
      <ip>0x9BE9069</ip>
      <obj>/usr/lib64/ld-2.17.so</obj>
    </frame>
    <frame>
      <ip>0x5</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CA2</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CAB</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CAE</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CB8</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CBD</ip>
    </frame>
    <frame>
      <ip>0x1FFF000CC0</ip>
    </frame>
  </stack>
</error>

On centos 7.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1250 from tchaikov/wip-sphinx-reformat

task: reformat the docstring for sphinx

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1251 from kshtsk/wip-matrix-tostr

Fix matrix dump tostr() method

Fix matrix dump tostr() method

Addresses issue while trying to dump matrix object:

  AttributeError: Base instance has no attribute '__getitem__'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

task: reformat the docstring for sphinx

the code block should start with `::`, and should be indented.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1247 from kshtsk/wip-allow-runtime-logging-for-sh

orchestra: allow runtime logging for sh

orchestra: allow runtime logging for sh

When run.run is used with 'stdout' argument, the output of run command
is read to the given stream and logged with corresponding level only after
completion, which is different from the default behaviour. This 'hides'
useful information when, for example, the command is hung or stuck.
Aso it is usually handy when there are some data can be found
in the log for a prolonged command execution during runtime.

This patch addresses the issue.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1246 from tchaikov/wip-downgrade

task/install: assert on installed_version > downgrade_version

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: assert on installed_version > downgrade_version

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1245 from tchaikov/wip-downgrade-packages

task/install: populate "downgrade_packages" to nested_config

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: populate "downgrade_packages" to nested_config

- populate "downgrade_packages" to `nested_config` in `task()`.
  otherwise, the "downgrade_packages" is invisible to the `install()`
  function.
- also print logging message before downgrading packages.
- cleanup rpm._downgrade_packages(), no need to format `downgrade_cmd`.
  it's already formated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1244 from tchaikov/wip-install-downgrade-packages

task/install: add "downgrade_packages" option for "install" task

Reviewed-by: Neha Ojha <nojha@redhat.com>

task/install: add "downgrade_packages" option for "install" task

in rados/thrash-old-clients, hammer or jewel packages is installed. but
yum does not allow downgrade a package by default, if a newer version is
already installed. in this case, librbd1 and librados2 are installed as
dependencies of qemu-kvm. their version is 1:10.2.5-4.el7 at the time of
writing in CentOS/RHEL 7.5. so if we want to install librbd1 or
librados2 from jewel, yum will simply consider the requirement is
already fulfilled and hence do nothing. if we want to install
ceph-radosgw from jewel, yum will fail, as ceph-radosgw depends on
librados2 and other Ceph packages of the same version from jewel. but
librbd1 and librados2 have been already installed. the error message
looks like:

Error: Package: 1:ceph-common-0.94.10-87.g116a558.el7.x86_64 (Ceph)
           Requires: librados2 = 1:0.94.10-87.g116a558.el7
           Installed: 1:librados2-10.2.5-4.el7.x86_64 (@base)
               librados2 = 1:10.2.5-4.el7
           Available: 1:librados2-0.94.10-87.g116a558.el7.x86_64 (Ceph)
               librados2 = 1:0.94.10-87.g116a558.el7

so we need to downgrade librbd1 and librados2 first.

in this change, "downgrade_package" option is added for "install" task,
so we can specify packages to be downgraded to given version, these
packages won't be installed twice if they are also specified by
"install" task elsewhere to be installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

task/install: define "project" before using it

no need to define `project` at the beginning of install(), we can
just use `config.get('project', 'ceph')`. better readability this way,
as we will reference `ceph` in the same expression.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1243 from tchaikov/wip-install-pkg-order

task/install: keep the order of pkgs to be installed

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

task/install: keep the order of pkgs to be installed

before this change, the package list to be installed are ordered by
the hash function used by set().

after this change, the order of packages are not changed. so we can
ensure the order of package installation is the identical to that
of package list in yaml.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1241 from ceph/wip-job-id

Remove unused 'machine_type' parameter from kill_job()

Merge pull request #1194 from adamwg/no-validate-refs

Add no-validate-refs option to teuthology-suite

Merge pull request #1239 from ceph/wip-addrvec

misc: optionally bind to v1+v2 addrs for mons

Merge pull request #1237 from ceph/wip-rhel8

opsys: Add RHEL8

Merge pull request #1238 from ceph/wip-instrument-import

run_tasks.py: log the right ImportError message

Remove unused 'machine_type' parameter from kill_job()

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

misc: optionally bind to v1+v2 addrs for mons

Signed-off-by: Sage Weil <sage@redhat.com>

run_tasks.py: allow _import to raise the right ImportError

It turns out it's possible for a file qa/tasks/foo.py to exist,
yet importing it still raises an ImportError because it references a
non-existent symbol.

In this case, teuthology was clobbering the real ImportError with its
own bogus text.

Fixes: http://tracker.ceph.com/issues/37370
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Add validate-sha1 option to teuthology-suite

Allow users to skip validation of SHA1s passed to -S when submitting
jobs via teuthology-suite. This is useful when testing against
non-GitHub repos, since teuthology can't validate refs for a plain git
repository.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

Merge PR #1234 into master

* refs/pull/1234/head:
only pass system errors to hub handler
small refactor of bash code

Reviewed-by: Kefu Chai <kchai@redhat.com>

opsys: Add RHEL8

Signed-off-by: David Galloway <dgallowa@redhat.com>

only pass system errors to hub handler

Fixes: http://tracker.ceph.com/issues/36731
Code suggested/contributed by Kefu.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

small refactor of bash code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1236 from ceph/wip-netaddr

ceph_ansible: Install netaddr

ceph_ansible: Install netaddr

See https://github.com/ceph/ceph-ansible/pull/3199

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1230 from tchaikov/wip-better-remote-run

@zmc @smithfarm gents, i've updated the change to address the concerns. if no objections, i'd like merge as it is in two days.

Merge pull request #1235 from ceph/wip-suppress-cpython

valgind: ignore all leaks relating to CPython code

valgind: ignore all leaks relating to CPython code

Yes, this is a big hammer, and we are ignoring a lot.  However, it is a
HUGE step forward to what we do now, which is not check for ceph-mgr
leaks at all.

By adding this suppress I found and fixed 3 separate ceph-mgr leaks.  This
will let us prevent others (in non-Py code) from being introduced.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1231 from kshtsk/wip-shortname-logging-for-ip

Fix ipv4 and ipv6 address logging for Remote.run

Reviewed-by: Greg Farnum <gfarnum@redhat.com>