task/install/daemon-helper: python3

Signed-off-by: Sage Weil <sage@redhat.com>

task/kernel: don't fail if we're confused by grub

If we don't have menuitem in the grub conf, and we dn't see /boot/loader/entries
file, just don't do grub2-set-default at all and hope for the best.  Worst
case we'll fail later because we didn't boot the right kernel.

Signed-off-by: Sage Weil <sage@redhat.com>

task/kernel: adapt to el8 grub2 /boot/loader/entries/*.conf

Signed-off-by: Sage Weil <sage@redhat.com>

task/kernel: drop removal of old kernels

1- this isn't necessary anymore with fog
2- the package-cleanup --oldkernels option doesn't exist in el8

Signed-off-by: Sage Weil <sage@redhat.com>

task/kernel: fix grub2 detection

grub2 rpm doesn't exist... el8 has

grub2-tools-extra-2.02-66.el8.x86_64
grub2-pc-modules-2.02-66.el8.noarch
grub2-tools-2.02-66.el8.x86_64
grub2-pc-2.02-66.el8.x86_64
grubby-8.40-34.el8.x86_64
grub2-common-2.02-66.el8.noarch
grub2-tools-minimal-2.02-66.el8.x86_64

and el7 has

grub2-tools-minimal-2.02-0.76.el7.centos.x86_64
grub2-pc-2.02-0.76.el7.centos.x86_64
grub2-common-2.02-0.76.el7.centos.noarch
grub2-tools-extra-2.02-0.76.el7.centos.x86_64
grubby-8.28-25.el7.x86_64
grub2-pc-modules-2.02-0.76.el7.centos.noarch
grub2-tools-2.02-0.76.el7.centos.x86_64
grub2-2.02-0.76.el7.centos.x86_64

The difference seems to be that el7 *also* has a /boot/grub directory. Let's
hope this change doesn't break it!

Signed-off-by: Sage Weil <sage@redhat.com>

task/install: make sure /etc/yum/plugin.conf.d/priorities.conf exists

If it doesn' teh subsequent commands all fail.

Signed-off-by: Sage Weil <sage@redhat.com>

misc: get_system_type: RedHatEnterprise is RHEL

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1381 from ceph/wip-cephadm

orchestra: ceph-daemon -> cephadm

orchestra: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1379 from tchaikov/wip-43192

setup.py: add missing dependencies of tests

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

setup.py: add missing dependencies of tests

Fixes: https://tracker.ceph.com/issues/43192
Signed-off-by: Kefu Chai <kchai@redhat.com>

docs: document the fix for incompatible pip-tools with newer pip

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: move more tests related deps to "test"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1375 from ceph/wip-centos8

orchestra/opsys: enable centos 8

orchestra/opsys: enable centos 8

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1374 from ceph/wip-valgrind-file

valgrind.supp is used from ceph/qa

valgrind.supp is used from ceph/qa, we should remove this from setup to avoid
any confusion

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1373 from kshtsk/wip-opensuse-distro-defaults

suite/util: add opensuse to distro defaults

suite/util: add opensuse to distro defaults

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1367 from tchaikov/wip-deps

update/cleanup dependencies

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

docs: document how requirements.txt is updated

we should update this document once
https://github.com/jazzband/pip-tools/issues/625 is fixed and/or
https://github.com/jazzband/pip-tools/issues/908 is merged

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: move test related stuff to optional deps

all of them are for testing teuthology itself.

Signed-off-by: Kefu Chai <kchai@redhat.com>

remove unused dependencies

following dependencies are removed in this change, as they are not
used anymore

* bunch
* web.py

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: move orchestra related stuff to optional deps

Signed-off-by: Kefu Chai <kchai@redhat.com>

pip-compile -P pyyaml

to be consistent with its name on https://pypi.org/project/PyYAML

Signed-off-by: Kefu Chai <kchai@redhat.com>

pip-compile -P gevent

so it's compatible with python3.7

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1371 from rhcs-dashboard/add-cryptography

Add cryptography as explicit dependency

Add cryptography as explicit dependency

Right now it's installed implicitly, but these feature tests require it:
https://github.com/ceph/ceph/pull/31922

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1361 from kshtsk/wip-circular-imports

remove circular imports

Merge pull request #1364 from kshtsk/wip-cloud-openstack-py3

make cloud openstack py3 compatible

Merge pull request #1347 from kshtsk/wip-libcloud-2.6.0

deps: bump apache-libcloud from 1.5.0 to 2.6.0

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

misc: drop 'basestring' usage for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/util: convert iter items to list for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe_tests: make test py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

cloud/util: make syntax py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

packaging: fix syntax for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/run: avoid usage of basestring for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

cloud/openstack: convert filter to list for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test/run: py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: make test_build_matrix py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

contextutil: fix test for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: use isinstance for type check

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test/timer: use py3 compatible syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: use py3 compatible floor division

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

cloud/openstack: do not use basestring

py3 does not have 'basestring' class, so it is better
to use 'str' or just avoid using 'basestring'.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

parallel: add __init__ for py3

Since python3 is using __init__ instead init
it is needed to add this method.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

report: do not use 'basestring' for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1363 from kshtsk/wip-sle-15

add sle 15 for opensuse and sle

orchestra/opsys: add 15.2 for sle and opensuse

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: enable testing on SLE-15-SP1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

orchestra/opsys: add missing SLES versions to DISTRO_CODENAME_MAP

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1360 from kshtsk/wip-py3-ceph-ansible

ceph_ansible: make code py3 compatible

lock/keys: remove useless module

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/ops: remove circular import of keys operations

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/ops: use module full path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/console: remove circular import

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/kernel: simplify filter/lambda for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

get dict's keys as list in py3 compatible way

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test/ansible: handle dict keys as list for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ceph_ansible: make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1359 from kshtsk/wip-remote-sh-instead-run

use remote sh instead run

orchestra/remote: use sh instead of run

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/test: make syntax py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

deps: bump apache-libcloud from 1.5.0 to 2.6.0

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Signed-off-by: Thomas Bechtold <thomas.bechtold@suse.com>

Merge pull request #1355 from kshtsk/wip-matrix-py3

make matrix py3 compatible

Merge pull request #1349 from ceph/wip-cdunit

orchestra: add ceph-daemon DaemonState support

Merge pull request #1353 from ceph/wip-concise-run-message

orchestra/run: more concise logging when we run commands

suite: get rid of cmp for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

py3: use module full path for imports for py3.7

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: remove flake8 messages for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: use dict comprehension for py3 compat

Instead of wierd dict+map+lambda construction it is
much better to use dictionary comprehension for
readability and py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: make code py2/py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1331 from kshtsk/wip-py3-urllib

python3: use compatible urllib packages

Merge pull request #1341 from toabctl/private-key-usage

openstack: Make used private keys explicit

Merge pull request #1329 from kshtsk/wip-drop-stringio-in-run

run: drop StringIO usage for py3 compatibility

python3: add teuthology.util.compat module

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: use compatible urllib packages

Use py2/py3 compatible functions url packages.

Python3
    from urllib.parse import parse_qs, urljoin, urlparse, urlencode
    from urllib.request import urlopen, Request
    from urllib.error import HTTPError
Python2
    from urlparse import parse_qs, urljoin, urlparse
    from urllib import urlencode
    from urllib2 import urlopen, Request, HTTPError

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1330 from kshtsk/wip-six-reraise

core: use six.reraise for py2/py3 compatibility

openstack: Make used private keys explicit

Instead of appending possible private keyfile paths internally, make
that the default for the --key-filename command line argument so users
see it when looking at the help.
If --key-filename is given, only that key is used now (instead of
silently trying the default keys)

Merge pull request #1352 from ceph/WIP-mRemoteCopy

using put_file method instead of python shutil.copyfileobj so it works for python 2 and python 3 both

Merge pull request #1343 from ceph/dependabot/pip/ansible-2.8.2

build(deps): bump ansible from 2.8.0 to 2.8.2

orchestra/daemon/group: simpler syntax

Signed-off-by: Sage Weil <sage@redhat.com>

orchestra/daemon/cephdaemonunit: log each daemon's stdout/stderr

This provides parity with the legacy ceph-daemon-tool behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

orchestra: add CephDaemonUnit daemonstate handler

This is far from complete, but it's sufficient to get a cluster up and
to start + stop daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

using put_file method instead of python shutil.copyfileobj

Signed-off-by: Rachana Patel <racpatel@redhat.com>

orchestra/run: more concise logging when we run commands

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1339 from kshtsk/wip-ghpr

repo_utils: support github pull request branches

repo_utils: support github refs for branches

This adds possibility to checkout github pr providing
special purpose ref spec (like refs/pull/X/merge and
refs/pull/X/head) as branch parameter to teuthology,
for example, "teuthology-suite -t".

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1348 from kshtsk/fix-use-zypper-add-error

install/rpm: fix use the 'zypper addrepo' command

install/rpm: fix use the 'zypper addrepo' command

The 74034fe0348b94aedb247910f0d1c88842f4c3ac instroduced a failure:

  Warning: No repositories defined.
  Use the 'zypper addrepo' command to add one or more repositories.

The wipe command returns exit code 6 if there is no repos in the system.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1342 from toabctl/fix-test-with-newer-git

Fix fetch_branch() check for BranchNotFoundError with newer git

Merge pull request #1340 from toabctl/mention-openstack-setup

docs: Mention step to setup nodes on OpenStack

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1333 from smithfarm/wip-wipe-zypper-repos

install/rpm: wipe all zypper repos before adding repos-under-test

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #1344 from kshtsk/wip-multiple-identityfile

connection: key_filename can be a list

connection: key_filename can be a list

This commit fixes 5ac7c2f and provides support for multiple identityfile,
besides paramiko.SSHClient.connect accepts list as well for key_filename,
pydoc excerpts:

key_filename:
    the filename, or list of filenames, of optional private key(s) to
    try for authentication

Fixes: https://tracker.ceph.com/issues/42543
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fix fetch_branch() check for BranchNotFoundError with newer git

With newer git versions, the string when a branch is not found
is lower case[1].
This fixes the test TestRepoUtils.test_fetch_branch_fake_branch which
was failing due to this problem.

[1] https://github.com/git/git/commit/0b9c3afdbfb629363

build(deps): bump ansible from 2.8.0 to 2.8.2

Bumps [ansible](https://github.com/ansible/community) from 2.8.0 to 2.8.2.
- [Release notes](https://github.com/ansible/community/releases)
- [Commits](https://github.com/ansible/community/commits)

Signed-off-by: dependabot[bot] <support@github.com>

docs: Drop duplicate openstack key pair creation step

It's already in the "Get OpenStack credentials and test it" section
documented how to create a key pair.

docs: Mention step to setup nodes on OpenStack

The --setup step was missing which lead to cryptic error messages when
running the following step (calling the dummy test suite).