docs: document how requirements.txt is updated

we should update this document once
https://github.com/jazzband/pip-tools/issues/625 is fixed and/or
https://github.com/jazzband/pip-tools/issues/908 is merged

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: move test related stuff to optional deps

all of them are for testing teuthology itself.

Signed-off-by: Kefu Chai <kchai@redhat.com>

remove unused dependencies

following dependencies are removed in this change, as they are not
used anymore

* bunch
* web.py

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: move orchestra related stuff to optional deps

Signed-off-by: Kefu Chai <kchai@redhat.com>

pip-compile -P pyyaml

to be consistent with its name on https://pypi.org/project/PyYAML

Signed-off-by: Kefu Chai <kchai@redhat.com>

pip-compile -P gevent

so it's compatible with python3.7

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1371 from rhcs-dashboard/add-cryptography

Add cryptography as explicit dependency

Add cryptography as explicit dependency

Right now it's installed implicitly, but these feature tests require it:
https://github.com/ceph/ceph/pull/31922

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #1361 from kshtsk/wip-circular-imports

remove circular imports

Merge pull request #1364 from kshtsk/wip-cloud-openstack-py3

make cloud openstack py3 compatible

Merge pull request #1347 from kshtsk/wip-libcloud-2.6.0

deps: bump apache-libcloud from 1.5.0 to 2.6.0

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

misc: drop 'basestring' usage for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/util: convert iter items to list for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe_tests: make test py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

cloud/util: make syntax py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

packaging: fix syntax for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/run: avoid usage of basestring for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

cloud/openstack: convert filter to list for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test/run: py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: make test_build_matrix py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

contextutil: fix test for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: use isinstance for type check

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test/timer: use py3 compatible syntax

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: use py3 compatible floor division

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

cloud/openstack: do not use basestring

py3 does not have 'basestring' class, so it is better
to use 'str' or just avoid using 'basestring'.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

parallel: add __init__ for py3

Since python3 is using __init__ instead init
it is needed to add this method.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

report: do not use 'basestring' for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1363 from kshtsk/wip-sle-15

add sle 15 for opensuse and sle

orchestra/opsys: add 15.2 for sle and opensuse

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: enable testing on SLE-15-SP1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

orchestra/opsys: add missing SLES versions to DISTRO_CODENAME_MAP

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1360 from kshtsk/wip-py3-ceph-ansible

ceph_ansible: make code py3 compatible

lock/keys: remove useless module

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/ops: remove circular import of keys operations

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/ops: use module full path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/console: remove circular import

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/kernel: simplify filter/lambda for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

get dict's keys as list in py3 compatible way

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install: py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test/ansible: handle dict keys as list for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

ceph_ansible: make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1359 from kshtsk/wip-remote-sh-instead-run

use remote sh instead run

orchestra/remote: use sh instead of run

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/test: make syntax py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

deps: bump apache-libcloud from 1.5.0 to 2.6.0

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Signed-off-by: Thomas Bechtold <thomas.bechtold@suse.com>

Merge pull request #1355 from kshtsk/wip-matrix-py3

make matrix py3 compatible

Merge pull request #1349 from ceph/wip-cdunit

orchestra: add ceph-daemon DaemonState support

Merge pull request #1353 from ceph/wip-concise-run-message

orchestra/run: more concise logging when we run commands

suite: get rid of cmp for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

py3: use module full path for imports for py3.7

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: remove flake8 messages for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: use dict comprehension for py3 compat

Instead of wierd dict+map+lambda construction it is
much better to use dictionary comprehension for
readability and py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/matrix: make code py2/py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1331 from kshtsk/wip-py3-urllib

python3: use compatible urllib packages

Merge pull request #1341 from toabctl/private-key-usage

openstack: Make used private keys explicit

Merge pull request #1329 from kshtsk/wip-drop-stringio-in-run

run: drop StringIO usage for py3 compatibility

python3: add teuthology.util.compat module

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

python3: use compatible urllib packages

Use py2/py3 compatible functions url packages.

Python3
    from urllib.parse import parse_qs, urljoin, urlparse, urlencode
    from urllib.request import urlopen, Request
    from urllib.error import HTTPError
Python2
    from urlparse import parse_qs, urljoin, urlparse
    from urllib import urlencode
    from urllib2 import urlopen, Request, HTTPError

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1330 from kshtsk/wip-six-reraise

core: use six.reraise for py2/py3 compatibility

openstack: Make used private keys explicit

Instead of appending possible private keyfile paths internally, make
that the default for the --key-filename command line argument so users
see it when looking at the help.
If --key-filename is given, only that key is used now (instead of
silently trying the default keys)

Merge pull request #1352 from ceph/WIP-mRemoteCopy

using put_file method instead of python shutil.copyfileobj so it works for python 2 and python 3 both

Merge pull request #1343 from ceph/dependabot/pip/ansible-2.8.2

build(deps): bump ansible from 2.8.0 to 2.8.2

orchestra/daemon/group: simpler syntax

Signed-off-by: Sage Weil <sage@redhat.com>

orchestra/daemon/cephdaemonunit: log each daemon's stdout/stderr

This provides parity with the legacy ceph-daemon-tool behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

orchestra: add CephDaemonUnit daemonstate handler

This is far from complete, but it's sufficient to get a cluster up and
to start + stop daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

using put_file method instead of python shutil.copyfileobj

Signed-off-by: Rachana Patel <racpatel@redhat.com>

orchestra/run: more concise logging when we run commands

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1339 from kshtsk/wip-ghpr

repo_utils: support github pull request branches

repo_utils: support github refs for branches

This adds possibility to checkout github pr providing
special purpose ref spec (like refs/pull/X/merge and
refs/pull/X/head) as branch parameter to teuthology,
for example, "teuthology-suite -t".

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1348 from kshtsk/fix-use-zypper-add-error

install/rpm: fix use the 'zypper addrepo' command

install/rpm: fix use the 'zypper addrepo' command

The 74034fe0348b94aedb247910f0d1c88842f4c3ac instroduced a failure:

  Warning: No repositories defined.
  Use the 'zypper addrepo' command to add one or more repositories.

The wipe command returns exit code 6 if there is no repos in the system.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1342 from toabctl/fix-test-with-newer-git

Fix fetch_branch() check for BranchNotFoundError with newer git

Merge pull request #1340 from toabctl/mention-openstack-setup

docs: Mention step to setup nodes on OpenStack

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1333 from smithfarm/wip-wipe-zypper-repos

install/rpm: wipe all zypper repos before adding repos-under-test

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #1344 from kshtsk/wip-multiple-identityfile

connection: key_filename can be a list

connection: key_filename can be a list

This commit fixes 5ac7c2f and provides support for multiple identityfile,
besides paramiko.SSHClient.connect accepts list as well for key_filename,
pydoc excerpts:

key_filename:
    the filename, or list of filenames, of optional private key(s) to
    try for authentication

Fixes: https://tracker.ceph.com/issues/42543
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fix fetch_branch() check for BranchNotFoundError with newer git

With newer git versions, the string when a branch is not found
is lower case[1].
This fixes the test TestRepoUtils.test_fetch_branch_fake_branch which
was failing due to this problem.

[1] https://github.com/git/git/commit/0b9c3afdbfb629363

build(deps): bump ansible from 2.8.0 to 2.8.2

Bumps [ansible](https://github.com/ansible/community) from 2.8.0 to 2.8.2.
- [Release notes](https://github.com/ansible/community/releases)
- [Commits](https://github.com/ansible/community/commits)

Signed-off-by: dependabot[bot] <support@github.com>

docs: Drop duplicate openstack key pair creation step

It's already in the "Get OpenStack credentials and test it" section
documented how to create a key pair.

docs: Mention step to setup nodes on OpenStack

The --setup step was missing which lead to cryptic error messages when
running the following step (calling the dummy test suite).

Merge pull request #1338 from toabctl/fix-link-to-ceph-qa-suite

docs: Fix link to ceph/qa suites in intro_testers.rst

docs: Fix link to ceph/qa suites in intro_testers.rst

According to [1], ceph-qa-suite is deprecated (last commit is from
2016) and the QA suites are now in the ceph git repo in the qa/
directory. So point to that directory.

[1] https://github.com/ceph/ceph-qa-suite/

misc: add cwd and env to sh

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1335 from ceph/wip-wwn2

misc: temporary fix for "No space left on device" errors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

misc: temporary fix for "No space left on device" errors

41a13eca480e38cfeeba7a180b4516b90598c39b fixed a longstanding bug that the lab
was relying on. Before the bug was fixed, the get_wwn_id_map function was doing:

    try:
        r = remote.run(
            args=[
                'ls',
                '-l',
                '/dev/disk/by-id/wwn-*',
            ],
            stdout=StringIO(),
        )
        stdout = r.stdout.getvalue()
    except Exception:
        log.info('Failed to get wwn devices! Using /dev/sd* devices...')
        return dict((d, d) for d in devs)

The bug was that "remote.run" was putting single quotes around the string
"/dev/disk/by-id/wwn-*" because it wasn't enclosed in Raw(...). The single
quotes were causing the command to fail, triggering the except clause, and that
was happening 100% of the time.

The fix in 41a13eca480e38cfeeba7a180b4516b90598c39b caused the command to start
succeeding, which caused execution to continue. As a result, MON stores and
OSDs started getting created on the wrong devices, and tests that were
previously succeeding started to fail due to "No space left on device".

In short, the wwn devices on today's smithis are not big enough for
/var/lib/ceph.

This commit "fixes the fix" by dropping the dead code and always returning the
value that qa/tasks/ceph.py has come to expect.

Fixes: https://tracker.ceph.com/issues/42313
Signed-off-by: Nathan Cutler <ncutler@suse.com>

install/rpm: wipe all zypper repos before adding repos-under-test

Our tests should be designed to not rely on any pre-existing repos. If there are
any pre-existing repos, these can potentially skew test results by causing
unexpected packages/package versions to be installed.

Therefore, wipe any pre-existing repos before adding the repos-under-test.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1326 from kshtsk/wip-great-again

make teuthology-openstack great again

openstack: copy missing user-data from suse repo

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: cleanup stale comments

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

openstack: make use base64 for python3 compatible code

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

openstack: delete unused __flavor and __flavor_range methods

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

openstack: use --limit 2000 with openstack image list command

The --limit defaults to 25, which is way too low.

Fixes: https://github.com/SUSE/teuthology/issues/82
Signed-off-by: Loic Dachary <loic@dachary.org>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: only consider private images

OVH only returns the first 25 images, and the public ones (which we don't use
at all) eat up all the spots.

Fixes: https://github.com/SUSE/teuthology/issues/82
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: increase ceph-fuse delays

In the same way it's done in machine_types/vps.yaml

Signed-off-by: Loic Dachary <loic@dachary.org>
Conflicts:
teuthology/openstack/openstack.yaml

setup-openstack.sh: setup_pulpito: respond to setuptools treachery

Apparently, setuptools has a "screw the users" policy.

Fixes: https://bugzilla.suse.com/show_bug.cgi?id=1023257
Signed-off-by: Nathan Cutler <ncutler@suse.com>

user-data: global s/sudo tee/tee/, if{up,down} -a

Signed-off-by: Nathan Cutler <ncutler@suse.com>

user-data: tweak Ubuntu 16.04 user-data

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: use wget instead of curl for Ubuntu-14.04

So that it also works with Ubuntu 14.04.1 which did not have curl by
default.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: support /etc/network/intefaces injection

Not all OpenStack providers have a dhcp server. Some of
them (dreamcompute for instance) may inject /etc/network/interfaces with
a pre-configured stanza. It is not enough to set the dns server in the
dhcp client configuration, it also needs to be set directly in
/etc/resolv.conf when that has no effect.

http://tracker.ceph.com/issues/14024 Fixes: #14024

Signed-off-by: Loic Dachary <loic@dachary.org>

ensure VMs always have /etc/hosts set up

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>