]> git.apps.os.sepia.ceph.com Git - teuthology.git/log
teuthology.git
5 years agoorchestra/remote: use sh instead of run 1359/head
Kyr Shatskyy [Fri, 15 Nov 2019 16:56:55 +0000 (17:56 +0100)]
orchestra/remote: use sh instead of run

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoorchestra/test: make syntax py3 compatible
Kyr Shatskyy [Fri, 15 Nov 2019 16:38:03 +0000 (17:38 +0100)]
orchestra/test: make syntax py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoMerge pull request #1355 from kshtsk/wip-matrix-py3
kshtsk [Thu, 14 Nov 2019 20:31:02 +0000 (21:31 +0100)]
Merge pull request #1355 from kshtsk/wip-matrix-py3

make matrix py3 compatible

5 years agoMerge pull request #1349 from ceph/wip-cdunit
Sage Weil [Thu, 14 Nov 2019 19:40:41 +0000 (13:40 -0600)]
Merge pull request #1349 from ceph/wip-cdunit

orchestra: add ceph-daemon DaemonState support

5 years agoMerge pull request #1353 from ceph/wip-concise-run-message
Sage Weil [Thu, 14 Nov 2019 19:40:14 +0000 (13:40 -0600)]
Merge pull request #1353 from ceph/wip-concise-run-message

orchestra/run: more concise logging when we run commands

5 years agosuite: get rid of cmp for py3 1355/head
Kyr Shatskyy [Thu, 14 Nov 2019 19:06:42 +0000 (20:06 +0100)]
suite: get rid of cmp for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agopy3: use module full path for imports for py3.7
Kyr Shatskyy [Thu, 14 Nov 2019 18:32:25 +0000 (19:32 +0100)]
py3: use module full path for imports for py3.7

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoopenstack: remove flake8 messages for py3
Kyr Shatskyy [Thu, 14 Nov 2019 18:30:38 +0000 (19:30 +0100)]
openstack: remove flake8 messages for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoopenstack: use dict comprehension for py3 compat
Kyr Shatskyy [Thu, 14 Nov 2019 18:25:06 +0000 (19:25 +0100)]
openstack: use dict comprehension for py3 compat

Instead of wierd dict+map+lambda construction it is
much better to use dictionary comprehension for
readability and py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agosuite/matrix: make code py2/py3 compatible
Kyr Shatskyy [Thu, 14 Nov 2019 18:07:23 +0000 (19:07 +0100)]
suite/matrix: make code py2/py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoMerge pull request #1331 from kshtsk/wip-py3-urllib
kshtsk [Thu, 14 Nov 2019 18:20:01 +0000 (19:20 +0100)]
Merge pull request #1331 from kshtsk/wip-py3-urllib

python3: use compatible urllib packages

5 years agoMerge pull request #1341 from toabctl/private-key-usage
kshtsk [Thu, 14 Nov 2019 16:54:06 +0000 (17:54 +0100)]
Merge pull request #1341 from toabctl/private-key-usage

openstack: Make used private keys explicit

5 years agoMerge pull request #1329 from kshtsk/wip-drop-stringio-in-run
kshtsk [Thu, 14 Nov 2019 16:41:03 +0000 (17:41 +0100)]
Merge pull request #1329 from kshtsk/wip-drop-stringio-in-run

run: drop StringIO usage for py3 compatibility

5 years agopython3: add teuthology.util.compat module 1331/head
Kyr Shatskyy [Thu, 14 Nov 2019 13:27:08 +0000 (14:27 +0100)]
python3: add teuthology.util.compat module

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agopython3: use compatible urllib packages
Kyr Shatskyy [Fri, 11 Oct 2019 23:42:00 +0000 (01:42 +0200)]
python3: use compatible urllib packages

Use py2/py3 compatible functions url packages.

Python3
    from urllib.parse import parse_qs, urljoin, urlparse, urlencode
    from urllib.request import urlopen, Request
    from urllib.error import HTTPError
Python2
    from urlparse import parse_qs, urljoin, urlparse
    from urllib import urlencode
    from urllib2 import urlopen, Request, HTTPError

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoMerge pull request #1330 from kshtsk/wip-six-reraise
kshtsk [Thu, 14 Nov 2019 13:14:12 +0000 (14:14 +0100)]
Merge pull request #1330 from kshtsk/wip-six-reraise

core: use six.reraise for py2/py3 compatibility

5 years agoopenstack: Make used private keys explicit 1341/head
Thomas Bechtold [Thu, 31 Oct 2019 09:48:01 +0000 (10:48 +0100)]
openstack: Make used private keys explicit

Instead of appending possible private keyfile paths internally, make
that the default for the --key-filename command line argument so users
see it when looking at the help.
If --key-filename is given, only that key is used now (instead of
silently trying the default keys)

5 years agoMerge pull request #1352 from ceph/WIP-mRemoteCopy
kshtsk [Wed, 13 Nov 2019 19:16:33 +0000 (20:16 +0100)]
Merge pull request #1352 from ceph/WIP-mRemoteCopy

using put_file method instead of python shutil.copyfileobj so it works for python 2 and python 3 both

5 years agoMerge pull request #1343 from ceph/dependabot/pip/ansible-2.8.2
kshtsk [Wed, 13 Nov 2019 19:15:39 +0000 (20:15 +0100)]
Merge pull request #1343 from ceph/dependabot/pip/ansible-2.8.2

build(deps): bump ansible from 2.8.0 to 2.8.2

5 years agoorchestra/daemon/group: simpler syntax 1349/head
Sage Weil [Wed, 13 Nov 2019 14:31:41 +0000 (14:31 +0000)]
orchestra/daemon/group: simpler syntax

Signed-off-by: Sage Weil <sage@redhat.com>
5 years agoorchestra/daemon/cephdaemonunit: log each daemon's stdout/stderr
Sage Weil [Wed, 13 Nov 2019 00:00:41 +0000 (00:00 +0000)]
orchestra/daemon/cephdaemonunit: log each daemon's stdout/stderr

This provides parity with the legacy ceph-daemon-tool behavior.

Signed-off-by: Sage Weil <sage@redhat.com>
5 years agoorchestra: add CephDaemonUnit daemonstate handler
Sage Weil [Tue, 12 Nov 2019 18:42:37 +0000 (18:42 +0000)]
orchestra: add CephDaemonUnit daemonstate handler

This is far from complete, but it's sufficient to get a cluster up and
to start + stop daemons.

Signed-off-by: Sage Weil <sage@redhat.com>
5 years agousing put_file method instead of python shutil.copyfileobj 1352/head
Rachana Patel [Wed, 13 Nov 2019 05:22:45 +0000 (00:22 -0500)]
using put_file method instead of python shutil.copyfileobj

Signed-off-by: Rachana Patel <racpatel@redhat.com>
5 years agoorchestra/run: more concise logging when we run commands 1353/head
Sage Weil [Tue, 12 Nov 2019 15:59:05 +0000 (15:59 +0000)]
orchestra/run: more concise logging when we run commands

Signed-off-by: Sage Weil <sage@redhat.com>
5 years agoMerge pull request #1339 from kshtsk/wip-ghpr
kshtsk [Tue, 12 Nov 2019 16:31:36 +0000 (17:31 +0100)]
Merge pull request #1339 from kshtsk/wip-ghpr

repo_utils: support github pull request branches

5 years agorepo_utils: support github refs for branches 1339/head
Kyr Shatskyy [Tue, 22 Oct 2019 09:44:11 +0000 (11:44 +0200)]
repo_utils: support github refs for branches

This adds possibility to checkout github pr providing
special purpose ref spec (like refs/pull/X/merge and
refs/pull/X/head) as branch parameter to teuthology,
for example, "teuthology-suite -t".

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoMerge pull request #1348 from kshtsk/fix-use-zypper-add-error
kshtsk [Tue, 12 Nov 2019 01:27:38 +0000 (02:27 +0100)]
Merge pull request #1348 from kshtsk/fix-use-zypper-add-error

install/rpm: fix use the 'zypper addrepo' command

5 years agoinstall/rpm: fix use the 'zypper addrepo' command 1348/head
Kyr Shatskyy [Mon, 11 Nov 2019 15:12:18 +0000 (16:12 +0100)]
install/rpm: fix use the 'zypper addrepo' command

The 74034fe0348b94aedb247910f0d1c88842f4c3ac instroduced a failure:

  Warning: No repositories defined.
  Use the 'zypper addrepo' command to add one or more repositories.

The wipe command returns exit code 6 if there is no repos in the system.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoMerge pull request #1342 from toabctl/fix-test-with-newer-git
kshtsk [Thu, 7 Nov 2019 18:56:33 +0000 (19:56 +0100)]
Merge pull request #1342 from toabctl/fix-test-with-newer-git

Fix fetch_branch() check for BranchNotFoundError with newer git

5 years agoMerge pull request #1340 from toabctl/mention-openstack-setup
Nathan Cutler [Wed, 6 Nov 2019 07:24:21 +0000 (08:24 +0100)]
Merge pull request #1340 from toabctl/mention-openstack-setup

docs: Mention step to setup nodes on OpenStack

Reviewed-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #1333 from smithfarm/wip-wipe-zypper-repos
Nathan Cutler [Wed, 6 Nov 2019 07:20:16 +0000 (08:20 +0100)]
Merge pull request #1333 from smithfarm/wip-wipe-zypper-repos

install/rpm: wipe all zypper repos before adding repos-under-test

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
5 years agoMerge pull request #1344 from kshtsk/wip-multiple-identityfile
kshtsk [Tue, 5 Nov 2019 18:49:13 +0000 (20:49 +0200)]
Merge pull request #1344 from kshtsk/wip-multiple-identityfile

connection: key_filename can be a list

5 years agoconnection: key_filename can be a list 1344/head
Kyr Shatskyy [Tue, 5 Nov 2019 10:40:12 +0000 (11:40 +0100)]
connection: key_filename can be a list

This commit fixes 5ac7c2f and provides support for multiple identityfile,
besides paramiko.SSHClient.connect accepts list as well for key_filename,
pydoc excerpts:

key_filename:
    the filename, or list of filenames, of optional private key(s) to
    try for authentication

Fixes: https://tracker.ceph.com/issues/42543
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoFix fetch_branch() check for BranchNotFoundError with newer git 1342/head
Thomas Bechtold [Fri, 1 Nov 2019 08:13:04 +0000 (09:13 +0100)]
Fix fetch_branch() check for BranchNotFoundError with newer git

With newer git versions, the string when a branch is not found
is lower case[1].
This fixes the test TestRepoUtils.test_fetch_branch_fake_branch which
was failing due to this problem.

[1] https://github.com/git/git/commit/0b9c3afdbfb629363

5 years agobuild(deps): bump ansible from 2.8.0 to 2.8.2 1343/head
dependabot[bot] [Sat, 2 Nov 2019 02:57:17 +0000 (02:57 +0000)]
build(deps): bump ansible from 2.8.0 to 2.8.2

Bumps [ansible](https://github.com/ansible/community) from 2.8.0 to 2.8.2.
- [Release notes](https://github.com/ansible/community/releases)
- [Commits](https://github.com/ansible/community/commits)

Signed-off-by: dependabot[bot] <support@github.com>
5 years agodocs: Drop duplicate openstack key pair creation step 1340/head
Thomas Bechtold [Fri, 1 Nov 2019 04:47:02 +0000 (05:47 +0100)]
docs: Drop duplicate openstack key pair creation step

It's already in the "Get OpenStack credentials and test it" section
documented how to create a key pair.

5 years agodocs: Mention step to setup nodes on OpenStack
Thomas Bechtold [Wed, 23 Oct 2019 10:19:32 +0000 (12:19 +0200)]
docs: Mention step to setup nodes on OpenStack

The --setup step was missing which lead to cryptic error messages when
running the following step (calling the dummy test suite).

5 years agoMerge pull request #1338 from toabctl/fix-link-to-ceph-qa-suite
Gregory Farnum [Tue, 22 Oct 2019 20:30:08 +0000 (13:30 -0700)]
Merge pull request #1338 from toabctl/fix-link-to-ceph-qa-suite

docs: Fix link to ceph/qa suites in intro_testers.rst

5 years agodocs: Fix link to ceph/qa suites in intro_testers.rst 1338/head
Thomas Bechtold [Tue, 22 Oct 2019 08:50:36 +0000 (10:50 +0200)]
docs: Fix link to ceph/qa suites in intro_testers.rst

According to [1], ceph-qa-suite is deprecated (last commit is from
2016) and the QA suites are now in the ceph git repo in the qa/
directory. So point to that directory.

[1] https://github.com/ceph/ceph-qa-suite/

5 years agomisc: add cwd and env to sh
Kyr Shatskyy [Tue, 22 Oct 2019 09:13:41 +0000 (11:13 +0200)]
misc: add cwd and env to sh

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoMerge pull request #1335 from ceph/wip-wwn2
Nathan Cutler [Thu, 17 Oct 2019 13:09:29 +0000 (15:09 +0200)]
Merge pull request #1335 from ceph/wip-wwn2

misc: temporary fix for "No space left on device" errors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
5 years agomisc: temporary fix for "No space left on device" errors 1335/head
Nathan Cutler [Wed, 16 Oct 2019 21:04:42 +0000 (23:04 +0200)]
misc: temporary fix for "No space left on device" errors

41a13eca480e38cfeeba7a180b4516b90598c39b fixed a longstanding bug that the lab
was relying on. Before the bug was fixed, the get_wwn_id_map function was doing:

    try:
        r = remote.run(
            args=[
                'ls',
                '-l',
                '/dev/disk/by-id/wwn-*',
            ],
            stdout=StringIO(),
        )
        stdout = r.stdout.getvalue()
    except Exception:
        log.info('Failed to get wwn devices! Using /dev/sd* devices...')
        return dict((d, d) for d in devs)

The bug was that "remote.run" was putting single quotes around the string
"/dev/disk/by-id/wwn-*" because it wasn't enclosed in Raw(...). The single
quotes were causing the command to fail, triggering the except clause, and that
was happening 100% of the time.

The fix in 41a13eca480e38cfeeba7a180b4516b90598c39b caused the command to start
succeeding, which caused execution to continue. As a result, MON stores and
OSDs started getting created on the wrong devices, and tests that were
previously succeeding started to fail due to "No space left on device".

In short, the wwn devices on today's smithis are not big enough for
/var/lib/ceph.

This commit "fixes the fix" by dropping the dead code and always returning the
value that qa/tasks/ceph.py has come to expect.

Fixes: https://tracker.ceph.com/issues/42313
Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoinstall/rpm: wipe all zypper repos before adding repos-under-test 1333/head
Nathan Cutler [Wed, 16 Oct 2019 09:11:53 +0000 (11:11 +0200)]
install/rpm: wipe all zypper repos before adding repos-under-test

Our tests should be designed to not rely on any pre-existing repos. If there are
any pre-existing repos, these can potentially skew test results by causing
unexpected packages/package versions to be installed.

Therefore, wipe any pre-existing repos before adding the repos-under-test.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoMerge pull request #1326 from kshtsk/wip-great-again
kshtsk [Tue, 15 Oct 2019 19:02:06 +0000 (21:02 +0200)]
Merge pull request #1326 from kshtsk/wip-great-again

make teuthology-openstack great again

5 years agoopenstack: copy missing user-data from suse repo 1326/head
Kyr Shatskyy [Mon, 14 Oct 2019 15:19:07 +0000 (17:19 +0200)]
openstack: copy missing user-data from suse repo

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoopenstack: cleanup stale comments
Kyr Shatskyy [Fri, 11 Oct 2019 04:46:08 +0000 (06:46 +0200)]
openstack: cleanup stale comments

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
5 years agoopenstack: make use base64 for python3 compatible code
Kyr Shatskyy [Fri, 11 Oct 2019 04:37:55 +0000 (06:37 +0200)]
openstack: make use base64 for python3 compatible code

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
5 years agoopenstack: delete unused __flavor and __flavor_range methods
Kyr Shatskyy [Fri, 11 Oct 2019 03:49:35 +0000 (05:49 +0200)]
openstack: delete unused __flavor and __flavor_range methods

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
5 years agoopenstack: use --limit 2000 with openstack image list command
Nathan Cutler [Fri, 2 Jun 2017 12:42:32 +0000 (14:42 +0200)]
openstack: use --limit 2000 with openstack image list command

The --limit defaults to 25, which is way too low.

Fixes: https://github.com/SUSE/teuthology/issues/82
Signed-off-by: Loic Dachary <loic@dachary.org>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoopenstack: only consider private images
Nathan Cutler [Fri, 2 Jun 2017 12:21:22 +0000 (14:21 +0200)]
openstack: only consider private images

OVH only returns the first 25 images, and the public ones (which we don't use
at all) eat up all the spots.

Fixes: https://github.com/SUSE/teuthology/issues/82
Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoopenstack: increase ceph-fuse delays
Loic Dachary [Tue, 5 Apr 2016 20:47:14 +0000 (22:47 +0200)]
openstack: increase ceph-fuse delays

In the same way it's done in machine_types/vps.yaml

Signed-off-by: Loic Dachary <loic@dachary.org>
Conflicts:
teuthology/openstack/openstack.yaml

5 years agosetup-openstack.sh: setup_pulpito: respond to setuptools treachery
Nathan Cutler [Thu, 2 Feb 2017 21:05:25 +0000 (22:05 +0100)]
setup-openstack.sh: setup_pulpito: respond to setuptools treachery

Apparently, setuptools has a "screw the users" policy.

Fixes: https://bugzilla.suse.com/show_bug.cgi?id=1023257
Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agouser-data: global s/sudo tee/tee/, if{up,down} -a
Nathan Cutler [Sun, 12 Feb 2017 11:33:12 +0000 (12:33 +0100)]
user-data: global s/sudo tee/tee/, if{up,down} -a

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agouser-data: tweak Ubuntu 16.04 user-data
Nathan Cutler [Sun, 12 Feb 2017 07:45:35 +0000 (08:45 +0100)]
user-data: tweak Ubuntu 16.04 user-data

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoopenstack: use wget instead of curl for Ubuntu-14.04
Loic Dachary [Mon, 28 Mar 2016 16:37:11 +0000 (18:37 +0200)]
openstack: use wget instead of curl for Ubuntu-14.04

So that it also works with Ubuntu 14.04.1 which did not have curl by
default.

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoopenstack: support /etc/network/intefaces injection
Loic Dachary [Tue, 8 Dec 2015 09:52:10 +0000 (10:52 +0100)]
openstack: support /etc/network/intefaces injection

Not all OpenStack providers have a dhcp server. Some of
them (dreamcompute for instance) may inject /etc/network/interfaces with
a pre-configured stanza. It is not enough to set the dns server in the
dhcp client configuration, it also needs to be set directly in
/etc/resolv.conf when that has no effect.

http://tracker.ceph.com/issues/14024 Fixes: #14024

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoensure VMs always have /etc/hosts set up
Loic Dachary [Tue, 8 Dec 2015 00:53:10 +0000 (01:53 +0100)]
ensure VMs always have /etc/hosts set up

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
5 years agobuildpackages: use items for py3 compatibility
Kyr Shatskyy [Mon, 14 Oct 2019 15:14:13 +0000 (17:14 +0200)]
buildpackages: use items for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agobuildpackages: sleep first in the wait/backoff loop
Nathan Cutler [Wed, 28 Feb 2018 18:27:27 +0000 (19:27 +0100)]
buildpackages: sleep first in the wait/backoff loop

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: change wait/backoff intervals
Nathan Cutler [Wed, 28 Feb 2018 15:42:17 +0000 (16:42 +0100)]
buildpackages: change wait/backoff intervals

The sshd in SLE15 is more paranoid and does not tolerate much banging on port 22:

+ sleep 8
+ for delay in 1 2 4 8 8 8 8 8 8 8 8 8 16 16 16 16 16 32 32 32 64 128 256 512
+ ssh -o ConnectTimeout=3 137.74.26.189 bash -c '"grep -q READYTORUN /var/log/cloud-init*.log"'
Warning: Permanently added '137.74.26.189' (ECDSA) to the list of known hosts.
Received disconnect from 137.74.26.189 port 22:2: Too many authentication failures

Also log the ssh status code.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoCleanup to use merging from suse stream
Kyr Shatskyy [Mon, 14 Oct 2019 15:00:10 +0000 (17:00 +0200)]
Cleanup to use merging from suse stream

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoopenstack: user-data: install chrony on SLE15 and openSUSE 15
Nathan Cutler [Wed, 14 Mar 2018 22:25:27 +0000 (23:25 +0100)]
openstack: user-data: install chrony on SLE15 and openSUSE 15

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agosle-15-user-data: copy authorized_keys from root
Nathan Cutler [Thu, 1 Mar 2018 15:38:18 +0000 (16:38 +0100)]
sle-15-user-data: copy authorized_keys from root

The image generation script is overriding the disable_root setting in
/etc/cloud/cloud.cfg. As a result, openstack seeds the key in
/root/.ssh/authorized_keys instead of /home/sles/.ssh/authorized_keys

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: drop unneeded packages in SLE15
Nathan Cutler [Wed, 28 Feb 2018 17:02:39 +0000 (18:02 +0100)]
buildpackages: drop unneeded packages in SLE15

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoSupport SLE-15
Nathan Cutler [Wed, 10 Jan 2018 13:33:42 +0000 (14:33 +0100)]
Support SLE-15

With SLE15 it is no longer possible to SSH in as root. Instead, the OpenStack
SSH public key is injected into /home/sles. Reflect this in the user-data
files.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoTeuthologyOpenStack: use yaml.safe_load
Kyr Shatskyy [Mon, 14 Oct 2019 21:48:59 +0000 (23:48 +0200)]
TeuthologyOpenStack: use yaml.safe_load

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agotask/internal: drop cStringIO for py3 compatibility 1329/head
Kyr Shatskyy [Fri, 11 Oct 2019 19:35:36 +0000 (21:35 +0200)]
task/internal: drop cStringIO for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agorun: drop StringIO usage for py3 compatibility
Kyr Shatskyy [Fri, 11 Oct 2019 18:59:20 +0000 (20:59 +0200)]
run: drop StringIO usage for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoteuthology-openstack: fix broken deployment
Nathan Cutler [Thu, 25 Oct 2018 15:37:52 +0000 (17:37 +0200)]
teuthology-openstack: fix broken deployment

d6be711301fe7331acd428ab261ea648ef86305f broke the t-o deployment,
causing it to fail with the following Traceback:

2018-10-25 17:23:46,078.078 DEBUG:teuthology.misc:No server with a name
or ID of 'teuth-jschmid' exists.
Traceback (most recent call last):
  File "/home/jxs/projects/teuthology/teuthology/v/bin/teuthology-openstack", line 11, in <module>
    load_entry_point('teuthology', 'console_scripts', 'teuthology-openstack')()
  File "/home/jxs/projects/teuthology/teuthology/scripts/openstack.py", line 7, in main
    sys.exit(teuthology.openstack.main(parse_args(argv), argv))
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 1310, in main
    return TeuthologyOpenStack(ctx, teuth_config, argv).main()
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 704, in main
    self.setup()
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 890, in setup
    self.instance = OpenStackInstance(self.server_name())
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 82, in __init__
    if self.info['status'] == 'ERROR':
TypeError: 'NoneType' object has no attribute '__getitem__'

Signed-off-by: Joshua Schmid <jschmid@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/openstack/__init__.py

5 years agoopenstack: fail immediately if VM creation fails
Nathan Cutler [Tue, 23 Oct 2018 11:08:34 +0000 (13:08 +0200)]
openstack: fail immediately if VM creation fails

Fixes: https://github.com/SUSE/teuthology/issues/159
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/openstack/__init__.py

5 years agoopenstack: retry if volume create+attach operation fails
Kyr Shatskyy [Thu, 18 Oct 2018 00:41:41 +0000 (02:41 +0200)]
openstack: retry if volume create+attach operation fails

Once in awhile, a created volume fails to become available within
a reasonable period of time. When this happens, the entire test job
fails with the following error:

2018-10-02T06:52:44.106 ERROR:teuthology.provision.openstack:'volume
target192168000058-2' reached maximum tries (100) after waiting for 200
seconds

With this commit, we catch that exception and retry with a new volume.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/provision/openstack.py

5 years agoprovision/openstack: put "add volume" in a retry loop
Nathan Cutler [Thu, 23 Mar 2017 13:47:29 +0000 (14:47 +0100)]
provision/openstack: put "add volume" in a retry loop

Attempted workaround for this issue:

2017-03-23T12:33:20.684 DEBUG:teuthology.misc::sh: openstack --quiet volume show  -f json target217182140122-1
2017-03-23T12:34:22.140 DEBUG:teuthology.misc:{
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"size": 10,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"status": "available",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"properties": "ownedby='217.182.143.170'",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"user_id": "122b249e82994f5aa8ec6a31b7891df5",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"description": null,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"availability_zone": "nova",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"bootable": "false",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"encrypted": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"created_at": "2017-03-23T12:33:17.000000",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"multiattach": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:driver_data": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"name": "target217182140122-1",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"snapshot_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"consistencygroup_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"replication_status": "disabled",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-vol-tenant-attr:tenant_id": "43583fef7b734bb1b6fb5f86d48abe5e",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"source_volid": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:extended_status": null,
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"type": "classic",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"id": "e6949180-87fe-4f4c-b0a0-e3ea35bdb2e2",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"attachments": []
2017-03-23T12:34:25.036 DEBUG:teuthology.misc:}
2017-03-23T12:34:25.042 DEBUG:teuthology.misc::sh: openstack server add volume target217182140122 target217182140122-1
2017-03-23T12:35:30.536 DEBUG:teuthology.misc:No volume with a name or ID of 'target217182140122-1' exists.
2017-03-23T12:35:34.512 ERROR:teuthology.provision.openstack:Command 'openstack server add volume target217182140122 target217182140122-1' returned non-zero exit status 1

Fixes: http://tracker.ceph.com/issues/17659
Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agotests: skip openstack tests if OS_AUTH_URL not set
Nathan Cutler [Tue, 7 Feb 2017 15:09:14 +0000 (16:09 +0100)]
tests: skip openstack tests if OS_AUTH_URL not set

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoopenstack: hide OS_ variables dumping in logs
Kyr Shatskyy [Thu, 31 Jan 2019 15:34:10 +0000 (16:34 +0100)]
openstack: hide OS_ variables dumping in logs

Hide OS_ variables in the log, so it can't be easily
read by human and steal secrets.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoopenstack: Support GitHub PR merge branches
Kyrylo Shatskyy [Thu, 6 Dec 2018 19:42:54 +0000 (20:42 +0100)]
openstack: Support GitHub PR merge branches

Add support for github pr merge branches, like 'origin/pr/*/merge',
so it can be used to setup teuthology server using teuthology-openstack
command line tool with --teuthology-branch argument.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoopenstack: add get_instance method
Kyr Shatskyy [Fri, 2 Nov 2018 13:47:18 +0000 (14:47 +0100)]
openstack: add get_instance method

Refactor teuthology-openstack so it get openstack instance once.
Allow to run suite without prior setup() invocation.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
5 years agoopenstack: fix pulpito and paddles status in init script
Kyr Shatskyy [Mon, 22 Oct 2018 19:58:16 +0000 (21:58 +0200)]
openstack: fix pulpito and paddles status in init script

After deployment teuthology cluster on openstack, teuthology
service status-paddles and status-pulpito commands report
'dead', however paddles and pulpito are running.
This fix addresses this issue.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
5 years agoteuthology-openstack: add banner after cluster creation
Kyr Shatskyy [Sat, 20 Oct 2018 03:18:34 +0000 (05:18 +0200)]
teuthology-openstack: add banner after cluster creation

When creating a teuthology openstack cluster and not
requesting any suite suite run the teuthology-openstack
quits silently not giving information where the cluster
is created and how its services can be reached.
This fix addresses the issue above.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
5 years agoteuthology-openstack: add nameserver option
Kyr Shatskyy [Tue, 16 Oct 2018 10:39:19 +0000 (12:39 +0200)]
teuthology-openstack: add nameserver option

Add --nameserver option to teuthology-openstack cli tool so
teuthology cluster can be deployed and configured to use
customly provided dns

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
5 years agobuildpackages: use makecheck image to instantiate build VM
Nathan Cutler [Wed, 25 Apr 2018 19:39:34 +0000 (21:39 +0200)]
buildpackages: use makecheck image to instantiate build VM

The teuthology images no longer have the right repos for building Ceph.
Use the makecheck images instead.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoFix ovh paddles populate
Kyr Shatskyy [Fri, 5 Oct 2018 12:22:17 +0000 (14:22 +0200)]
Fix ovh paddles populate

Unfortunate typo instroduced in sha1 e491969d4b5b9dca710d1ce89f0cde918bbee1b4
which fixed ECP nodes creation and broke OVH nodes creation.

5 years agoFix cannot populate paddles
Kyr Shatskyy [Thu, 13 Sep 2018 12:03:21 +0000 (14:03 +0200)]
Fix cannot populate paddles

When trying to deploy teuthology on openstack with several networks
and non unique subnet addresses setup-openstack script fails to add
nodes to paddles databases.

This patch removes duplicates while getting default_subnets.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
5 years agoconnection: key_filename is a string, not an array
Loic Dachary [Tue, 7 Feb 2017 14:23:55 +0000 (15:23 +0100)]
connection: key_filename is a string, not an array

Signed-off-by: Loic Dachary <ldachary@redhat.com>
5 years agoAdd ECP support
Kyr Shatskyy [Wed, 21 Mar 2018 23:16:27 +0000 (00:16 +0100)]
Add ECP support

Isolated __flavor and __flavor_range from openstack invocation.
Flavor and Network parameters can be taken from teuthology config
under 'openstack' records.
Added filters for new OVH flavors.

 Conflicts:
teuthology/openstack/__init__.py

5 years agoopenstack: Support Leap 42.3
Nathan Cutler [Wed, 31 May 2017 12:49:16 +0000 (14:49 +0200)]
openstack: Support Leap 42.3

Without Salt for now, due to https://bugzilla.opensuse.org/show_bug.cgi?id=1042379

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agoopenstack: cloudlab support (part 2)
Loic Dachary [Sun, 3 Apr 2016 16:40:24 +0000 (18:40 +0200)]
openstack: cloudlab support (part 2)

The ctl hostname was replaced by a fqdn that always ends with cloudlab.us

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoopenstack: do not use cached tokens for neutron
Loic Dachary [Mon, 22 Aug 2016 16:20:05 +0000 (18:20 +0200)]
openstack: do not use cached tokens for neutron

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoopenstack: teuthology must use aarch64, not arm64
Loic Dachary [Tue, 5 Apr 2016 08:45:37 +0000 (10:45 +0200)]
openstack: teuthology must use aarch64, not arm64

teuthology / gitbuilder / ansible_architecture arch is

   dpkg-architecture --query DEB_HOST_GNU_CPU (x86_64 and aarch64)

deb package Architectures: (conf/distributions) is

   dpkg-architecture --query DEB_HOST_GNU_ARCH (amd64 and arm64)

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoopenstack: move net() to OpenStack
Loic Dachary [Mon, 28 Mar 2016 17:49:02 +0000 (19:49 +0200)]
openstack: move net() to OpenStack

It is not only needed when provisionning the teuthology instance.

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoopenstack: rework floating IP handling
Loic Dachary [Mon, 28 Mar 2016 17:35:34 +0000 (19:35 +0200)]
openstack: rework floating IP handling

Signed-off-by: Loic Dachary <loic@dachary.org>
Conflicts:
teuthology/openstack/__init__.py

5 years agoopenstack: there may be more than one IPv4 subnet
Loic Dachary [Sun, 27 Mar 2016 11:50:01 +0000 (13:50 +0200)]
openstack: there may be more than one IPv4 subnet

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoopenstack: increase teuthology flavors
Loic Dachary [Mon, 10 Apr 2017 07:29:28 +0000 (09:29 +0200)]
openstack: increase teuthology flavors

Ansible has higher memory requirements these days. Keep the default
flavor to the minimum because it is used during tests.

Signed-off-by: Loic Dachary <loic@dachary.org>
5 years agoAdd multi-user support for openstack
Kyr Shatskyy [Thu, 11 Jan 2018 12:26:52 +0000 (14:26 +0200)]
Add multi-user support for openstack

Conflicts:
requirements.txt
teuthology/misc.py
teuthology/openstack/__init__.py
teuthology/provision/openstack.py

5 years agobuildpackages: use createrepo_c package on newer SUSEs
Nathan Cutler [Tue, 6 Mar 2018 06:25:35 +0000 (07:25 +0100)]
buildpackages: use createrepo_c package on newer SUSEs

The old createrepo package brings in Python 2. As of Code 15, that's no longer
going to work.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: run zypper refresh in try/wait loop
Nathan Cutler [Mon, 5 Mar 2018 00:22:03 +0000 (01:22 +0100)]
buildpackages: run zypper refresh in try/wait loop

Avoid silly things like:

+ sudo zypper --non-interactive --no-gpg-checks refresh
System management is locked by the application with pid 2560 (zypper).

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: fix dates in spec file changelog
Nathan Cutler [Sat, 3 Mar 2018 07:05:56 +0000 (08:05 +0100)]
buildpackages: fix dates in spec file changelog

Eliminates the following warnings:

2018-03-03T06:56:01.948 DEBUG:teuthology.misc:warning: bogus date in %changelog: Tue Mar 10 2013 Gary Lowell <glowell@inktank.com> - 1-0
2018-03-03T06:56:01.948 DEBUG:teuthology.misc:warning: bogus date in %changelog: Tue Aug 27 2012 Gary Lowell <glowell@inktank.com> - 1-0

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: drop suse-specific code in ceph-release.spec
Nathan Cutler [Sat, 3 Mar 2018 07:05:19 +0000 (08:05 +0100)]
buildpackages: drop suse-specific code in ceph-release.spec

The ceph-release RPM is not used on SUSE.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: ensure ceph.spec has correct Version and Release
Nathan Cutler [Fri, 2 Mar 2018 23:19:19 +0000 (00:19 +0100)]
buildpackages: ensure ceph.spec has correct Version and Release

In the buildpackages task we don't care what the spec file thinks the Version
and Release should be. We force it to be what we need.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: ensure spec file has correct tarball name
Nathan Cutler [Fri, 2 Mar 2018 21:01:17 +0000 (22:01 +0100)]
buildpackages: ensure spec file has correct tarball name

We are using this script to build various versions of ceph.spec.in, which do
various things for different build environments. Ignore whatever the spec file
is trying to do with the tarball name and force it to the name of the tarball
at hand.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
5 years agobuildpackages: make-rpm.sh: tweak rpmbuild options
Nathan Cutler [Thu, 1 Mar 2018 21:08:03 +0000 (22:08 +0100)]
buildpackages: make-rpm.sh: tweak rpmbuild options

Signed-off-by: Nathan Cutler <ncutler@suse.com>