code changes for multiple rgw daemons in single host

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1442 from sunilkumarn417/rh

updated config error exception w.r.t rhbuild

updated config error exception w.r.t rhbuild

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1441 from ceph/add-extra-packages

extra_packages and extra_system_packages added to redhat install task

extra_packages and extra_system_packages added to redhat install task

Signed-off-by: rakeshgm <rgowdege@redhat.com>

Merge pull request #1440 from sunilkumarn417/rh

added rhbuild attribute for redhat task and changes to retrieve correct daemon pid.

code changes to retrieve correct daemon pid

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

added rhbuild attribute

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1435 from ceph/add-extra-packages

teuthology/task/install

added extra_packages and extra_system_packages in rh install task

Signed-off-by: rakeshgm <rgowdege@redhat.com>

update apache-libcloud to 2.8.1

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

workaround needed for downstream rhos-d openstack

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

misc: get_system_type: RedHatEnterprise is RHEL

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1393 from ceph/dbench_workaround

workaround to install dbench for rhel-8

workaround to install dbench for rhel-8

Signed-off-by: rakeshgm <rakeshgm014@gmail.com>

task/install/daemon-helper: python3

Signed-off-by: Sage Weil <sage@redhat.com>

decreased the timeout

Signed-off-by: rakeshgm <rakeshgm014@gmail.com>

Merge pull request #1365 from ceph/rh-subscription

changed subscription username

changed subscription username

Signed-off-by: rakeshgm <rakeshgm014@gmail.com>

Merge pull request #1358 from ceph/wip-rh-subscription

skipping `yum udpate metadata` for rhel-8

skipping `yum udpate metadata` for rhel-8

Signed-off-by: rakeshgm <rakeshgm014@gmail.com>

Merge branch 'master' into rh_new

Signed-off-by: rakeshgm <rakeshgm014@gmail.com>

Merge pull request #1352 from ceph/WIP-mRemoteCopy

using put_file method instead of python shutil.copyfileobj so it works for python 2 and python 3 both

Merge pull request #1343 from ceph/dependabot/pip/ansible-2.8.2

build(deps): bump ansible from 2.8.0 to 2.8.2

using put_file method instead of python shutil.copyfileobj

Signed-off-by: Rachana Patel <racpatel@redhat.com>

Merge pull request #1350 from ceph/rhel-8-rpms

added rhel-8 appstream and baseos rpms for subscripion

added rhel-8 appstream and baseos rpms for subscripion

Signed-off-by: rakeshgm <rakeshgm014@gmail.com>

Merge pull request #1339 from kshtsk/wip-ghpr

repo_utils: support github pull request branches

disabling ceph packages checking for now

Signed-off-by: rakeshgm <rakeshgm014@gmail.com>

repo_utils: support github refs for branches

This adds possibility to checkout github pr providing
special purpose ref spec (like refs/pull/X/merge and
refs/pull/X/head) as branch parameter to teuthology,
for example, "teuthology-suite -t".

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge branch 'master' into rh_new

Merge pull request #1348 from kshtsk/fix-use-zypper-add-error

install/rpm: fix use the 'zypper addrepo' command

install/rpm: fix use the 'zypper addrepo' command

The 74034fe0348b94aedb247910f0d1c88842f4c3ac instroduced a failure:

  Warning: No repositories defined.
  Use the 'zypper addrepo' command to add one or more repositories.

The wipe command returns exit code 6 if there is no repos in the system.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge branch 'master' into rh_new

Merge pull request #1342 from toabctl/fix-test-with-newer-git

Fix fetch_branch() check for BranchNotFoundError with newer git

Merge pull request #1340 from toabctl/mention-openstack-setup

docs: Mention step to setup nodes on OpenStack

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1333 from smithfarm/wip-wipe-zypper-repos

install/rpm: wipe all zypper repos before adding repos-under-test

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge branch 'master' into rh_new

Merge pull request #1344 from kshtsk/wip-multiple-identityfile

connection: key_filename can be a list

connection: key_filename can be a list

This commit fixes 5ac7c2f and provides support for multiple identityfile,
besides paramiko.SSHClient.connect accepts list as well for key_filename,
pydoc excerpts:

key_filename:
    the filename, or list of filenames, of optional private key(s) to
    try for authentication

Fixes: https://tracker.ceph.com/issues/42543
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fix fetch_branch() check for BranchNotFoundError with newer git

With newer git versions, the string when a branch is not found
is lower case[1].
This fixes the test TestRepoUtils.test_fetch_branch_fake_branch which
was failing due to this problem.

[1] https://github.com/git/git/commit/0b9c3afdbfb629363

build(deps): bump ansible from 2.8.0 to 2.8.2

Bumps [ansible](https://github.com/ansible/community) from 2.8.0 to 2.8.2.
- [Release notes](https://github.com/ansible/community/releases)
- [Commits](https://github.com/ansible/community/commits)

Signed-off-by: dependabot[bot] <support@github.com>

docs: Drop duplicate openstack key pair creation step

It's already in the "Get OpenStack credentials and test it" section
documented how to create a key pair.

docs: Mention step to setup nodes on OpenStack

The --setup step was missing which lead to cryptic error messages when
running the following step (calling the dummy test suite).

timeout added

Signed-off-by: rakeshgm <rgowdege@redhat.com>

Merge branch 'master' into rh

Merge pull request #1338 from toabctl/fix-link-to-ceph-qa-suite

docs: Fix link to ceph/qa suites in intro_testers.rst

docs: Fix link to ceph/qa suites in intro_testers.rst

According to [1], ceph-qa-suite is deprecated (last commit is from
2016) and the QA suites are now in the ceph git repo in the qa/
directory. So point to that directory.

[1] https://github.com/ceph/ceph-qa-suite/

misc: add cwd and env to sh

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1335 from ceph/wip-wwn2

misc: temporary fix for "No space left on device" errors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

misc: temporary fix for "No space left on device" errors

41a13eca480e38cfeeba7a180b4516b90598c39b fixed a longstanding bug that the lab
was relying on. Before the bug was fixed, the get_wwn_id_map function was doing:

    try:
        r = remote.run(
            args=[
                'ls',
                '-l',
                '/dev/disk/by-id/wwn-*',
            ],
            stdout=StringIO(),
        )
        stdout = r.stdout.getvalue()
    except Exception:
        log.info('Failed to get wwn devices! Using /dev/sd* devices...')
        return dict((d, d) for d in devs)

The bug was that "remote.run" was putting single quotes around the string
"/dev/disk/by-id/wwn-*" because it wasn't enclosed in Raw(...). The single
quotes were causing the command to fail, triggering the except clause, and that
was happening 100% of the time.

The fix in 41a13eca480e38cfeeba7a180b4516b90598c39b caused the command to start
succeeding, which caused execution to continue. As a result, MON stores and
OSDs started getting created on the wrong devices, and tests that were
previously succeeding started to fail due to "No space left on device".

In short, the wwn devices on today's smithis are not big enough for
/var/lib/ceph.

This commit "fixes the fix" by dropping the dead code and always returning the
value that qa/tasks/ceph.py has come to expect.

Fixes: https://tracker.ceph.com/issues/42313
Signed-off-by: Nathan Cutler <ncutler@suse.com>

install/rpm: wipe all zypper repos before adding repos-under-test

Our tests should be designed to not rely on any pre-existing repos. If there are
any pre-existing repos, these can potentially skew test results by causing
unexpected packages/package versions to be installed.

Therefore, wipe any pre-existing repos before adding the repos-under-test.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1326 from kshtsk/wip-great-again

make teuthology-openstack great again

openstack: copy missing user-data from suse repo

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: cleanup stale comments

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

openstack: make use base64 for python3 compatible code

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

openstack: delete unused __flavor and __flavor_range methods

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

openstack: use --limit 2000 with openstack image list command

The --limit defaults to 25, which is way too low.

Fixes: https://github.com/SUSE/teuthology/issues/82
Signed-off-by: Loic Dachary <loic@dachary.org>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: only consider private images

OVH only returns the first 25 images, and the public ones (which we don't use
at all) eat up all the spots.

Fixes: https://github.com/SUSE/teuthology/issues/82
Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: increase ceph-fuse delays

In the same way it's done in machine_types/vps.yaml

Signed-off-by: Loic Dachary <loic@dachary.org>
Conflicts:
teuthology/openstack/openstack.yaml

setup-openstack.sh: setup_pulpito: respond to setuptools treachery

Apparently, setuptools has a "screw the users" policy.

Fixes: https://bugzilla.suse.com/show_bug.cgi?id=1023257
Signed-off-by: Nathan Cutler <ncutler@suse.com>

user-data: global s/sudo tee/tee/, if{up,down} -a

Signed-off-by: Nathan Cutler <ncutler@suse.com>

user-data: tweak Ubuntu 16.04 user-data

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: use wget instead of curl for Ubuntu-14.04

So that it also works with Ubuntu 14.04.1 which did not have curl by
default.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: support /etc/network/intefaces injection

Not all OpenStack providers have a dhcp server. Some of
them (dreamcompute for instance) may inject /etc/network/interfaces with
a pre-configured stanza. It is not enough to set the dns server in the
dhcp client configuration, it also needs to be set directly in
/etc/resolv.conf when that has no effect.

http://tracker.ceph.com/issues/14024 Fixes: #14024

Signed-off-by: Loic Dachary <loic@dachary.org>

ensure VMs always have /etc/hosts set up

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

buildpackages: use items for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

buildpackages: sleep first in the wait/backoff loop

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: change wait/backoff intervals

The sshd in SLE15 is more paranoid and does not tolerate much banging on port 22:

+ sleep 8
+ for delay in 1 2 4 8 8 8 8 8 8 8 8 8 16 16 16 16 16 32 32 32 64 128 256 512
+ ssh -o ConnectTimeout=3 137.74.26.189 bash -c '"grep -q READYTORUN /var/log/cloud-init*.log"'
Warning: Permanently added '137.74.26.189' (ECDSA) to the list of known hosts.
Received disconnect from 137.74.26.189 port 22:2: Too many authentication failures

Also log the ssh status code.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Cleanup to use merging from suse stream

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: user-data: install chrony on SLE15 and openSUSE 15

Signed-off-by: Nathan Cutler <ncutler@suse.com>

sle-15-user-data: copy authorized_keys from root

The image generation script is overriding the disable_root setting in
/etc/cloud/cloud.cfg. As a result, openstack seeds the key in
/root/.ssh/authorized_keys instead of /home/sles/.ssh/authorized_keys

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages: drop unneeded packages in SLE15

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Support SLE-15

With SLE15 it is no longer possible to SSH in as root. Instead, the OpenStack
SSH public key is injected into /home/sles. Reflect this in the user-data
files.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

TeuthologyOpenStack: use yaml.safe_load

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology-openstack: fix broken deployment

d6be711301fe7331acd428ab261ea648ef86305f broke the t-o deployment,
causing it to fail with the following Traceback:

2018-10-25 17:23:46,078.078 DEBUG:teuthology.misc:No server with a name
or ID of 'teuth-jschmid' exists.
Traceback (most recent call last):
  File "/home/jxs/projects/teuthology/teuthology/v/bin/teuthology-openstack", line 11, in <module>
    load_entry_point('teuthology', 'console_scripts', 'teuthology-openstack')()
  File "/home/jxs/projects/teuthology/teuthology/scripts/openstack.py", line 7, in main
    sys.exit(teuthology.openstack.main(parse_args(argv), argv))
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 1310, in main
    return TeuthologyOpenStack(ctx, teuth_config, argv).main()
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 704, in main
    self.setup()
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 890, in setup
    self.instance = OpenStackInstance(self.server_name())
  File "/home/jxs/projects/teuthology/teuthology/teuthology/openstack/__init__.py", line 82, in __init__
    if self.info['status'] == 'ERROR':
TypeError: 'NoneType' object has no attribute '__getitem__'

Signed-off-by: Joshua Schmid <jschmid@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/openstack/__init__.py

openstack: fail immediately if VM creation fails

Fixes: https://github.com/SUSE/teuthology/issues/159
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
teuthology/openstack/__init__.py

openstack: retry if volume create+attach operation fails

Once in awhile, a created volume fails to become available within
a reasonable period of time. When this happens, the entire test job
fails with the following error:

2018-10-02T06:52:44.106 ERROR:teuthology.provision.openstack:'volume
target192168000058-2' reached maximum tries (100) after waiting for 200
seconds

With this commit, we catch that exception and retry with a new volume.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@suse.com>
Conflicts:
teuthology/provision/openstack.py

provision/openstack: put "add volume" in a retry loop

Attempted workaround for this issue:

2017-03-23T12:33:20.684 DEBUG:teuthology.misc::sh: openstack --quiet volume show  -f json target217182140122-1
2017-03-23T12:34:22.140 DEBUG:teuthology.misc:{
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"size": 10,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"status": "available",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"properties": "ownedby='217.182.143.170'",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"user_id": "122b249e82994f5aa8ec6a31b7891df5",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"description": null,
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"availability_zone": "nova",
2017-03-23T12:34:22.146 DEBUG:teuthology.misc:"bootable": "false",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"encrypted": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"created_at": "2017-03-23T12:33:17.000000",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"multiattach": false,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:driver_data": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"name": "target217182140122-1",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"snapshot_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"consistencygroup_id": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"replication_status": "disabled",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-vol-tenant-attr:tenant_id": "43583fef7b734bb1b6fb5f86d48abe5e",
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"source_volid": null,
2017-03-23T12:34:22.147 DEBUG:teuthology.misc:"os-volume-replication:extended_status": null,
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"type": "classic",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"id": "e6949180-87fe-4f4c-b0a0-e3ea35bdb2e2",
2017-03-23T12:34:22.148 DEBUG:teuthology.misc:"attachments": []
2017-03-23T12:34:25.036 DEBUG:teuthology.misc:}
2017-03-23T12:34:25.042 DEBUG:teuthology.misc::sh: openstack server add volume target217182140122 target217182140122-1
2017-03-23T12:35:30.536 DEBUG:teuthology.misc:No volume with a name or ID of 'target217182140122-1' exists.
2017-03-23T12:35:34.512 ERROR:teuthology.provision.openstack:Command 'openstack server add volume target217182140122 target217182140122-1' returned non-zero exit status 1

Fixes: http://tracker.ceph.com/issues/17659
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: skip openstack tests if OS_AUTH_URL not set

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: hide OS_ variables dumping in logs

Hide OS_ variables in the log, so it can't be easily
read by human and steal secrets.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: Support GitHub PR merge branches

Add support for github pr merge branches, like 'origin/pr/*/merge',
so it can be used to setup teuthology server using teuthology-openstack
command line tool with --teuthology-branch argument.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

openstack: add get_instance method

Refactor teuthology-openstack so it get openstack instance once.
Allow to run suite without prior setup() invocation.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

openstack: fix pulpito and paddles status in init script

After deployment teuthology cluster on openstack, teuthology
service status-paddles and status-pulpito commands report
'dead', however paddles and pulpito are running.
This fix addresses this issue.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

teuthology-openstack: add banner after cluster creation

When creating a teuthology openstack cluster and not
requesting any suite suite run the teuthology-openstack
quits silently not giving information where the cluster
is created and how its services can be reached.
This fix addresses the issue above.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

teuthology-openstack: add nameserver option

Add --nameserver option to teuthology-openstack cli tool so
teuthology cluster can be deployed and configured to use
customly provided dns

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

buildpackages: use makecheck image to instantiate build VM

The teuthology images no longer have the right repos for building Ceph.
Use the makecheck images instead.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Fix ovh paddles populate

Unfortunate typo instroduced in sha1 e491969d4b5b9dca710d1ce89f0cde918bbee1b4
which fixed ECP nodes creation and broke OVH nodes creation.

Fix cannot populate paddles

When trying to deploy teuthology on openstack with several networks
and non unique subnet addresses setup-openstack script fails to add
nodes to paddles databases.

This patch removes duplicates while getting default_subnets.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

connection: key_filename is a string, not an array

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Add ECP support

Isolated __flavor and __flavor_range from openstack invocation.
Flavor and Network parameters can be taken from teuthology config
under 'openstack' records.
Added filters for new OVH flavors.

 Conflicts:
teuthology/openstack/__init__.py

openstack: Support Leap 42.3

Without Salt for now, due to https://bugzilla.opensuse.org/show_bug.cgi?id=1042379

Signed-off-by: Nathan Cutler <ncutler@suse.com>

openstack: cloudlab support (part 2)

The ctl hostname was replaced by a fqdn that always ends with cloudlab.us

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: do not use cached tokens for neutron

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: teuthology must use aarch64, not arm64

teuthology / gitbuilder / ansible_architecture arch is

   dpkg-architecture --query DEB_HOST_GNU_CPU (x86_64 and aarch64)

deb package Architectures: (conf/distributions) is

   dpkg-architecture --query DEB_HOST_GNU_ARCH (amd64 and arm64)

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: move net() to OpenStack

It is not only needed when provisionning the teuthology instance.

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: rework floating IP handling

Signed-off-by: Loic Dachary <loic@dachary.org>
Conflicts:
teuthology/openstack/__init__.py

openstack: there may be more than one IPv4 subnet

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: increase teuthology flavors

Ansible has higher memory requirements these days. Keep the default
flavor to the minimum because it is used during tests.

Signed-off-by: Loic Dachary <loic@dachary.org>

Add multi-user support for openstack

Conflicts:
requirements.txt
teuthology/misc.py
teuthology/openstack/__init__.py
teuthology/provision/openstack.py