requirements,setup: add ipy module for cephfs mount.py

The IPy module is a tool for handling of IPv4 and IPv6 addresses and
networks, the cephfs mount.py will use it to manage the IP addresses
to isolate the netns for each mount, which could let us get ride of
using ipmi command to hard shutdown the client node for some test cases.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1423 from ishanrai05/bootsrap-linuxmint

Add bootstrap support for LinuxMint

Add bootstrap support for LinuxMint

Signed-off-by: ishanrai05 <ishanrai05@gmail.com>

Merge pull request #1429 from hualongfeng/master

orchestra: add ed25519key keytype to connection

Merge pull request #1428 from ceph/debug-mgr

suite: debug mgr by default

Merge pull request #1431 from kshtsk/fix-kernel-args

task/kernel.py: should pass args with name 2

Reviewed-by: Kefu Chai <kchai@redhat.com>

task/kernel.py: should pass args with name 2

There is leftover, there should be remote.sh used instead of remote.run

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1430 from tchaikov/wip-run-with-args

task/kernel.py: should pass args with name

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

task/kernel.py: should pass args with name

to address the failure of

```
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 1326, in task
    install_and_reboot(ctx, need_install)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 505, in install_and_reboot
    install_kernel(role_remote, remote_pkg_path(role_remote))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 834, in install_kernel
    version = get_image_version(remote, path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 1013, in get_image_version
    files = remote.run(['rpm', '-qlp', path])
TypeError: run() takes exactly 1 argument (2 given)
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

orchestra: add ed25519key keytype to connection

Signed-off-by: Feng Hualong <hualong.feng@intel.com>

Merge pull request #1362 from kshtsk/wip-py3-compat

py3 compatibility

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

suite: debug mgr by default

Like we do the mon and osd.

Signed-off-by: Sage Weil <sage@redhat.com>

orchestra/run: fix string handling

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1425 from ceph/avx

valgrind: disable openssl's use of avx instructions

valgrind: set OPENSSL_ia32cap to ~0x1000000000000000

This disables AVX instructions, which confuse valgrinda nd openssl.

https://tracker.ceph.com/issues/44362

Signed-off-by: Sage Weil <sage@redhat.com>

suite/util: add debug log for sha1s response

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

py3: get rid of cStringIO

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

py3: fix teuthology version banner for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/kernel: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox: enable py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox: enable flake8-py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/ansible: fix mock issue in test_ansible

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

misc: get_file must use mode='rb' for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/daemon/systemd: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstrap: use python version based required packages

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

packaging: make sure _get_repo returns str

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1419 from toabctl/drop-coverage-cmd

Drop teuthology-coverage command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1421 from kshtsk/wip-cluster-sh

orchestra/cluster: add sh() method

Merge pull request #1420 from kshtsk/wip-pump-markupsafe

requirements: bump up markupsafe to 1.1.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

requirements: bump up markupsafe to 1.1.1

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/cluster: add sh() method

For simplified outputs processing it might be usefull
to have a function will return all captured log from
all remote hosts in the given sequence.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Drop teuthology-coverage command

It seems to be unused and there was no change in the last 6 years. I
guess it's not used and not needed so drop it to remove the dead
code.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

teuthology/misc: pass --vgdb=yes to valgrind

This lets us attach to the process later.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1372 from ryg-/wip-suse-hw-provisioning

provision: add Pelagos support

provision: add Pelagos support

Add provisioning support via Pelagos provisioner
https://github.com/SUSE/pelagos/

Pelagos is pxe boot and provisioning system which created especially
for connecting bare metal nodes to ceph/teuthology testing system.
Integration tests here pelagos/test_pelagos_teuthology/test_pelagos.py
because depends on executed Pelagios service

For enabling pelagos you need add section to teuthology configuration:
    pelagos:
      endpoint: http://your.server.host:5000/
      machine_types: ['type1', 'type2', 'type3']

provision/pelagos.py: added support of Pelagos provisioning project,
                      interface is compatible with FOG provisioner
provision/__init.py: added processing of pelagos section in teuthology
                     configuration and provisioner instantiation
lock/*: added Pelagos provisioner instantiation
nuke/__init__.py: added call to pelagos module for nodes, which are
                   controlled by pelagos, for booting to live images.

Signed-off-by: Roman Grigorev <rgrigorev@suse.de>

Merge pull request #1404 from gekios/wip-libcloud_update

use libcloud 2.8.0 version

use libloud 2.8.0 version

Suggesting using 2.8.0 version of apache-libcloud that introduces
a fix on openstack driver.
(https://github.com/apache/libcloud/pull/1367 is fixing issue
https://github.com/apache/libcloud/issues/1365)

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>

Merge pull request #1415 from smithfarm/wip-install-shaman-suse

packaging: remove zypper repo kludge

packaging: remove zypper repo kludge

Now that https://github.com/ceph/chacra/pull/275 has been merged and
deployed, we can write this code without any kludge.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1413 from smithfarm/wip-44187

task/kernel: after reboot, do not try to reconnect immediately

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #1412 from smithfarm/wip-43655

packaging: add Shaman repo correctly on opensuse

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #1411 from smithfarm/wip-opensuse-kernel-distro-2

kernel.py: fix -k distro for opensuse/sle (part 2)

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Luis Henriques <lhenriques@suse.com>

Merge pull request #1410 from badone/wip-smithi-lvm-scratch-disks

task/ceph_ansible: Work with LVM scratch devices

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

task/kernel: after reboot, do not try to reconnect immediately

Do not try to reconnect immediately after triggering the reboot,
because the reboot sequence might not have started yet (!)

Fixes: https://tracker.ceph.com/issues/44187
Signed-off-by: Nathan Cutler <ncutler@suse.com>

packaging: add Shaman repo correctly on opensuse

Fixes: https://tracker.ceph.com/issues/43655
Works Around: https://tracker.ceph.com/issues/44183
Signed-off-by: Nathan Cutler <ncutler@suse.com>

kernel.py: fix -k distro for opensuse/sle (part 2)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1408 from smithfarm/wip-kernel-distro-opensuse

task/kernel.py: fix -k distro for opensuse/sle (part 1)

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

task/ceph_ansible: Work with LVM scratch devices

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

task/kernel.py: fix -k distro for opensuse/sle

Part one - this does not get us all the way, but it's a start!

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1409 from kshtsk/wip-pump-six

requirements: pip-compile -P six

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1407 from kshtsk/wip-archive-dir

worker_start: allow ARCHIVE variable override

requirements: pip-compile -P six

We need to pump six to the latest 1.14.0 so we have
six.ensure_str() is available

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

worker_start: allow ARCHIVE variable override

The worker_start.sh script is using archive directory inside
the worker user home which cannot be changed, this patches
allows override ARCHIVE environment variable to address
this issue.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1405 from rzarzynski/wip-hammer-sh-do-not-nuke

hammer.sh: don't do teuthology-nuke anymore.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

hammer.sh: don't do teuthology-nuke anymore.

Apparently `hammer.sh` is pretty old. It calls `teuthology-nuke`
expecting the further call to `teuhology` will be able to still
reach the nodes. However, it doesn't seem that anybody is turning
them back these days.

The commit just drops the `teuthology-nuke` call.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1403 from badone/wip-ceph-ansible-cryptography-incompatible

ceph_ansible: Make sure cryptography version is compatible

Merge pull request #1402 from badone/wip-ceph-ansible-virtualenv-force-python3

ceph_ansible: Enforce python3 for virtualenv

Merge pull request #1401 from badone/wip-ceph-ansible-args-attribute-error

ceph_ansible: Make ceph_installer.run args consistent

ceph_ansible: Make sure cryptography version is compatible

Fixes: https://tracker.ceph.com/issues/43843
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

ceph_ansible: Enforce python3 for virtualenv

Fixes: https://tracker.ceph.com/issues/43799
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

ceph_ansible: Make ceph_installer.run args consistent

Fixes: https://tracker.ceph.com/issues/43798
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #1400 into master

* refs/pull/1400/head:
selinux: use text processing for audit.log

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

selinux: use text processing for audit.log

We ran into some odd grep bug treating the audit.log as binary and
wrongly returning a match result.

Fixes: https://tracker.ceph.com/issues/43797
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1369 from ryg-/wip-fix-opensuse-provisioning

Fix opensuse provisioning support

Merge pull request #1398 from smithfarm/wip-teuthology-branch-sha1

suite/run.py: provide teuthology hash along with branch

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #1399 from ceph/fix-cephadm-logs

orchestra/daemon/cephadmunit: use journalctl to monitor stdout/err

orchestra/daemon/cephadmunit: use journalctl to monitor stdout/err

Using 'podman logs' is racy because the command may start before the
container starts, or even after it start and exits.

Signed-off-by: Sage Weil <sage@redhat.com>

Fix opensuse provisioning support

task/selinux.py, orchestra/remote.py:
    disable SELinux for opensuse/suse distros

nuke/actions.py:
    added support for sle/opensuse distros

Signed-off-by: Roman Grigorev <rgrigorev@suse.de>

suite/run.py: provide teuthology hash along with branch

When running teuthology-suite with -t option, I see:

teuthology branch: wip-fix-opensuse-provisioning
ceph branch: wip-foo eb2fd5065c6e489ca8dd9282ba62ab231f144f7a

In other words, the ceph branch message has the SHA1 but the
teuthology branch message does not. This commit adds it.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1396 from tchaikov/wip-podman-selinux

task/selinux: ignore SElinux denials caused by podman

Reviewed-by: Sage Weil <sage@redhat.com>

task/selinux: ignore SElinux denials caused by podman

it's a workaround of https://tracker.ceph.com/issues/43635

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1394 from SUSE/bootstrap-setuptools-44

bootstrap: install setuptools 44

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1395 from ceph/fix-sysctl-core

tasks/internal: put kernel.core_pattern in sysctl.conf too

tasks/internal: put kernel.core_pattern in sysctl.conf too

This way if systemd restarts  or is reinstalled (as it is on el8 at the
moment) we won't lose this setting.

Signed-off-by: Sage Weil <sage@redhat.com>

bootstrap: install setuptools 44

With setuptools 45 python2 support was dropped.

Fixes: https://tracker.ceph.com/issues/43606
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #1383 from toabctl/worker-drop-dead-code

worker: Drop dead code handling old releases

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1391 from batrick/i43396

task/selinux: ignore recent rhel8 selinux denials

Reviewed-by: Kefu Chai <kchai@redhat.com>

task/selinux: ignore recent rhel8 selinux denials

Fixes: https://tracker.ceph.com/issues/43396
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1390 from ceph/wip-centos8

orchestra/daemon/cephadmunit: implement kill, wait

orchestra/daemon/cephadmunit: implement kill, wait

These are both needed by thrashosds.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1387 from tchaikov/wip-requires-paramiko

setup.py: move more modules back to "install_requires"

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

setup.py: move paramiko back to "install_requires"

move following python modules back to "install_requires"

* apache-libcloud
* netaddr
* paramiko

they are imported even when teuthology is used by vstart_runner.py

Signed-off-by: Kefu Chai <kchai@redhat.com>

worker: Drop dead code handling old releases

We no longer need to check for branches like 'argonaut', 'bobtail',
'cuttlefish' or 'dumpling'. These are long gone.
Removing the code simplifies it a bit.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #1388 from tchaikov/wip-fix-could-not-find-an-image

provision: fix the error message when a distro image is not found

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

provision: fix the error message when a distro image is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1386 from ceph/fix-43335

task/kernel: fix typo

task/kernel: fix typo

menuentry, not menuitem.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1385 from shangdehao1/master

task/install: fix UnboundLocalError of local variable repos

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1382 from kshtsk/wip-cast-dict-keys

py3: cast dict keys to list

Merge pull request #1384 from ceph/wip-centos8

task/install/daemon-helper: python3

task/install: fix UnboundLocalError of local variable repos

If overrides is None, repos will don't be assignmented.
In this case, will occur error "UnboundLocalError: local variable
repos referenced before assignment"

Signed-off-by: dehao.shang@intel.com

task/install/daemon-helper: python3

Signed-off-by: Sage Weil <sage@redhat.com>

test/fake_fs: fix tests for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test: replace __builtin__ with module path for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/ops: cast dict_keys to list for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/internal/syslog: use remote.sh for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/iscsi: use remote.sh instead of remote.run for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox: add flake8-py3 env

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/internal/vm_setup: get rid of cString import for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/connection: cast 'key' to bytes for base64.decodestring

Because base64.decodestring accepts bytes, the key variable
should be converted from string obj.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>