provision: add Pelagos support

Add provisioning support via Pelagos provisioner
https://github.com/SUSE/pelagos/

Pelagos is pxe boot and provisioning system which created especially
for connecting bare metal nodes to ceph/teuthology testing system.
Integration tests here pelagos/test_pelagos_teuthology/test_pelagos.py
because depends on executed Pelagios service

For enabling pelagos you need add section to teuthology configuration:
    pelagos:
      endpoint: http://your.server.host:5000/
      machine_types: ['type1', 'type2', 'type3']

provision/pelagos.py: added support of Pelagos provisioning project,
                      interface is compatible with FOG provisioner
provision/__init.py: added processing of pelagos section in teuthology
                     configuration and provisioner instantiation
lock/*: added Pelagos provisioner instantiation
nuke/__init__.py: added call to pelagos module for nodes, which are
                   controlled by pelagos, for booting to live images.

Signed-off-by: Roman Grigorev <rgrigorev@suse.de>

Merge pull request #1415 from smithfarm/wip-install-shaman-suse

packaging: remove zypper repo kludge

packaging: remove zypper repo kludge

Now that https://github.com/ceph/chacra/pull/275 has been merged and
deployed, we can write this code without any kludge.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1413 from smithfarm/wip-44187

task/kernel: after reboot, do not try to reconnect immediately

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #1412 from smithfarm/wip-43655

packaging: add Shaman repo correctly on opensuse

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #1411 from smithfarm/wip-opensuse-kernel-distro-2

kernel.py: fix -k distro for opensuse/sle (part 2)

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Luis Henriques <lhenriques@suse.com>

Merge pull request #1410 from badone/wip-smithi-lvm-scratch-disks

task/ceph_ansible: Work with LVM scratch devices

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

task/kernel: after reboot, do not try to reconnect immediately

Do not try to reconnect immediately after triggering the reboot,
because the reboot sequence might not have started yet (!)

Fixes: https://tracker.ceph.com/issues/44187
Signed-off-by: Nathan Cutler <ncutler@suse.com>

packaging: add Shaman repo correctly on opensuse

Fixes: https://tracker.ceph.com/issues/43655
Works Around: https://tracker.ceph.com/issues/44183
Signed-off-by: Nathan Cutler <ncutler@suse.com>

kernel.py: fix -k distro for opensuse/sle (part 2)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1408 from smithfarm/wip-kernel-distro-opensuse

task/kernel.py: fix -k distro for opensuse/sle (part 1)

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

task/ceph_ansible: Work with LVM scratch devices

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

task/kernel.py: fix -k distro for opensuse/sle

Part one - this does not get us all the way, but it's a start!

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1409 from kshtsk/wip-pump-six

requirements: pip-compile -P six

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1407 from kshtsk/wip-archive-dir

worker_start: allow ARCHIVE variable override

requirements: pip-compile -P six

We need to pump six to the latest 1.14.0 so we have
six.ensure_str() is available

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

worker_start: allow ARCHIVE variable override

The worker_start.sh script is using archive directory inside
the worker user home which cannot be changed, this patches
allows override ARCHIVE environment variable to address
this issue.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1405 from rzarzynski/wip-hammer-sh-do-not-nuke

hammer.sh: don't do teuthology-nuke anymore.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

hammer.sh: don't do teuthology-nuke anymore.

Apparently `hammer.sh` is pretty old. It calls `teuthology-nuke`
expecting the further call to `teuhology` will be able to still
reach the nodes. However, it doesn't seem that anybody is turning
them back these days.

The commit just drops the `teuthology-nuke` call.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1403 from badone/wip-ceph-ansible-cryptography-incompatible

ceph_ansible: Make sure cryptography version is compatible

Merge pull request #1402 from badone/wip-ceph-ansible-virtualenv-force-python3

ceph_ansible: Enforce python3 for virtualenv

Merge pull request #1401 from badone/wip-ceph-ansible-args-attribute-error

ceph_ansible: Make ceph_installer.run args consistent

ceph_ansible: Make sure cryptography version is compatible

Fixes: https://tracker.ceph.com/issues/43843
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

ceph_ansible: Enforce python3 for virtualenv

Fixes: https://tracker.ceph.com/issues/43799
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

ceph_ansible: Make ceph_installer.run args consistent

Fixes: https://tracker.ceph.com/issues/43798
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #1400 into master

* refs/pull/1400/head:
selinux: use text processing for audit.log

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

selinux: use text processing for audit.log

We ran into some odd grep bug treating the audit.log as binary and
wrongly returning a match result.

Fixes: https://tracker.ceph.com/issues/43797
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1369 from ryg-/wip-fix-opensuse-provisioning

Fix opensuse provisioning support

Merge pull request #1398 from smithfarm/wip-teuthology-branch-sha1

suite/run.py: provide teuthology hash along with branch

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #1399 from ceph/fix-cephadm-logs

orchestra/daemon/cephadmunit: use journalctl to monitor stdout/err

orchestra/daemon/cephadmunit: use journalctl to monitor stdout/err

Using 'podman logs' is racy because the command may start before the
container starts, or even after it start and exits.

Signed-off-by: Sage Weil <sage@redhat.com>

Fix opensuse provisioning support

task/selinux.py, orchestra/remote.py:
    disable SELinux for opensuse/suse distros

nuke/actions.py:
    added support for sle/opensuse distros

Signed-off-by: Roman Grigorev <rgrigorev@suse.de>

suite/run.py: provide teuthology hash along with branch

When running teuthology-suite with -t option, I see:

teuthology branch: wip-fix-opensuse-provisioning
ceph branch: wip-foo eb2fd5065c6e489ca8dd9282ba62ab231f144f7a

In other words, the ceph branch message has the SHA1 but the
teuthology branch message does not. This commit adds it.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1396 from tchaikov/wip-podman-selinux

task/selinux: ignore SElinux denials caused by podman

Reviewed-by: Sage Weil <sage@redhat.com>

task/selinux: ignore SElinux denials caused by podman

it's a workaround of https://tracker.ceph.com/issues/43635

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1394 from SUSE/bootstrap-setuptools-44

bootstrap: install setuptools 44

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1395 from ceph/fix-sysctl-core

tasks/internal: put kernel.core_pattern in sysctl.conf too

tasks/internal: put kernel.core_pattern in sysctl.conf too

This way if systemd restarts  or is reinstalled (as it is on el8 at the
moment) we won't lose this setting.

Signed-off-by: Sage Weil <sage@redhat.com>

bootstrap: install setuptools 44

With setuptools 45 python2 support was dropped.

Fixes: https://tracker.ceph.com/issues/43606
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #1383 from toabctl/worker-drop-dead-code

worker: Drop dead code handling old releases

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1391 from batrick/i43396

task/selinux: ignore recent rhel8 selinux denials

Reviewed-by: Kefu Chai <kchai@redhat.com>

task/selinux: ignore recent rhel8 selinux denials

Fixes: https://tracker.ceph.com/issues/43396
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1390 from ceph/wip-centos8

orchestra/daemon/cephadmunit: implement kill, wait

orchestra/daemon/cephadmunit: implement kill, wait

These are both needed by thrashosds.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1387 from tchaikov/wip-requires-paramiko

setup.py: move more modules back to "install_requires"

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

setup.py: move paramiko back to "install_requires"

move following python modules back to "install_requires"

* apache-libcloud
* netaddr
* paramiko

they are imported even when teuthology is used by vstart_runner.py

Signed-off-by: Kefu Chai <kchai@redhat.com>

worker: Drop dead code handling old releases

We no longer need to check for branches like 'argonaut', 'bobtail',
'cuttlefish' or 'dumpling'. These are long gone.
Removing the code simplifies it a bit.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #1388 from tchaikov/wip-fix-could-not-find-an-image

provision: fix the error message when a distro image is not found

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

provision: fix the error message when a distro image is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1386 from ceph/fix-43335

task/kernel: fix typo

task/kernel: fix typo

menuentry, not menuitem.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1385 from shangdehao1/master

task/install: fix UnboundLocalError of local variable repos

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1382 from kshtsk/wip-cast-dict-keys

py3: cast dict keys to list

Merge pull request #1384 from ceph/wip-centos8

task/install/daemon-helper: python3

task/install: fix UnboundLocalError of local variable repos

If overrides is None, repos will don't be assignmented.
In this case, will occur error "UnboundLocalError: local variable
repos referenced before assignment"

Signed-off-by: dehao.shang@intel.com

task/install/daemon-helper: python3

Signed-off-by: Sage Weil <sage@redhat.com>

test/fake_fs: fix tests for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

test: replace __builtin__ with module path for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

lock/ops: cast dict_keys to list for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/internal/syslog: use remote.sh for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/iscsi: use remote.sh instead of remote.run for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox: add flake8-py3 env

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/internal/vm_setup: get rid of cString import for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/connection: cast 'key' to bytes for base64.decodestring

Because base64.decodestring accepts bytes, the key variable
should be converted from string obj.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

worker: give a text file to yaml.safe_dump for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

py3: cast dict_keys to list

Addresses:

  TypeError: 'dict_keys' object does not support indexing

and:

  RuntimeError: dictionary changed size during iteration

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

prune: simplify expression for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #1378 from ceph/wip-centos8

task/kernel,install: misc fixes for el8

task/kernel: don't fail if we're confused by grub

If we don't have menuitem in the grub conf, and we dn't see /boot/loader/entries
file, just don't do grub2-set-default at all and hope for the best.  Worst
case we'll fail later because we didn't boot the right kernel.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1380 from kshtsk/wip-py3-requirements

py3 requirements

task/kernel: adapt to el8 grub2 /boot/loader/entries/*.conf

Signed-off-by: Sage Weil <sage@redhat.com>

task/kernel: drop removal of old kernels

1- this isn't necessary anymore with fog
2- the package-cleanup --oldkernels option doesn't exist in el8

Signed-off-by: Sage Weil <sage@redhat.com>

task/kernel: fix grub2 detection

grub2 rpm doesn't exist... el8 has

grub2-tools-extra-2.02-66.el8.x86_64
grub2-pc-modules-2.02-66.el8.noarch
grub2-tools-2.02-66.el8.x86_64
grub2-pc-2.02-66.el8.x86_64
grubby-8.40-34.el8.x86_64
grub2-common-2.02-66.el8.noarch
grub2-tools-minimal-2.02-66.el8.x86_64

and el7 has

grub2-tools-minimal-2.02-0.76.el7.centos.x86_64
grub2-pc-2.02-0.76.el7.centos.x86_64
grub2-common-2.02-0.76.el7.centos.noarch
grub2-tools-extra-2.02-0.76.el7.centos.x86_64
grubby-8.28-25.el7.x86_64
grub2-pc-modules-2.02-0.76.el7.centos.noarch
grub2-tools-2.02-0.76.el7.centos.x86_64
grub2-2.02-0.76.el7.centos.x86_64

The difference seems to be that el7 *also* has a /boot/grub directory. Let's
hope this change doesn't break it!

Signed-off-by: Sage Weil <sage@redhat.com>

task/install: make sure /etc/yum/plugin.conf.d/priorities.conf exists

If it doesn' teh subsequent commands all fail.

Signed-off-by: Sage Weil <sage@redhat.com>

misc: get_system_type: RedHatEnterprise is RHEL

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1381 from ceph/wip-cephadm

orchestra: ceph-daemon -> cephadm

orchestra: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

tox: use requirements3.txt for py3 env

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox: requirements.txt -> requirement2.txt

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

requirements: pip-compile -P paramiko

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

requirements: pip-compile -P cryptography

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

requirements: pip-compile -P pexpect

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

requirements: pip-compile -P pip-tools

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

py3: add requirements file

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstrap: use separate requirements file for py3

Because pip-compile produces different set of packages
for different python versions it needs to have
a dedicated requirements.txt for py2 and py3.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox: add py3 env

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstrap: add PYTHON and VENV variables

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1379 from tchaikov/wip-43192

setup.py: add missing dependencies of tests

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

setup.py: add missing dependencies of tests

Fixes: https://tracker.ceph.com/issues/43192
Signed-off-by: Kefu Chai <kchai@redhat.com>

docs: document the fix for incompatible pip-tools with newer pip

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: move more tests related deps to "test"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1375 from ceph/wip-centos8

orchestra/opsys: enable centos 8

orchestra/opsys: enable centos 8

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1374 from ceph/wip-valgrind-file

valgrind.supp is used from ceph/qa

valgrind.supp is used from ceph/qa, we should remove this from setup to avoid
any confusion

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1373 from kshtsk/wip-opensuse-distro-defaults

suite/util: add opensuse to distro defaults

suite/util: add opensuse to distro defaults

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1367 from tchaikov/wip-deps

update/cleanup dependencies

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

docs: document how requirements.txt is updated

we should update this document once
https://github.com/jazzband/pip-tools/issues/625 is fixed and/or
https://github.com/jazzband/pip-tools/issues/908 is merged

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: move test related stuff to optional deps

all of them are for testing teuthology itself.

Signed-off-by: Kefu Chai <kchai@redhat.com>