use downstream.yaml to install packages and version check

remove hardcoded checks instead use downstream.yaml
defined in suites dir rh/downstream.yaml
for packages and version checks

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use rh/downstream.yaml to define packages

Remove hardcoded checks, use rh/downstream.yaml
from suite dir to define packages to install

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #983 from ceph/wip-local-kernel-regex

kernel: update get_sha1_from_pkg_name() regex

kernel: update get_sha1_from_pkg_name() regex

ceph-build kernel packages will be named -ceph-g${SHA1:0:12}.  As the
first step towards using ceph-build, change the local package regex.
With this, I'm able to download post ceph/ceph-build#540 packages and
install them as a local deb/rpm.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #980 from jcsp/wip-python-rgw

teuthology/nuke: update for python-rgw

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #981 from ceph/wip-unpin-ansible

Revert "Pin ansible to <2.2.0"

teuthology/nuke: update for python-rgw

Signed-off-by: John Spray <john.spray@redhat.com>

Revert "Pin ansible to <2.2.0"

This reverts commit f2576e7a719a129a3c8be603bd9993d2b1ec94f2.

We finally heeded all the deprecation warnings.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #979 from tchaikov/wip-remove-libcephfs

teuthology: update nuke for libcephfs2

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

teuthology: update nuke for libcephfs2

try to remove libcephfs2 as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #974 from ceph/wip-install-subpackage

Make the install task a subpackage

Reviewed-by: Dan Mick <dmick@redhat.com>

ShamanProject: add repo installation and removal

Signed-off-by: Zack Cerza <zack@redhat.com>

Move repo installation to GitbuilderProject

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: bdev debug aio = true

detect aio stalls

Signed-off-by: Sage Weil <sage@redhat.com>

Add NoRemoteError

Signed-off-by: Zack Cerza <zack@redhat.com>

Create a task.install.redhat submodule

This will hold most of the code specific to RHCS

Signed-off-by: Zack Cerza <zack@redhat.com>

Create a task.install.deb submodule

This will hold most of the code specific to Debian-based systems

Signed-off-by: Zack Cerza <zack@redhat.com>

Create a task.install.rpm submodule

This will hold most of the code specific to RPM-based systems

Signed-off-by: Zack Cerza <zack@redhat.com>

Create a task.install.util submodule

This will hold miscellaneous code that doesn't need to be top-level and
isn't distro-specific

Signed-off-by: Zack Cerza <zack@redhat.com>

Linter fixes

Signed-off-by: Zack Cerza <zack@redhat.com>

Make the install task a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #977 from ceph/wip-fix-vm

Move edit_sudoers.sh to task/internal/

Merge pull request #978 from SUSE/wip-fix-suse-user-data

openstack: fix location of network script in openSUSE user-data

openstack: fix location of network script in openSUSE user-data

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Move edit_sudoers.sh to task/internal/

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #975 from ceph/wip-install-excludes

install: apply exclude_packages globally

Reviewed-by: Dan Mick <dmick@redhat.com>

install: apply exclude_packages globally

Previously, we were only applying the setting for upgrades

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Call the check_lock function, not the module

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #823 from ceph/wip-internal

Task module refactoring

Reviewed-by: Dan Mick <dmick@redhat.com>

ceph.conf: mon allow pool delete = true

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #973 from dmick/wip-pull-directory

Change pull_directory to use a remote tar streaming to the local extract

Merge pull request #970 from ceph/wip-shaman-tags

ShamanProject: query tags

Reviewed-by: Dan Mick <dmick@redhat.com>

Use remote.get_tar_stream in misc.pull_directory

Avoids remote and local temp files, and subsequent read/write
thrashing of local dir.  Since some of those directories are
multiple tens or hundreds of gigabytes, the extra file load
is important.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Add orchestra.remote.get_tar_stream(); like get_tar but no tmpfile

Avoid creating a temporary file either on the remote or on the
local host; just open a connection to a remote tar and let 'er
rip.  Intended to be used by get_directory to avoid temp files
and local read/write hammering.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

misc.pull_directory: fix misplaced log message, remove useless continue

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Move task.internal.syslog to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Move internal.vm_setup to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Move internal.check_lock to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Move internal.lock_machines to its own module

Signed-off-by: Zack Cerza <zack@redhat.com>

Make the internal task a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

run_tasks: support tasks that are subpackages

Signed-off-by: Zack Cerza <zack@redhat.com>

Pin ansible to <2.2.0

Until we fix the deprecations

Signed-off-by: Zack Cerza <zack@redhat.com>

ShamanProject: translate tags to sha1s

Since Shaman doesn't want to have to know about tags, let's do the
translation from tag->sha1 ourselves using 'git ls-remote'.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #965 from ceph/wip-exiter

Don't leak console-logging processes when a job is killed

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #971 from ceph/wip-ceph-mgr

teuthology: update nuke for ceph-mgr

teuthology: update nuke for ceph-mgr

Signed-off-by: John Spray <john.spray@redhat.com>

Move suite.util.build_git_url to repo_utils

Signed-off-by: Zack Cerza <zack@redhat.com>

Add repo_utils.ls_remote()

... by moving most of suite.util.git_ls_remote's code to that module.

Signed-off-by: Zack Cerza <zack@redhat.com>

Ensure console loggers die when the job does

Signed-off-by: Zack Cerza <zack@redhat.com>

Use psutil.Popen instead of subprocess.Popen

psutil.Popen is identical other than offering nice features like making
sure a PID hasn't been reused before it sends signals to it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Revert "ceph.conf: debug monc = 1"

This reverts commit 5137d3c6cfd065c69fe7e3bb570ef0fc795febcb.

Merge pull request #967 from ceph/wip-fedora-python

downburst: Make sure python 2 is installed on Fedora

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #964 from ceph/wip-ssh-config

orchestra.connect: do not override user/host from ~/.ssh/config

Make sure python 2 is installed on Fedora

We need this for ansible.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #966 from ceph/wip-archive-default

Update the default archive_base path

Update the default archive_base path

None of our labs are using /var/lib/teuthworker/archive anymore. The
incorrect default could trip up users.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: debug monc = 1

Signed-off-by: Sage Weil <sage@redhat.com>

Add exit module

This module contains a helper to add functions to be run when a given
signal is received. These signal handlers ought to do as little as
possible to avoid interfering with the process shutting down.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.connect: do not override user/host from ~/.ssh/config

I don't know why this was ever the case, but ignoring user@host
from the targets list just seems like a bad idea, and causes
problems, because tasks/ansible/etc. assume that the user to use
is ubuntu.  If that's not what .ssh/config says, there's just
going to be trouble...but there's no reason to pay attention
to .ssh/config there; if you really want to play tricks, edit
your target list accordingly.

Also, the original code was apparently not prepared for the
key_filename to be a list, which it apparently always is,
at least now; handle that too.

Fixes: http://tracker.ceph.com/issues/17125
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #963 from ceph/wip-rerun

Add rerun capability to teuthology-suite

Reviewed-by: Dan Mick <dmick@redhat.com>

suite: Add rerun capability

Using a new --rerun argument for teuthology-suite, optionally paired
with --rerun-statuses, we can now ask teuthology to re-schedule a
suite-run. The main use-case for the feature is to re-schedule only
those jobs from a given run which failed.

Fixes: #10945
http://tracker.ceph.com/issues/10945

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ResultsReporter.get_run()

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: parse filter args into lists sooner

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #951 from ceph/wip-shaman-2

Support fetching packages from shaman instead of gitbuilder

Reviewed-by: Dan Mick <dmick@redhat.com>

ceph.conf: debug auth on mon

Tracking down an intermitted EPERM from mon... I think related to
rotating keys.

See http://tracker.ceph.com/issues/17495

Signed-off-by: Sage Weil <sage@redhat.com>

GitbuilderProject: remove 'v' from version values

Some version values have a leading 'v', e.g.:
http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/v10.2.0/version
is 'v10.2.0', whereas:
http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel/version
is '10.2.3-0'.

Signed-off-by: Zack Cerza <zack@redhat.com>

ShamanProject: correct ref precedence

Signed-off-by: Zack Cerza <zack@redhat.com>

More properly test ref precedence for Shaman

Signed-off-by: Zack Cerza <zack@redhat.com>

ShamanProject: split out search URI generation

Signed-off-by: Zack Cerza <zack@redhat.com>

Move logic from _get_uri_reference...

Into new _choose_reference() method. This is because uri_reference is
irrelevant to ShamanProject, but we still want to use the same rules for
deciding whether to search for ref/tag/branch/sha1 when more than one is
specified.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add tests for GitbuilderProject._get_uri_reference

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix typo in GitbuilderProject._get_uri_reference()

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject: always set codename

When a GitbuilderProject (and also ShamanProject) was being init-ed from
a Remote object, we weren't setting the codename.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.install: use builder.codename for deb repos

Signed-off-by: Zack Cerza <zack@redhat.com>

GitbuilderProject: don't mangle rpm versions

Up until now, given a package like:
"ceph-test-11.0.0-2135.g8403c95.x86_64", we were returning "11.0.0",
which doesn't allow us to properly verify installed package versions.
Start returning the entire string provided by gitbuilder so that we can
ensure we match the value returned by packaging.get_package_version().

For .deb packages, we have always used string like:
"11.0.0-2135-g8403c95-1xenial"

Signed-off-by: Zack Cerza <zack@redhat.com>

get_package_version(): return V-R for rpm

Even though strictly speaking "version" implies just the %{VERSION}
field, that is hardly ever useful on its own.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal: use get_builder_project()

Signed-off-by: Zack Cerza <zack@redhat.com>

run: Allow a job to override config.use_shaman

Signed-off-by: Zack Cerza <zack@redhat.com>

suite tests: use GitbuilderProject and not ShamanProject

Signed-off-by: Zack Cerza <zack@redhat.com>

task.install: use get_builder_project()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add and use get_builder_project()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ShamanProject

Signed-off-by: Zack Cerza <zack@redhat.com>

Move much of TestGitbuilderProject into base class

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #960 from ceph/wip-17391

teuthology-nuke: lazy-unmount osd mount points

Reviewed-by: Dan Mick <dmick@redhat.com>

remove_osd_mounts(): do lazy unmounts

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Add unlock_firmware_repo()

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: move logging calls, and fix linter warnings

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: remove ceph data later

Signed-off-by: Zack Cerza <zack@redhat.com>

valgrind.supp: tweak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #406 from ceph/wip-journal-zero

ceph.conf: journal_zero_on_create = true

ceph.conf: journal_zero_on_create = true

Make sure we test this code path.

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind.supp: another rocksdb leak

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind.supp: a rocksdb Env leak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #958 from ceph/wip-reweight-config

ceph.conf: set min rweight min bytes per osd = 10

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.conf: set min rweight min bytes per osd = 10

This lets the reweight-by-utilization command work even after mons are
restarted (ceph_manager is currently setting this via injectargs, but that
is not persistent).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #953 from rjfd/wip-opensuse-pr

OpenSUSE Leap support in teuthology

install: use --capability with zypper {install,remove}

Otherwise we pick up python-ceph 0.80.9 from the distro repo instead of
a newer python-ceph-compat from the test repo.

References: https://bugzilla.opensuse.org/show_bug.cgi?id=997751
Signed-off-by: Nathan Cutler <ncutler@suse.com>

install: opensuse: support for zypper package manager

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #949 from dachary/wip-openstack-token-endpoint

openstack: use OS_AUTH_TYPE=token_endpoint when possible

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #956 from ceph/wip-nodeless

Set ctx.cluster even when a job has no roles

Reviewed-by: Dan Mick <dmick@redhat.com>