kernel: add rpm: key

Add 'rpm:' key to not confuse people with stuff like

  kernel:
    deb: /path/to/foo.rpm

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: maybe_generate_initrd_rpm()

gitbuilder and local packages may not have initrd hooks in them,
depending on how old is the kernel - an appropriate %post section was
added to upstream spec only in 3.12.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: get_latest_image_version_{rpm, deb}()

A bit of refactoring to make things more clear and align distro case
with the rest of the code.  get_version_from_pkg() is renamed to
get_latest_image_version_deb() and get_latest_image_version_rpm() is
introduced to match it.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: local rpm support

install_kernel() now takes a path to a package in /tmp and can use
get_image_version() to obtain image version.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: get_image_version()

Use 'dpkg-deb -c' for debs instead of rolling our own.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: download_kernel() shouldn't install kernels

Stop using rpm http url capabilities and download packages ourselves.
This keeps the structure the same for both rpms and debs and makes it
possible to fit in local rpm support.  Drop the "rpm already installed"
check for the same reasons.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: fix 'sha1: distro' vs per role configs

We may want to setup different kernels for different roles (as long as
all roles on the same node agree on their choice), so returning as soon
as 'distro' is seen is not right.  Also, no need to use find() as
'distro' is always an exact match.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: gitbuilder_pkg_name(), remote_pkg_path()

In preparation for local rpm support, encapsulate package naming.
Using sudo to rm copied/downloaded package is not necessary.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: get_sha1_from_pkg_name()

Move package name parsing into its own routine and make the regex more
robust:
  - match only hex chars
  - don't limit ourselves to 7 chars, match up to 40 chars
  - accept 'g' prefixed with either _ (rpm) or - (deb)

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: fix commit hash matching in need_to_install(), take 2

Commit 10d7f9095fbb ("kernel: fix commit hash matching in
need_to_install()") switched startswith() around to accommodate
a shorter "want" version.  That however broke the opposite case: when
installing from gitbuilders "want" is actually a full 40-char hash.
Do away with starswith() and just compare substrings.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Minor change for distro kernel bug.

forgot to break from the loop. I guess I was mainly testing git
kernels and not distro when I did this before.

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

Merge pull request #380 from ceph/wip-kernel-fixes-rebase

Various bug-fixes with kernel installs.

Various bug-fixes with kernel installs.

Fixed a typo causing grub list to not incrament and other issues
with finding a reliable uname string via inspecting RPM directly.

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

Merge pull request #378 from ceph/wip-opsys

Add OS.codename

Merge pull request #379 from andrewschoen/worker-tests

increases test coverage on teuthology.worker

adds 50% test coverage to teuthology.worker without any changes to teuthology.worker itself

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Add OS.codename

The codename can be reliably set by parsing
platform.linux_distribution() or lsb_release -a. We now default to
trying the former first.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #375 from ceph/wip-dmick

A bunch of little fixes found during addition of calamari_nosetest task

Merge pull request #376 from ceph/wip-sha-compare-fix

kernel: fix commit hash matching in need_to_install()

Merge pull request #377 from ceph/wip-kill-fixes

Two minor teuthology-kill fixes

No need to invoke teuthology-nuke if there is nothing to nuke

Slipped in with 26803da18d1b ("Rewrite portions of teuthology.kill").

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Don't fetch owner info from job_info when --owner given

Commit 293c2bd49c0d ("Be more informative if owner info is missing")
introduced --owner but it is still pulled from job_info no matter what.
Fix it.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

kernel: fix commit hash matching in need_to_install()

We don't need to install a new kernel if the "want" (version) commit
hash and the "have" (cur_sha1) commit hash match but the former is
shorter than the latter.

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

misc.append_lines_to_file: respect sudo

Fixes: #10239
Signed-off-by: Dan Mick <dan.mick@redhat.com>

misc.create_file: if sudo, must sudo for chmod as well.

Fixes: #10238
Signed-off-by: Dan Mick <dan.mick@redhat.com>

misc.get_system_type(): fix distro=False version=True, add comments

Fixes: #10237
Signed-off-by: Dan Mick <dan.mick@redhat.com>

orchestra/run.py: document use of Raw()

Signed-off-by: Dan Mick <dan.mick@redhat.com>

packaging.py: Use misc.get_system_type()

Dump _get_relmap() and _RELEASE_MAP in favor of common code.
Also remove the repo management stuff; it was only used by the
now-obsolete calamari task.

Fixes: #10236
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Remove obsolete calamari task

Fixes: #10235
Signed-off-by: Dan Mick <dan.mick@redhat.com>

packaging.py: import misc as misc, not teuthology

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #374 from andrewschoen/orchestra-test

test improvements

moved all the cluster.write_file tests into their own class; rewrote the existing test to have one assertion per test case

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #373 from andrewschoen/orchestra-test

added a test for teuthology.orchestra.cluster.write_file

Refuse to schedule against missing suite branches

Signed-off-by: Zack Cerza <zack@redhat.com>

removed fudge and unneeded asserts

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

added a test for teuthology.orchestra.cluster.write_file

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #372 from ceph/wip-10034-wusui

Added calamari-server

Merge pull request #366 from dachary/wip-10170-write-file

fix Cluster::write_file extra argument & support owner

Added calamari-server

Calamari-server has been added to the list of installed packages
to be removed.  Also added salt-common and salt-minion to the list
of rpms to be removed.

Fixes #10034
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #371 from ceph/docs

Add autogenerated API docs to Sphinx output

Add autogenerated API docs to Sphinx output

Also drop -W for now. We'll have to focus on PEP-257 stuff later.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #369 from ceph/docs

Add lab setup docs

Reorganize documentation and Sphinx files

This should allow a more consistent experience between reading
Sphinx-generated HTML docs and just using GitHub's renderer.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add docs to tox envlist

So PRs don't break documentation

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove fancy source file handling

Signed-off-by: Zack Cerza <zack@redhat.com>

Add stub source files linking to canonical documents

Now we can drop the Makefile magic.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #360 from andrewschoen/docopt-ls

port teuthology-ls to use docopt instead of argparse

adding tests for teuthology.ls, coverage now at 88%

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Add LAB_SETUP.rst to Sphinx build

Signed-off-by: Zack Cerza <zack@redhat.com>

Add reference configs

Signed-off-by: Zack Cerza <zack@redhat.com>

Add lab setup doc

Signed-off-by: Zack Cerza <zack@redhat.com>

port teuthology-ls to use docopt instead of argparse

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Check individual jobs for packages

Previously we were just checking one time per run to see if packages had
been built. Now, let's check each job since os_type values can differ.
Fail if any are missing. Results are cached to avoid hugging the
archive to death.

Signed-off-by: Zack Cerza <zack@redhat.com>

Revert "ceph.conf: filestore fiemap = true"

This reverts commit 0f97481ce44e0487ac6cffa051a05590f907a081.

Until we fix #10166

Merge pull request #367 from ceph/wip-rgw-swift

swift: set full access to subusers creation

swift: set full access to subusers creation

Default subuser permissions are 'none'.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cluster: add owner argument to write_file

The misc.sudo_write_file now supports the optional owner argument, add
support for it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

cluster: remove extra write_file argument

http://tracker.ceph.com/issues/10170 Fixes: #10170

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #365 from ceph/wip-kill-pid-check

Verify running pid before passing it to be killed.

Verify running pid before passing it to be killed.

So we don't kill other processes that have taken jobs pid since
job has been no longer running.

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

ceph.conf: filestore fiemap = true

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #364 from ceph/wip-scheduling

Fix / work around gitbuilder problems

Choose SHA1 sums more intelligently

By checking the actual gitbuilder archive that we care about, if
--distro is passed. If it is not, tweak the default to 'rhel' instead of
'ubuntu' since that seems to produce better results. To be clear, this
is a workaround I'm putting in place since our gitbuilders never seem to
be in sync.

Signed-off-by: Zack Cerza <zack@redhat.com>

More useful logging when unlocking single nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

Mention test name when failing to schedule

Signed-off-by: Zack Cerza <zack@redhat.com>

Add a default results_sending_email

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #363 from ceph/wip-10089

drop default os_type of 'ubuntu' in teuthology-suite

Drop null values in placeholders

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #358 from andrewschoen/docopt-porting

porting teuthology-coverage to docopt

added a simple test for teuthology.coverage.main

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

added a simple test for teuthology.coverage.main

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

refactored teuthology.coverage to use an dict from docopt instead of a namespace from argparse

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Drop ubuntu as default distro for scheduled runs

Signed-off-by: Zack Cerza <zack@cerza.org>

added a test to make sure docopt was parsing things correctly for teuthology-coverage

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

cleaning up usage for teuthology-coverage

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

porting teuthology-coverage to docopt

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph.conf: mon pg warn max object skew = 0 (disable warning)

Make the 'too few pgs' warnings go away from qa runs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #357 from andrewschoen/tox-integration-tests

Moves all integration tests to an integration folder and provides a new ...

Merge pull request #356 from ceph/wip-default-machine-type

Allow setting default machine_type per-lab

Moves all integration tests to an integration folder and provides a new tox env to run only those tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Allow setting default machine_type per-lab

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #355 from ceph/wip-email

teuthology_results: add --dry-run

If timeout is None or zero, don't log

Signed-off-by: Zack Cerza <zack@redhat.com>

Add dry-run option

Signed-off-by: Zack Cerza <zack@redhat.com>

Use docopt instead of argparse

Signed-off-by: Zack Cerza <zack@redhat.com>

Use new config API

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #354 from andrewschoen/tox-cov-report

Adds a coverage report when running the tox tests

Adds a coverage report when running the tox tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #351 from ceph/wip-9925-wusui

Allow more strings in filters

Mention hostnames in ConnectionLostErrors

Signed-off-by: Zack Cerza <zack@cerza.org>

More suite filter changes.

Comment change, and update help message to reflect the ability to
use a comma separated list as a filter.

Merge pull request #352 from ceph/wip-update-vm-hostkeys

Continue fix for 9871

Merge pull request #353 from ceph/wip-9975

remove configuration files when nuking

use ctx instead of remotes

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Also look for /etc/teuthology.yaml

First we look for ~/.teuthology.yaml - if that is not found, we look for
a system-wide configuration in /etc/teuthology.yaml

Signed-off-by: Zack Cerza <zack@cerza.org>

Fix duplicate test name

Signed-off-by: Zack Cerza <zack@cerza.org>

call the new helper in nuke_helper

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

create a configuration removal function in nuke.py

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Don't explode if results_sending_email isn't set

Signed-off-by: Zack Cerza <zack@cerza.org>

Suppress tracebacks when coverage isn't configured

Signed-off-by: Zack Cerza <zack@cerza.org>