Merge pull request #475 from ceph/wip-newtasks

New-style tasks

Add unit tests for teuthology.task.Task

Signed-off-by: Zack Cerza <zack@redhat.com>

Add teuthology.task.Task

This is intended to be a base class providing a more modern, featured,
tested foundation for tasks. It has built-in overrides, host selection,
and optional skipping of the teardown stage.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #474 from ceph/wip-11466

Fix branch selection logic

Fix branch selection logic

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #469 from zhouyuan/add_missing_files

Adding missing files in setup.py

Merge pull request #473 from ceph/wip-rate-limit

Don't call get_branch_info() for master branches

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #471 from dmick/master

ssh_keys: save each remote's backup file name separately in a dict for u...

Don't call get_branch_info() for master branches

We're hitting the github API rate limit; if master doesn't exist we lose
anyway

Signed-off-by: Zack Cerza <zack@redhat.com>

ssh_keys: save each remote's backup file name separately in a dict for undo

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #470 from ceph/wip-remote

Add timeouts to Remote connection functions

Split internal.connect() into two subtasks

One creates the ctx.cluster object; the other initiates connections

Signed-off-by: Zack Cerza <zack@redhat.com>

Pass timeout value down to connection.connect()

Signed-off-by: Zack Cerza <zack@redhat.com>

Add timeout arg to connect()

Default behavior is unchanged

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't call Remote.connect() in Remote.__init__()

Signed-off-by: Zack Cerza <zack@redhat.com>

Make Remote.reconnect() retry optionally

Signed-off-by: Zack Cerza <zack@redhat.com>

Properly close connection on object deletion

Signed-off-by: Zack Cerza <zack@redhat.com>

Adding missing files in setup.py

Some non-python scripts are missing when install.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #468 from dmick/wip-ssh-keys

ssh_keys: stop using sed, use saved backup of authorized_keys instead

ssh_keys: stop using sed, use saved backup of authorized_keys instead

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #466 from ceph/wip-11426

Log stderr in get_latest_image_version_deb()

Log stderr in get_latest_image_version_deb()

It was being suppressed needlessly. Also fix lots of linter issues.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #465 from ceph/kernel-koji-docs

Docs for using koji in the kernel task.

Docs for using koji in the kernel task.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #464 from ceph/wip-git.ceph.com

git.ceph.com

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #462 from ceph/wip-ssh-keys

When modifying authorized_keys, store a backup

When modifying authorized_keys, store a backup

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #460 from zhouyuan/mkdir_p

Make parent directories as needed

Merge pull request #463 from ceph/wip-suite-tests

Basic teuthology.suite.build_matrix() unit tests

Make test_gitbuilder_url() use the local config

Signed-off-by: Zack Cerza <zack@redhat.com>

Add unit tests for suite.build_matrix()

Signed-off-by: Zack Cerza <zack@redhat.com>

Parametrize build_matrix() for unit tests

Signed-off-by: Zack Cerza <zack@redhat.com>

Always call unlock_one() with the user arg

I'd seen a few cases where vps creation would fail, and then unlocking
failed because we were incorrectly guessing the owner of the lock.

Signed-off-by: Zack Cerza <zack@redhat.com>

Skip passing os_type/version when destroying

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #459 from ceph/wip-downburst

Refactor downburst provisioning

Add basic unit tests for Downburst

Signed-off-by: Zack Cerza <zack@redhat.com>

Make *_if_vm() more unit-testable

Allow passing in a custom Downburst instance. Also clean up several
linter issues.

Signed-off-by: Zack Cerza <zack@redhat.com>

Use provision.downburst_executable()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #461 from ceph/wip-update-inventory

Add teuthology-update-inventory command

Add teuthology-update-inventory command

This ought to also replace teuthology-updatekeys, but might not be as
fast yet.

Signed-off-by: Zack Cerza <zack@redhat.com>

Move downburst code into new Downburst class

Signed-off-by: Zack Cerza <zack@redhat.com>

Make parent directories as needed

with '-p' when mkdir

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Clarify error logging in destroy_if_vm()

Signed-off-by: Zack Cerza <zack@redhat.com>

Be smarter about logging in create_if_vm()

Also return False on failure.

Signed-off-by: Zack Cerza <zack@redhat.com>

Strip ssh-keyscan output before logging it

Signed-off-by: Zack Cerza <zack@redhat.com>

Be more tolerant of git connection errors

Signed-off-by: Zack Cerza <zack@redhat.com>

Be slightly smarter when looking for stale locks

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #458 from ceph/wip-11248

Disable version check when ceph-deploy is used to install ceph.

Disable version check when ceph-deploy is used to install ceph.

See: http://tracker.ceph.com/issues/11248

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

On BranchNotFoundError, mark job as dead

We had been simply deleting jobs from paddles if the teuthology branch
was not found. Instead, mark them as dead with a useful failure_reason.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #457 from ceph/wip-lttng-valgrind

valgrind: ignore lttng warning on un-cleaned-up state

valgrind: ignore lttng warning on un-cleaned-up state

This appears to have come from dca722ec7b2a7fc9214844ec92310074b5cb2faa,
which merged in support for use of lttng ust functions on fork. Valgrind
started warning on a Leak_StillReachable with lttng_ust_init in
the function call trace.

Hopefully this is specific enough that we don't mask out any other
lttng errors.

Fixes: #11247
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #456 from ceph/wip-11243

Verify version for other projects besides ceph in the install task

Verify version for other projects besides ceph in the install task

Fixes: http://tracker.ceph.com/issues/11243
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Avoid double-unlocking with unlock_on_failure

If both unlock_on_failure and nuke-on-error are set, don't unlock now
because we're just going to nuke (and unlock) later.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #451 from ceph/verify-version

Verify that the correct ceph version is installed on updates and install

Merge pull request #454 from ktdreyer/wip-comment-libvirt-bootstrap

bootstrap: explain the system python-libvirt pkg

bootstrap: explain the system python-libvirt pkg

In #sepia today, Zack explained the reasoning for the python-libvirt
package on the Debian family of distros.

Merge pull request #453 from ceph/wip-11170

Allow running teuthology-lock --update on broken vps nodes

Tweak ssh-keyscan timing

Make the call itself timeout after 1s instead of 5s. Make update_lock()
try 15 times, sleeping for 1s between attempts. A full timeout of
update_lock() should take ~30s.

Signed-off-by: Zack Cerza <zack@redhat.com>

Use safe_while instead of an unsafe loop

If ssh-keyscan fails, we would hang forever. Stop that nonsense, and
give up eventually, print a warning, and move on.

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix error formatting when 'action' is passed

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #452 from dmick/master

lock.py: allow machines list for --brief

lock.py: allow machines list for --brief

There was never any reason not to.

Fixes: #11167
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #424 from ceph/wip-5629

Download ELFs (regular + debug) from remotes when coredump occurs.

Use Remote.get_file()

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix some linter issues

Signed-off-by: Zack Cerza <zack@redhat.com>

Download ELFs (regular + debug) from remotes when coredump occurs.

Per issue #5629

Since we need the binaries (regular stripped + non-stripped with
debug symbols) to do debugging this will make it more convinient
for developers so they don't have to track down the files from
the specific build in question and instead they will just be
sitting in the archive folder with the coredump files.

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

Verify that the correct ceph version is installed on updates and install

This adds a function to teuthology.packaging that can be reused to
fetch the installed version of a package for the given remote.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #450 from ceph/wip-10989

Add optional exclude_packages arg to upgrade subtask

Add optional exclude_packages arg to upgrade

Signed-off-by: Zack Cerza <zack@redhat.com>

Smarter option handling in upgrade_common()

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix a couple bugs and mark down hosts as down

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't use yaml in FakeNamespace's str()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #449 from ceph/wip-repr

Don't use yaml in FakeNamespace's repr()

Merge pull request #441 from ceph/wip-10803

Bug #10803: Try apt-get -f install to workaround nuke failure.

Bug #10803: Try apt-get -f install to workaround nuke failure.

Sometimes packages or dpkg gets in a weird state where a nuke will
fail to run and simply running an apt-get -f install on the
machine will fix things up so it can be nuked again. Currently
we only ran that being dependent on dpkg --configure completing
where as I changed it so it will still try apt-get -f install
to potentially require less manual intervention when cleaning up
machines. I kept the original behavior but adding some arguments
to insure it will not prompt you for a yes/no (like apt-get -f
install usually would do).

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

Don't use yaml in FakeNamespace's repr()

Signed-off-by: Zack Cerza <zack@redhat.com>

install/nuke: add rbd-fuse package

This is now used by rbd's merge_diff test. rbd-fuse has been around
since before dumpling, so this is compatible with all current ceph
testing.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #447 from ceph/upgrade-issue

Do not specify a version when upgrading ceph on rpm-based systems.

Do not specify a version when upgrading ceph on rpm-based systems.

Because of the split of ceph-devel, upgrading ceph-devel with an
explicit version does not allow check_obsoletes to work.

For, http://tracker.ceph.com/issues/10936

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #448 from ceph/hadoop-workunits

hadoop: support rhel and ubuntu

hadoop: support rhel and ubuntu

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #427 from ceph/travisci

add Travis CI integration

Merge pull request #446 from ceph/wip-10926-andrew

Do not install packages with a specified version, fixes #10926

Do not install packages with a specified version, fixes #10926

Now that we've got yum setup to check for obsoletes we don't need to
specify a version. Specifying the version actually keeps yum from
checking for obsoletes.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #445 from ceph/hadoop

hadoop: clean-up and sigkill delay fix

hadoop: add yarn sigkill delay hack

This instructs yarn to wait more time (10sec) than the default (250ms)
before killing containers to give ceph clients time to properly
shutdown.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

hadoop: use dict-to-conf converter

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #443 from ceph/wip-10910

Allow Remote.get_file() to use original filename

Allow Remote.get_file() to use original filename

If dest_dir != '/tmp', attempt to use the original filename.

Signed-off-by: Zack Cerza <zack@redhat.com>

Update out-of-date integration test

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #442 from ceph/wip-7364

Don't fail just because apt-get update does

Don't fail just because apt-get update does

If we fail later anyway, fine. If not, yay!

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't fail if task runs twice

This was happening because _yum_unset_check_obsoletes() was failing

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #440 from ceph/wip-10893

Set check_obsoletes = 1 in yum's priorities.conf

Set check_obsoletes = 1 in yum's priorities.conf

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #439 from ceph/pytest-better-errors

put test failures in failure_reason; skip test_correct_os_version for debian

Make an exception for debian in tests.test_correct_os_version

This is because of a known issue where downburst gives us 7.1 when we
ask for 7.0.  We're ok with this behavior for now. See: issue #10878

Signed-off-by: Andrew Schoen <aschoen@redhat.com>