display informative error message if schedule fails to deep_merge

Display the two yaml structures that fail to merge in case of an
exception. That's a useful hint to find out what's wrong.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #342 from ceph/wip-5876

task/nfs: stop nfs server after test finishes

Reviewed-by: John Spray <john.spray@redhat.com>

task/nfs: stop nfs server after test finishes more thoroughly

In addition to "exportfs -au", we take a sledgehammer to it
by shutting down all the worker threads and flushing out
the client export table.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
Signed-off-by: Greg Farnum <Greg@inktank.com>

Better error handling in list_locks()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix linter errors

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #343 from ceph/wip-dmick

Fixes for #9811 and #9812

task.ssh_keys(): manage authorized_keys, not keys2

See the issue for a description, but authorized_keys2 is dead
on RHEL7 and dying elsewhere.

Fixes: #9812
Signed-off-by: Dan Mick <dan.mick@inktank.com>

lock.lock_many(): use passed os_{type,version}, not ctx

Affects teuthology --lock with os_{type,version} in the yaml.

Fixes: #9811
Signed-off-by: Dan Mick <dan.mick@inktank.com>

Instead of setting status='pass', set success=True

Keeping this old behavior will mean that any tasks that are not updated
to use set_status() and still simply set success=False will still be
able to properly report their failures.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use new functions provided by job_status

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add new job_status module

Provides methods for getting and setting job status. This is necessary
because we used to just use a boolean called 'success'. In order to stop
reporting infrastructure errors as failures, we need more than just a
boolean.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove setting of unused 's3_branch' value

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Stop obsuring stderr when installing with yum

Over time, the code that actually used the buffer we stored stderr in
was removed, leaving us holding error messages and never displaying
them.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Properly quote args that are strings

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

On SSH connection loss, mark jobs as 'dead'

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove stray exit statement

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't assume Ubuntu; also support MacOS X

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Drop libmysqlclient-dev

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use os.path.expanduser() instead of /home

... for locating downburst executable

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Minor fix for vps-only locks

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #339 from ceph/wip-9292-wusui

Cleaned up documentation.

Cleaned up documentation.

Removed some Inktank references, mentioned the split of tasks (between
teuthology/task and chef-qa-suite/tasks), fixed some wording, removed
chef reference that no longer applies.

Did not update the teuthology-suite section at the end of this file
because further documentation changes are coming.

Fixes: #9292
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Use arch in lock_machines()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Query arch in lock_many()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use uname -m instead of uname -p

Debian hosts were reporting as 'unknown'

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

All BaseExceptions should hit failure codepath

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move VersionNotFoundError to exceptions module

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move ScheduleFailError to exceptions module

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use os_type and os_version in lock_machines()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Query os_type and os_version in lock_many()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove default for --os-type

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add OS.__repr__() and OS.to_dict()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move OS class to its own files

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

tasks/nfs: sync before unmounting or unexporting nfs clients/servers

We have some hope that this will resolve #8576 in the Ceph tracker.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Bump requests version to >= 2.3.0

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Allow mixing of bare-metal machine_types

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #335 from ceph/wip-kernel-downgrade

Fix RPM kernel downgrade.

nfs: more lsof output, on clients too

Signed-off-by: Greg Farnum <greg@inktank.com>

Revert "Move default os_type to misc.get_distro()"

This reverts commit 3675a2e25c5d9a370922b714871492f31ed1783d.

Revert "Add os_type and os_version args to lock_many()"

This reverts commit f39b6958c4dd94f7e682833f4cf3122c4537dcde.

Merge pull request #337 from ceph/wip-9597

lock.py: Log failing os-version if it doesn't validate

lock.py: Log failing os-version if it doesn't validate

Fixes: #9597
Signed-off-by: Dan Mick <dan.mick@inktank.com>

Add 'lsb_release -a' parsing back to remote.OS

It can now parse either that or /etc/os-release, and remote.Remote will
attempt the latter first.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add os_type and os_version args to lock_many()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move default os_type to misc.get_distro()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #336 from ceph/wip-nfs

enable lsof when unmounting nfs clients

enable lsof when unmounting nfs clients

Because the lsof interface is strange, use the verbose flag so we get
some output even if there is no use, and ignore errors because it will
return 1 if there's no filesystem use.

Signed-off-by: Greg Farnum <greg@inktank.com>

Fix RPM kernel downgrade.

Force even if an older version or existing.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Merge pull request #334 from ceph/wip-9519-wusui

Handle list-json downburst failure.

Refuse to schedule jobs with 'multi' machine_type

That's not a real machine_type - it's a beanstalkd tube that is used
when a machine_type like 'plana,mira,burnupi' is used.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Raise ConnectionLostError when the socket dies

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Handle list-json downburst failure.

Use check_output instead of Popen and communicate()
Fixes: #9519
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Improve os-release parsing

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Rewrite teuthology-updatekeys

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Skip hostname test

Due to the workaround put in place recently

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Re-implement Distribution as OS

It now uses /etc/os-release instead of lsb_release, like the cool kids
are doing.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #328 from ceph/wip-8700-wusui

Use docopts to parse daemon-helper

Merge pull request #333 from ceph/wip-9495-wusui

Run chef before operations that need it.

Merge pull request #331 from ceph/wip-transport-none

orchestra: fix ConnectionLostError path in wait()

Add a workaround for Remote.hostname

Since 'hostname --fqdn' isn't working reliably

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Run chef before operations that need it.

Chef, if it is needed, should be run before kernel tasks
and before all other user defined tasks.  It should not be
run by default.

Fixes: 9495
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Fix daemon-helper argument parsing.

Fixes: 8700
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #332 from ceph/wip-yaml-path

teuthology-suite: fix handling of extra YAML files

teuthology-suite: fix handling of extra YAML files

Previously these had to be absolute paths, or relative
paths had to be present in the pwd and in ceph-qa-suite.

With this change you can now pass a relative path
like 'debug/mds_client.yaml' to reference a debug
fragment from ceph-qa-suite.

Signed-off-by: John Spray <john.spray@redhat.com>

orchestra: fix ConnectionLostError path in wait()

Sometimes it would generate AttributeError instead
when it tried to call is_active() on None.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #326 from ceph/wip-9318-wusui

Avoid running chef automatically.

Merge pull request #297 from ceph/wip-8712-wusui

Check os-type and os-version for vps

Make push_inventory() a regular function

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Avoid running chef automatically

teuthology --lock formerly automatically ran chef on startup of
virtual machines. This covered up a bug where ceph-deploy would
not install properly. These changes remove the automatic running
of chef, and fixed some related issues when setting up keys on
virtual machines.

Fixes 9318:
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Use teuthology.task.internal.push_inventory

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add push_inventory() subtask

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add lock.update_inventory()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add Remote.inventory_info

A property that returns a dict of attributes in the format that the
paddles lock server likes.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add Remote.host_key

Turns out we can easily grab the ssh key from paramiko once a connection
is established.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Set Remote.hostname by asking the host itself

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add Remote.arch

Returns the result of 'uname -p'

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add caching lsb_release parser

In far too many places do we remotely exec lsb_release in some form,
then parse the output. In shifting to a more stable interface between
teuthology and its tasks, this seemed important to me.

Remote objects now have a 'distro' property that, when first accessed,
calls 'lsb_release -a' remotely and parses the results.

I've included tests and documentation.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Check os-type and os-version for vps

Fixes: 8712
Signed-off-by: Warren Usui warren.usui@inktank.com

Merge pull request #327 from dachary/wip-exec-documentation

documentation: updates to the exec tasks

documentation: updates to the exec tasks

* Quoting is important
* Behavior on error

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

In Remote.run(), attempt to reconnect if necessary

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move more exceptions

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

valgrind: suppress tcmalloc crap for centos 6.5

Fixes: #9336
Signed-off-by: Sage Weil <sage@redhat.com>

Remove unnecessary filter in do_update_keys()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't explode if get_status fails

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #325 from ceph/wip-valgrind

valgrind: use --time-stamp=yes

valgrind: adjust suppressions for wheezy

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind: use --time-stamp=yes

This doesn't help on the precise version, but maybe it will on
future versions... because if so, it would be really helpful
it matching valgrind errors to daemon logs!

In any case, though, I verified it is at worse harmless.

Signed-off-by: Sage Weil <sage@redhat.com>

Update downburst docs.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Merge pull request #324 from ceph/wip-hadoop-nuke

nuke: kill hadoop processes

nuke: kill hadoop processes

Signed-off-by: John Spray <john.spray@redhat.com>

Drop --allow-hosts None

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use setuptools, not distribute

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #323 from ceph/pypi

Prepare for PyPI

Reviewed-by: Sage Weil <sage@redhat.com>

Add install_requires and tests_require

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add MANIFEST.in

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add LICENSE

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #308 from ceph/wip-6146-wusui

Add ability to update with ceph-deploy

Merge pull request #314 from ceph/wip-8923

Speed up teuthology-kill.

Add more info in comments.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>