Merge pull request #1059 from ceph/wip-daemon-failure

If we're going to fail, fail sooner

Reviewed-by: Dan Mick <dmick@redhat.com>

parallel: Let exceptions interrupt other greenlets

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.wait_until_osds_up(): Check for daemon death

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1057 from ceph/wip-exception-logging

Improve exception logging

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1056 from ceph/wip-daemon-failure

Provide a way for jobs to detect daemon failure

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1054 from ceph/wip-19425-wusui

Add missing run.Raw() to ceph_ansible.py

Merge pull request #1055 from tchaikov/wip-workunit-to-use-suite-branch

suite: use "suite_branch" for workunit if avaiable

Patch gevent.Hub.handle_error

So we don't miss out on any fun exceptions!

Lifted from:
https://github.com/ceph/calamari/commit/7edc4cfc7159175a552222700810dc943a0ca8ce

Signed-off-by: Zack Cerza <zack@redhat.com>

Consolidate sys.excepthook replacements

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Catch CommandFailedError during stop

So that job teardown isn't hindered.

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Add check_status()

So far, this just wraps self.proc.poll(). It's intended to provide a way
for tasks to know if their daemons are even running.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: use "suite_branch" for workunit if avaiable

"git clone" allows us to clone a single branch, but not a sha1, so
prefer "branch" if it's available.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Add missing run.Raw() to ceph_ansible.py

Fixes issue: #19425

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1053 from ceph/wip-ovh-destroy

cloud.openstack: Destroy multiple nodes

cloud.openstack: Destroy multiple nodes

In the very rare case where more than one node exists with the same name
(this can happen in case of an outage), destroy all nodes that match
instead of failing altogether.

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud.openstack: Split out node-finding

... into its own method.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1052 from ceph/wip-mgr

ceph.conf: debug mgr by default

ceph.conf: debug mgr by default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1007 from ceph/wip-rh-internal-task

Add redhat internal tasks that setup repo

run redhat internal tasks based on config item

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Always run tests with latest kernel on RHEL

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add redhat internal task to setup repos

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1047 from ceph/wip-daemon-register

orchestra/daemon: separate register_daemon from add_daemon

orchestra/daemon: separate register_daemon from add_daemon

The former registers without starting; the latter does both (as before).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1050 from ceph/wip-pg-remap

ceph.conf: mon osd allow pg remap = true

ceph.conf: mon osd allow pg remap = true

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1045 from ceph/wip-daemon

Fix a couple bugs re: remote daemon execution

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1048 from dmick/master

setup.py: disallow requests 2.13.0

setup.py: disallow requests 2.13.0

python-cinderclient and/or keystoneauth1 apparently don't want
requests 2.13.0, so make it illegal for now

Signed-off-by: Dan Mick <dan.mick@redhat.com>

orchestra.run: More consistently notice failures

check_status=True in combination with wait=False resulted in
CommandFailedError never being raised if poll() was used instead of
wait(). Fix that.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.run: Notice when short-lived procs exit

If a command exits immediately, there is a race between greenlet
completion (which flushes the ChannelFile buffers) and the call to
exit_status_ready(). Waiting for 0.1s on the greenlets removes the race
condition.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.run: Fix bug in RemoteProcess.finished

Previously, if wait() had not been called, we didn't have the returncode
attribute set. Set it from within the finished property if it isn't
already.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1044 from ceph/wip-vps-docs

docs: Update libvirt conf to match current vps_hosts list

Reviewed-by: Dan Mick <dmick@redhat.com>

docs: Update libvirt conf to match current vps_hosts list

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1043 from ceph/wip-fix-1041

provision.downburst: Fix a bug in PR #1041

provision.downburst: Fix a bug in PR #1041

Looks like an argument was dropped to a string format().

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1042 from dmick/wip-downburst-logging

provision: ctx does not always have 'config' attribute

provision: ctx does not always have 'config' attribute

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1041 from dmick/wip-downburst-logging

Invoke downburst with logging

downburst: always log output and error, check returncode for failure

The more info the better; always log everything about the downburst
execution to the teuthology log.  Check for command failure by
checking for returncode != 0 rather than "presence of stderr", since
logging always happens to stderr.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

provision: invoke downburst with -v and --logfile

Verbose output isn't verbose enough to matter, and can be helpful
tracking down weirdness.  Also, log to private file in case
downburst hangs mid-operation, to avoid having to do any
select() madness in teuthology.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1040 from ceph/wip-downburst-unlock

provision.downburst: Tweak destroy return code

Reviewed-by: Dan Mick <dmick@redhat.com>

provision.downburst: Tweak destroy return code

Specifically, if the instance doesn't even exist, consider the destroy
op to have succeeded.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1035 from ceph/wip-f25

bootstrap: Update required package names for Fedora 25

Merge pull request #1039 from dmick/wip-clock

Include missing pieces from ntpd clock task change

ntpdc is deprecated; use ntpq (in the ntp package everywhere)

See https://www.eecis.udel.edu/~mills/ntp/html/ntpdc.html

Signed-off-by: Dan Mick <dan.mick@redhat.com>

clock: handle different service names

Restarting ntpd involves two different service names (thanks
again, distro maintainers, much appreciated)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1038 from tchaikov/wip-yaml-with-repo

docs/detailed_test_config.rst: fix the sample yaml

docs/detailed_test_config.rst: fix the sample yaml

so it is able to work with latest teuthology

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1037 from ceph/wip-cloud-volume-retry

cloud.openstack: Also retry on BaseHTTPError

cloud.openstack: Also retry on BaseHTTPError

We attach volumes immediately after creating them; sometimes they are
still momentarily in the 'creating' state, causing the attach call to
throw a BaseHTTPError. When that happens, simply retry the request
instead of failing node creation, starting the entire cycle all over
again.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Update required package names for Fedora 25

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1012 from ceph/wip-libcloud

libcloud provisioning backend with OpenStack support

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1034 from ceph/wip-nuke-nfsganesha

nuke: Remove nfs-ganesha repos

nuke: Remove nfs-ganesha repos

Fixes: http://tracker.ceph.com/issues/18974
Signed-off-by: David Galloway <dgallowa@redhat.com>

cloud.openstack: Exclude windows-specific sizes

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Do not log the namespace object

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud.openstack: Cache authentication tokens

Constantly causing Keystone to regenerate auth tokens was the cause of
our hitting rate limits during testing. This will let us reuse auth
tokens - including across processes - to avoid hitting those limits.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add cloud.util.AuthToken

This provides a mechanism for caching OpenStack authentication tokens

Signed-off-by: Zack Cerza <zack@redhat.com>

Move FileLock out of repo_utils

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud: Volume creation fails node creation

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud: Retry failed requests in libcloud

It's common to see "429 Rate limit exceeded", at least with OVH. When we
encounter the exception associated with that exception, backoff and
retry for an interval before eventually giving up.

Signed-off-by: Zack Cerza <zack@redhat.com>

updatekeys: Fix argument parsing

Signed-off-by: Zack Cerza <zack@redhat.com>

.gitignore: .idea

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't unlock VMs when destroy fails

Signed-off-by: Zack Cerza <zack@redhat.com>

Update SSH keys when creating VMs

Signed-off-by: Zack Cerza <zack@redhat.com>

Handle VMs missing consoles safely

Signed-off-by: Zack Cerza <zack@redhat.com>

Make teuthology.lock a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

More intelligently implement is_vm()

Move to using one decent implementation instead of multiple (some naive)
implementations

Signed-off-by: Zack Cerza <zack@redhat.com>

Locking changes needed for libcloud

Signed-off-by: Zack Cerza <zack@redhat.com>

Add libcloud backend

Initially this supports OpenStack but will grow to support other methods
of cloud-like deployment. Some assuptions are made regarding supporting
infrastructure (FIXME document these)

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1033 from ceph/wip-wait-osd-timeout

misc.wait_until_osds_up(): timeout after 5min

misc.wait_until_osds_up(): timeout after 5min

It doesn't make any sense to wait more than a few minutes for OSDs to
come up. If they take more than five minutes, fail the job.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1030 from tchaikov/wip-suite-sha1-for-workunit

suite: use "suite_hash" as the default sha1 for workunit

Merge pull request #1032 from ceph/wip-nuke-samba

nuke: Remove samba repos

nuke: Remove samba repos

Fixes: http://tracker.ceph.com/issues/19061
Signed-off-by: David Galloway <dgallowa@redhat.com>

run: allow using alternate suite repo

Do the same thing we do for ceph repo to make ceph.git commit
1f82b9b9446d ("qa/tasks/workunit: use the suite repo for cloning
workunit") work for scheduled jobs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

suite: use "suite_hash" as the default sha1 for workunit

as "workunits" reside in ceph/qa/workunits, it's more intuitive to
respect suite-branch option when cloning workunits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1027 from ceph/wip-ceph-ansible-boot

ceph_ansible: update pip and ansible versions

Merge pull request #1013 from ceph/wip-clock

Change to use task 'clock' instead of 'clock.check'

update default ansible version to 2.2.1

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

upgrade pip first before setuptools

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1018 from jan--f/wip-bootstrap-tumbleweed

bootstrap: add support for opensuse tumbleweed.

Merge pull request #1021 from ceph/wip-suite-default

teuthology-suite: Drop default for --ceph

Merge pull request #1004 from SUSE/wip-17981

nuke: Use pkill -KILL to unconditionally wipe out hadoop processes

Merge pull request #1026 from dmick/master

prune: use the shortest time of -p or -r to decide on processing

prune: use the shortest time of -p or -r to decide on processing

invoking -p 7 -r 30 was only removing passed jobs 30 days or older.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1025 from ceph/wip-console-force-into-spy

console: force existing connections into spy mode if !readonly

console: force existing connections into spy mode if !readonly

If someone watching the console didn't think of using "console -s", we
end up power cycling the node in an attempt to get the login prompt.
This is futile -- if the watcher is still there after the node comes
back up, our connection will get dropped to spy mode again.

Use -f to temporarily force existing connections into spy mode when we
attach to save a power cycle.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1024 from ceph/wip-kclient-nuke-nosync

nuke: work around a reboot -n trouble

nuke: work around a reboot -n trouble

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

nuke: improve stale_kernel_mount() check

Commit 7db9e8b76fd5 ("nuke: bring stale kernel client handling back")
resurrected the check that was removed in commit 1d47a121b385 ("Fix
nuke, redo some cleanup functions").  It isn't sufficient though -- for
example, if a workunit already issued a umount, /etc/mtab won't have
a '^/dev/rbd' entry.

debugfs is enabled and mounted on all distros we care about.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1022 from ceph/wip-unbreak-kclient-nuke

nuke: bring stale kernel client handling back

nuke: drop remove_kernel_mounts()

Calling remove_kernel_mounts() after reboot() is pretty useless.  Also,
as explained in the previous commit, there isn't much we can do in the
krbd case, so just drop it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

nuke: bring stale kernel client handling back

Commit 1d47a121b385 ("Fix nuke, redo some cleanup functions") broke
stale kernel client map/mount handling by dropping reboot arguments.
While for kcephfs we can use 'umount -f' to avoid sync (it used to not
work, but is mostly fixed now, I believe), currently there is nothing
we can do for a local filesystem mounted on top of krbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

teuthology-suite: Drop default for --ceph

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1020 from ceph/wip-waitpid

Tell gevent not to patch os.waitpid()

Reviewed-by: Dan Mick <dmick@redhat.com>

Tell gevent not to patch os.waitpid()

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: add support for opensuse tumbleweed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #1017 from ceph/wip-manhole

Use manhole to provide a way to debug hung jobs

Reviewed-by: Dan Mick <dmick@redhat.com>