task/install: should pass `node` as `config` to get_upgrade_version()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1086 from smithfarm/wip-20392

Drop swift.py task

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1087 from smithfarm/wip-16521-rebase

Pre-upgrade sanity check

install: s/_get_gitbuilder_project/_get_gitbuilder_project/g

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test_upgrade_common: add mock for get_package_version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

install._upgrade_is_downgrade unit test

Signed-off-by: Nathan Cutler <ncutler@suse.com>

install.upgrade: implement pre-upgrade sanity check

Add a pre-upgrade sanity check to the install.upgrade task. This sanity check:

1. determines the version of Ceph installed on the remote ("installed_version")
2. determines the version of Ceph in the target gitbuilder repo
("upgrade_version")
3. if "installed_version" is greater than "upgrade_version", fail immediately

Since the upgrade itself cannot succeed if upgrade_version < installed_version,
it's better to fail before attempting the upgrade.

Before this patch, we were failing in the verify_package_version() sanity check
which is run after attempting (and failing) to upgrade the packages.

Fixes: http://tracker.ceph.com/issues/16521
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Drop swift.py task

Complete the move of This task into ceph/ceph.git

Fixes: http://tracker.ceph.com/issues/20392
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1084 from tchaikov/wip-20171

task/internal/syslog: blacklist log from ceph-create-keys

Merge pull request #1083 from ceph/wip-ansible-purge

purge cluster during teardown

Merge pull request #1070 from ceph/wip-ansible-mgr

[tasks/ansible] Add support for mgr role

purge cluster during teardown

It achieves 2 things
a) tests out purge-cluster and iron out any issues as we find them
b) can make ceph-ansible tests run on any nodes in lab (smithi, mira etc)

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

run gather facts at the start

since we are using custom playbook inside task, we need
to run gather facts before other playbooks are run.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

task/internal/syslog: blacklist log from ceph-create-keys

the log format of ceph-create-keys is "$level:$name:$message"

Fixes: http://tracker.ceph.com/issues/20171
Signed-off-by: Kefu Chai <kchai@redhat.com>

Add support for mgr role

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

remove any stale /var/run/ceph

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1082 from ceph/wip-stale-improve

remove any stale /var/run/ceph

remove any stale /var/run/ceph

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1081 from ceph/wip-ansible-deb

[tasks/ceph-ansible]: Add debug output for ceph-ansible runs

Add debug output for ceph-ansible runs

As requested by Andrew and Alfredo to debug failed ansible jobs
properly

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1079 from ceph/wip-kernel-usage

suite: update usage for -k

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

suite: update usage for -k

-k defaults to distro since e1f5a3a76971 ("use distro kernel as default
for all tests").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1078 from ceph/wip-distro-kernel-as-default

scripts/suite.py: use distro kernel as default for all tests

use distro kernel as default for all tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1075 from ceph/wip-syslog-fix

task/internal/syslog: make kern.log and misc.log world-writable

task/internal/syslog: make kern.log and misc.log world-writable

kern.log and misc.log weren't getting written to on ubuntu machines.
On centos chcon is saving the day.

Fixes: http://tracker.ceph.com/issues/16835
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1069 from ceph/wip-nose-for-rgw-multi

rgw: add nose dependency for rgw_multisite_tests task

rgw: add nose dependency for rgw_multisite_tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1071 from ceph/wip-19859

Update default Ubuntu version to 16.04

Update default Ubuntu version to 16.04

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1068 from tchaikov/wip-python-in-ceph-mgr

valgrind: suppression for python embedded in ceph-mgr

valgrind: suppression for python embedded in ceph-mgr

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1067 from ceph/wip-change-order

[task/internal]: run the redhat internal task after ansible.cephlab task

Merge pull request #1066 from ceph/wip-full-sequential-finally

task/full_sequential_finally: run commands on unwind/cleanup

change the order when redhat internal task is run

ceph-ansible task is modifying some of the repo's set earlier
and affecting cdn runs, change the order to run it after ansible
task is run.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1065 from ceph/wip-teuth-rgw

ceph_ansible: Add the missing rgw role

task/full_sequential_finally: run commands on unwind/cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

Add the missing rgw role

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1060 from ceph/wip-use-sudo-health

Add option to use sudo to check ceph health when required

Merge pull request #1064 from ceph/wip-fix-pip

Fixes related to pip-tools

Reviewed-by: Dan Mick <dmick@redhat.com>

Run 'python setup.py develop' in bootstrap

Our old wonky requirements.txt used to magically do this for us, but no
longer.

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't use pip-sync

It was removing pkg-resources, which breaks 'python setup.py develop'

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1063 from ceph/wip-pip-tools

Use pip-tools to manage dependencies

Reviewed-by: Dan Mick <dmick@redhat.com>

Use pip-tools to manage dependencies

After yet another instance of pip failing to resolve dependencies, I
think it's time to pin everything. The pip-tools utilities provide a
simple way to pin libraries and update them when necessary.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1062 from tchaikov/wip-19631

ceph.conf: adjust "mon reweight min *" for mgr also

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.conf: adjust "mon reweight min *" for mgr also

Fixes: http://tracker.ceph.com/issues/19631
Signed-off-by: Kefu Chai <kchai@redhat.com>

Add option to use sudo to check ceph health when required

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1061 from ceph/wip-fix-checkstatus

check_status(): No-op if there's no proc object

Reviewed-by: Dan Mick <dmick@redhat.com>

check_status(): No-op if there's no proc object

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1049 from SUSE/misc-add-usecase

misc: add flexibility to delete_ and move_file

Merge pull request #1059 from ceph/wip-daemon-failure

If we're going to fail, fail sooner

Reviewed-by: Dan Mick <dmick@redhat.com>

parallel: Let exceptions interrupt other greenlets

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.wait_until_osds_up(): Check for daemon death

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1057 from ceph/wip-exception-logging

Improve exception logging

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1056 from ceph/wip-daemon-failure

Provide a way for jobs to detect daemon failure

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1054 from ceph/wip-19425-wusui

Add missing run.Raw() to ceph_ansible.py

Merge pull request #1055 from tchaikov/wip-workunit-to-use-suite-branch

suite: use "suite_branch" for workunit if avaiable

Patch gevent.Hub.handle_error

So we don't miss out on any fun exceptions!

Lifted from:
https://github.com/ceph/calamari/commit/7edc4cfc7159175a552222700810dc943a0ca8ce

Signed-off-by: Zack Cerza <zack@redhat.com>

Consolidate sys.excepthook replacements

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Catch CommandFailedError during stop

So that job teardown isn't hindered.

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Add check_status()

So far, this just wraps self.proc.poll(). It's intended to provide a way
for tasks to know if their daemons are even running.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: use "suite_branch" for workunit if avaiable

"git clone" allows us to clone a single branch, but not a sha1, so
prefer "branch" if it's available.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Add missing run.Raw() to ceph_ansible.py

Fixes issue: #19425

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1053 from ceph/wip-ovh-destroy

cloud.openstack: Destroy multiple nodes

cloud.openstack: Destroy multiple nodes

In the very rare case where more than one node exists with the same name
(this can happen in case of an outage), destroy all nodes that match
instead of failing altogether.

Signed-off-by: Zack Cerza <zack@redhat.com>

cloud.openstack: Split out node-finding

... into its own method.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1052 from ceph/wip-mgr

ceph.conf: debug mgr by default

ceph.conf: debug mgr by default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1007 from ceph/wip-rh-internal-task

Add redhat internal tasks that setup repo

misc: add flexibility and move_file

Add preserve_perms argument to move_file to make the preservation of
permissions optional. This makes move_file usable if the target
file does not yet exist.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

run redhat internal tasks based on config item

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Always run tests with latest kernel on RHEL

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add redhat internal task to setup repos

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1047 from ceph/wip-daemon-register

orchestra/daemon: separate register_daemon from add_daemon

orchestra/daemon: separate register_daemon from add_daemon

The former registers without starting; the latter does both (as before).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1050 from ceph/wip-pg-remap

ceph.conf: mon osd allow pg remap = true

ceph.conf: mon osd allow pg remap = true

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1045 from ceph/wip-daemon

Fix a couple bugs re: remote daemon execution

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1048 from dmick/master

setup.py: disallow requests 2.13.0

setup.py: disallow requests 2.13.0

python-cinderclient and/or keystoneauth1 apparently don't want
requests 2.13.0, so make it illegal for now

Signed-off-by: Dan Mick <dan.mick@redhat.com>

orchestra.run: More consistently notice failures

check_status=True in combination with wait=False resulted in
CommandFailedError never being raised if poll() was used instead of
wait(). Fix that.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.run: Notice when short-lived procs exit

If a command exits immediately, there is a race between greenlet
completion (which flushes the ChannelFile buffers) and the call to
exit_status_ready(). Waiting for 0.1s on the greenlets removes the race
condition.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.run: Fix bug in RemoteProcess.finished

Previously, if wait() had not been called, we didn't have the returncode
attribute set. Set it from within the finished property if it isn't
already.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1044 from ceph/wip-vps-docs

docs: Update libvirt conf to match current vps_hosts list

Reviewed-by: Dan Mick <dmick@redhat.com>

docs: Update libvirt conf to match current vps_hosts list

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1043 from ceph/wip-fix-1041

provision.downburst: Fix a bug in PR #1041

provision.downburst: Fix a bug in PR #1041

Looks like an argument was dropped to a string format().

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1042 from dmick/wip-downburst-logging

provision: ctx does not always have 'config' attribute

provision: ctx does not always have 'config' attribute

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1041 from dmick/wip-downburst-logging

Invoke downburst with logging

downburst: always log output and error, check returncode for failure

The more info the better; always log everything about the downburst
execution to the teuthology log.  Check for command failure by
checking for returncode != 0 rather than "presence of stderr", since
logging always happens to stderr.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

provision: invoke downburst with -v and --logfile

Verbose output isn't verbose enough to matter, and can be helpful
tracking down weirdness.  Also, log to private file in case
downburst hangs mid-operation, to avoid having to do any
select() madness in teuthology.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1040 from ceph/wip-downburst-unlock

provision.downburst: Tweak destroy return code

Reviewed-by: Dan Mick <dmick@redhat.com>

provision.downburst: Tweak destroy return code

Specifically, if the instance doesn't even exist, consider the destroy
op to have succeeded.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1035 from ceph/wip-f25

bootstrap: Update required package names for Fedora 25

Merge pull request #1039 from dmick/wip-clock

Include missing pieces from ntpd clock task change

ntpdc is deprecated; use ntpq (in the ntp package everywhere)

See https://www.eecis.udel.edu/~mills/ntp/html/ntpdc.html

Signed-off-by: Dan Mick <dan.mick@redhat.com>

clock: handle different service names

Restarting ntpd involves two different service names (thanks
again, distro maintainers, much appreciated)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1038 from tchaikov/wip-yaml-with-repo

docs/detailed_test_config.rst: fix the sample yaml

docs/detailed_test_config.rst: fix the sample yaml

so it is able to work with latest teuthology

Signed-off-by: Kefu Chai <kchai@redhat.com>