ceph.conf: set new pg max per osd limit

This is replacing the old one, but we need to keep the old one for
upgrades etc.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1091 from ceph/wip-fix-cephlab-overrides

[ansible-doc-fix]: correct the doc to fix the overrides name

Merge pull request #1109 from ceph/wip-openstack-log

task/internal/syslog: blacklist openstack datasource log

task/internal/syslog: blacklist openstack datasource log

Caused OVH node jobs to fail when it shouldn't have

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1107 from ceph/wip-prune-fail

prune: Optionally remove failed jobs entirely

prune: Use summary.yaml to determine age of job

If a job has been partially pruned, its directory's mtime will get
bumped, causing total removal to be delayed by potentially months.
Looking at the mtime of summary.yaml should provide results that are
more intuitive.

Signed-off-by: Zack Cerza <zack@redhat.com>

prune: Optionally remove failed jobs entirely

This works exactly like --pass, except it defaults to being disabled

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1101 from ceph/wip-ansible-fixes

[tasks/ceph-ansible]: use site.yml from the ceph-ansible repo

handle libselinux init issues similar to one seen in
https://bugzilla.redhat.com/show_bug.cgi?id=1364703

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Default ansible version to 2.3.2

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

needs system-site-package since libselinux modules not available
from within virtualenv

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Create rbd pool since it doesn't exist by default for luminous+ builds

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use site.yml from the ceph-ansible repo

we have been using custome site.yml from within the task which is not
accurate when compared to the file provided by ceph-ansible repo, instead
use the sample file as the main playbook for tests.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1106 from ceph/wip-remove-rwg-timeout

Removed default_idle_timeout

Removed default_idle_timeout

Fixes http://tracker.ceph.com/issues/21140

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1093 from rzarzynski/wip-cwd-in-remote-process

RemoteProcess: command can be now executed in specified directory.

RemoteProcess: add an unit test for the cwd parameter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1102 from zenglg/master

Supprted for CentOS

python-pip --> python2-pip

In CentOS, Fedora and Redhat, there is only python2-pip, no python-pip.

Supported for CentOS

Merge pull request #1096 from batrick/usage-fix

misc. doc/usage improvements

Merge pull request #1100 from ceph/wip-vps-disk

downburst: Decrease disk size to 100GB

downburst: Decrease disk size to 100GB

The first VPS on every VPSHOST runs off the VPSHOST root drive.  Because
of this, the root drive is filling up due to the addition of another OSD
drive: https://github.com/ceph/teuthology/pull/1099

This not only results in nagios alerts but could also result in needless
job failures.  OSD drives/partitions are already only 100G on smithi so
this should be fine.

Signed-off-by: David Galloway <dgallowa@redhat.com>

correct the doc to fix the overrides name

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1099 from ceph/wip-vps-add-osd

[teuthology/provision]: Increase number of drives for vps to 4

Increase number of drives to 4 per vm

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph.conf: do not warn on mon_osd_down_out_interval=0

Signed-off-by: Sage Weil <sage@redhat.com>

docs: add missing required options to ex cmds

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology-suite: add required -m to usage

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1094 from ceph/wip-apt-get-force-yes

packaging: pass --force-yes to allow for unsigned deb packages

Merge pull request #1095 from jcsp/wip-clog-debug

teuthology: set cluster log level to debug

teuthology: set cluster log level to debug

Since we're testing, we always want the full thing.  This
used to not really matter because almost everything was
going to the clog at info level, but as of luminous things like
the cluster map version echos are at debug level.

Signed-off-by: John Spray <john.spray@redhat.com>

packaging: pass --force-yes to allow for unsigned deb packages

rbd_fio task has been switched to {install,remove}_package().  This
commit mirrors ceph.git commit d14f2da437a3 ("task/rbd_fio: allow for
unsigned packages"):

"Similar to what the teuthology install.py task does, add --force-yes to
the apt-get install so that unsigned packages are successfully
installed. It is needed when the buildpackages task is used to create
packages on the fly."

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph.conf: debug mgr = 20

We fixed the pgmap update bug!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1085 from ceph/wip-collect-logs

[task/ceph_ansible]: collect logs when the test fails

RemoteProcess: command can be now executed in specified directory.

RemoteProcess, a class built around Paramiko's exec_command(),
currently lacks support for the "cwd" parameter. It specifies
the directory in which the requested command should be executed.
This stays in contrast to LocalRemoteProcess from the vstart
runner as it wraps Python's subprocess, and thus naturally
offers the feature. The patch unifies these two implementations.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph.conf: crush-failure-domain=osd in ec profile

Leave the old ruleset- one too for old ceph versions.  We can remove it
when luminous is the oldest version we want to test (or we move ceph.conf
into ceph.git instead of teuthology.git).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.conf: debug mgr = 30 for a bit

Looking for http://tracker.ceph.com/issues/20208

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.conf: do not warn about too few pgs

Signed-off-by: Sage Weil <sage@redhat.com>

Remove gzip as its done by pull_directory function

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1089 from ceph/wip-bootstrap-deadlock

When bootstrapping teuthology, avoid deadlocks

When bootstrapping teuthology, avoid deadlocks

Use Popen.communicate() instead of Popen.wait()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1080 from ceph/wip-deps

requirements.txt: Add nose and update libvirt-python

Merge pull request #1088 from tchaikov/wip-get-upgrade-version

task/install: should pass `node` as `config` to get_upgrade_version()

task/install: should pass `node` as `config` to get_upgrade_version()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1086 from smithfarm/wip-20392

Drop swift.py task

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1087 from smithfarm/wip-16521-rebase

Pre-upgrade sanity check

install: s/_get_gitbuilder_project/_get_gitbuilder_project/g

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test_upgrade_common: add mock for get_package_version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

install._upgrade_is_downgrade unit test

Signed-off-by: Nathan Cutler <ncutler@suse.com>

install.upgrade: implement pre-upgrade sanity check

Add a pre-upgrade sanity check to the install.upgrade task. This sanity check:

1. determines the version of Ceph installed on the remote ("installed_version")
2. determines the version of Ceph in the target gitbuilder repo
("upgrade_version")
3. if "installed_version" is greater than "upgrade_version", fail immediately

Since the upgrade itself cannot succeed if upgrade_version < installed_version,
it's better to fail before attempting the upgrade.

Before this patch, we were failing in the verify_package_version() sanity check
which is run after attempting (and failing) to upgrade the packages.

Fixes: http://tracker.ceph.com/issues/16521
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Drop swift.py task

Complete the move of This task into ceph/ceph.git

Fixes: http://tracker.ceph.com/issues/20392
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1084 from tchaikov/wip-20171

task/internal/syslog: blacklist log from ceph-create-keys

collect logs when the test fails

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1083 from ceph/wip-ansible-purge

purge cluster during teardown

Merge pull request #1070 from ceph/wip-ansible-mgr

[tasks/ansible] Add support for mgr role

purge cluster during teardown

It achieves 2 things
a) tests out purge-cluster and iron out any issues as we find them
b) can make ceph-ansible tests run on any nodes in lab (smithi, mira etc)

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

run gather facts at the start

since we are using custom playbook inside task, we need
to run gather facts before other playbooks are run.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

task/internal/syslog: blacklist log from ceph-create-keys

the log format of ceph-create-keys is "$level:$name:$message"

Fixes: http://tracker.ceph.com/issues/20171
Signed-off-by: Kefu Chai <kchai@redhat.com>

Add support for mgr role

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

remove any stale /var/run/ceph

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1082 from ceph/wip-stale-improve

remove any stale /var/run/ceph

remove any stale /var/run/ceph

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1081 from ceph/wip-ansible-deb

[tasks/ceph-ansible]: Add debug output for ceph-ansible runs

Add debug output for ceph-ansible runs

As requested by Andrew and Alfredo to debug failed ansible jobs
properly

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Update libvirt-python

Signed-off-by: Zack Cerza <zack@redhat.com>

Add missing nose requirement

We forgot to run pip-compile in PR #1069

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1079 from ceph/wip-kernel-usage

suite: update usage for -k

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

suite: update usage for -k

-k defaults to distro since e1f5a3a76971 ("use distro kernel as default
for all tests").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1078 from ceph/wip-distro-kernel-as-default

scripts/suite.py: use distro kernel as default for all tests

use distro kernel as default for all tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1075 from ceph/wip-syslog-fix

task/internal/syslog: make kern.log and misc.log world-writable

task/internal/syslog: make kern.log and misc.log world-writable

kern.log and misc.log weren't getting written to on ubuntu machines.
On centos chcon is saving the day.

Fixes: http://tracker.ceph.com/issues/16835
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1069 from ceph/wip-nose-for-rgw-multi

rgw: add nose dependency for rgw_multisite_tests task

rgw: add nose dependency for rgw_multisite_tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1071 from ceph/wip-19859

Update default Ubuntu version to 16.04

Update default Ubuntu version to 16.04

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1068 from tchaikov/wip-python-in-ceph-mgr

valgrind: suppression for python embedded in ceph-mgr

valgrind: suppression for python embedded in ceph-mgr

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1067 from ceph/wip-change-order

[task/internal]: run the redhat internal task after ansible.cephlab task

Merge pull request #1066 from ceph/wip-full-sequential-finally

task/full_sequential_finally: run commands on unwind/cleanup

change the order when redhat internal task is run

ceph-ansible task is modifying some of the repo's set earlier
and affecting cdn runs, change the order to run it after ansible
task is run.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1065 from ceph/wip-teuth-rgw

ceph_ansible: Add the missing rgw role

task/full_sequential_finally: run commands on unwind/cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

Add the missing rgw role

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1060 from ceph/wip-use-sudo-health

Add option to use sudo to check ceph health when required

Merge pull request #1064 from ceph/wip-fix-pip

Fixes related to pip-tools

Reviewed-by: Dan Mick <dmick@redhat.com>

Run 'python setup.py develop' in bootstrap

Our old wonky requirements.txt used to magically do this for us, but no
longer.

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't use pip-sync

It was removing pkg-resources, which breaks 'python setup.py develop'

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1063 from ceph/wip-pip-tools

Use pip-tools to manage dependencies

Reviewed-by: Dan Mick <dmick@redhat.com>

Use pip-tools to manage dependencies

After yet another instance of pip failing to resolve dependencies, I
think it's time to pin everything. The pip-tools utilities provide a
simple way to pin libraries and update them when necessary.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1062 from tchaikov/wip-19631

ceph.conf: adjust "mon reweight min *" for mgr also

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.conf: adjust "mon reweight min *" for mgr also

Fixes: http://tracker.ceph.com/issues/19631
Signed-off-by: Kefu Chai <kchai@redhat.com>

Add option to use sudo to check ceph health when required

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1061 from ceph/wip-fix-checkstatus

check_status(): No-op if there's no proc object

Reviewed-by: Dan Mick <dmick@redhat.com>

check_status(): No-op if there's no proc object

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1049 from SUSE/misc-add-usecase

misc: add flexibility to delete_ and move_file

Merge pull request #1059 from ceph/wip-daemon-failure

If we're going to fail, fail sooner

Reviewed-by: Dan Mick <dmick@redhat.com>

parallel: Let exceptions interrupt other greenlets

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.wait_until_osds_up(): Check for daemon death

Signed-off-by: Zack Cerza <zack@redhat.com>