DaemonGroup: Avoid using systemd by default

To use systemd, any tasks which create a DaemonGroup object must pass
use_systemd=True. This could be accomplished by looking for a flag in
the job config. Note that this option is mainly for regression testing;
it may be removed in the future, where the default is to use systemd
when possible.

Signed-off-by: Zack Cerza <zack@redhat.com>

SystemDState: Use correct output_cmd for rgw

We had this right for systemctl commands, but not for the journalctl
command.

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup.add_daemon(): Properly start daemons

For some reason we weren't calling restart() inside add_daemons() for
systemd.

Signed-off-by: Zack Cerza <zack@redhat.com>

SystemDState: Tweak logging

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup: Don't use systemd with valgrind

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup: Detect and use systemd

Don't rely on a configuration item to tell us whether or not to use
systemd.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remote: add init_system property

So far, this only exists to answer the question: "does this host use
systemd?" - in which case it will return 'systemd'. Else it will return
None.

Signed-off-by: Zack Cerza <zack@redhat.com>

SystemDState: Treat PIDs as strings when killing

Signed-off-by: Zack Cerza <zack@redhat.com>

Split DaemonState in two

Put systemd stuff in SystemDState

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/daemon: Convert to a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Raise on startup failure

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Use consistent daemon IDs

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState.check_status(): support systemd

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Drop use of self.id

... we already had self.id_

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup: Simplify setting self.use_init

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/daemon: Tweak logging

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Factor out PID discovery

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Don't needlessly reset self.id

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/daemon: Fix linter errors

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: factor out command generation

Signed-off-by: Zack Cerza <zack@redhat.com>

Changes for daemon-helper to work with systemd

when cluster is setup using ceph-ansible or ceph-deploy
use systemd commands to kill/revive deamons.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1111 from ceph/wip-pg-limit

ceph.conf: set new pg max per osd limit

ceph.conf: set new pg max per osd limit

This is replacing the old one, but we need to keep the old one for
upgrades etc.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1091 from ceph/wip-fix-cephlab-overrides

[ansible-doc-fix]: correct the doc to fix the overrides name

Merge pull request #1109 from ceph/wip-openstack-log

task/internal/syslog: blacklist openstack datasource log

task/internal/syslog: blacklist openstack datasource log

Caused OVH node jobs to fail when it shouldn't have

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1107 from ceph/wip-prune-fail

prune: Optionally remove failed jobs entirely

prune: Use summary.yaml to determine age of job

If a job has been partially pruned, its directory's mtime will get
bumped, causing total removal to be delayed by potentially months.
Looking at the mtime of summary.yaml should provide results that are
more intuitive.

Signed-off-by: Zack Cerza <zack@redhat.com>

prune: Optionally remove failed jobs entirely

This works exactly like --pass, except it defaults to being disabled

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1101 from ceph/wip-ansible-fixes

[tasks/ceph-ansible]: use site.yml from the ceph-ansible repo

handle libselinux init issues similar to one seen in
https://bugzilla.redhat.com/show_bug.cgi?id=1364703

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Default ansible version to 2.3.2

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

needs system-site-package since libselinux modules not available
from within virtualenv

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Create rbd pool since it doesn't exist by default for luminous+ builds

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use site.yml from the ceph-ansible repo

we have been using custome site.yml from within the task which is not
accurate when compared to the file provided by ceph-ansible repo, instead
use the sample file as the main playbook for tests.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1106 from ceph/wip-remove-rwg-timeout

Removed default_idle_timeout

Removed default_idle_timeout

Fixes http://tracker.ceph.com/issues/21140

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1093 from rzarzynski/wip-cwd-in-remote-process

RemoteProcess: command can be now executed in specified directory.

RemoteProcess: add an unit test for the cwd parameter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1102 from zenglg/master

Supprted for CentOS

python-pip --> python2-pip

In CentOS, Fedora and Redhat, there is only python2-pip, no python-pip.

Supported for CentOS

Merge pull request #1096 from batrick/usage-fix

misc. doc/usage improvements

Merge pull request #1100 from ceph/wip-vps-disk

downburst: Decrease disk size to 100GB

downburst: Decrease disk size to 100GB

The first VPS on every VPSHOST runs off the VPSHOST root drive.  Because
of this, the root drive is filling up due to the addition of another OSD
drive: https://github.com/ceph/teuthology/pull/1099

This not only results in nagios alerts but could also result in needless
job failures.  OSD drives/partitions are already only 100G on smithi so
this should be fine.

Signed-off-by: David Galloway <dgallowa@redhat.com>

correct the doc to fix the overrides name

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1099 from ceph/wip-vps-add-osd

[teuthology/provision]: Increase number of drives for vps to 4

Increase number of drives to 4 per vm

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph.conf: do not warn on mon_osd_down_out_interval=0

Signed-off-by: Sage Weil <sage@redhat.com>

docs: add missing required options to ex cmds

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology-suite: add required -m to usage

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1094 from ceph/wip-apt-get-force-yes

packaging: pass --force-yes to allow for unsigned deb packages

Merge pull request #1095 from jcsp/wip-clog-debug

teuthology: set cluster log level to debug

teuthology: set cluster log level to debug

Since we're testing, we always want the full thing.  This
used to not really matter because almost everything was
going to the clog at info level, but as of luminous things like
the cluster map version echos are at debug level.

Signed-off-by: John Spray <john.spray@redhat.com>

packaging: pass --force-yes to allow for unsigned deb packages

rbd_fio task has been switched to {install,remove}_package().  This
commit mirrors ceph.git commit d14f2da437a3 ("task/rbd_fio: allow for
unsigned packages"):

"Similar to what the teuthology install.py task does, add --force-yes to
the apt-get install so that unsigned packages are successfully
installed. It is needed when the buildpackages task is used to create
packages on the fly."

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph.conf: debug mgr = 20

We fixed the pgmap update bug!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1085 from ceph/wip-collect-logs

[task/ceph_ansible]: collect logs when the test fails

RemoteProcess: command can be now executed in specified directory.

RemoteProcess, a class built around Paramiko's exec_command(),
currently lacks support for the "cwd" parameter. It specifies
the directory in which the requested command should be executed.
This stays in contrast to LocalRemoteProcess from the vstart
runner as it wraps Python's subprocess, and thus naturally
offers the feature. The patch unifies these two implementations.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph.conf: crush-failure-domain=osd in ec profile

Leave the old ruleset- one too for old ceph versions.  We can remove it
when luminous is the oldest version we want to test (or we move ceph.conf
into ceph.git instead of teuthology.git).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.conf: debug mgr = 30 for a bit

Looking for http://tracker.ceph.com/issues/20208

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.conf: do not warn about too few pgs

Signed-off-by: Sage Weil <sage@redhat.com>

Remove gzip as its done by pull_directory function

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1089 from ceph/wip-bootstrap-deadlock

When bootstrapping teuthology, avoid deadlocks

When bootstrapping teuthology, avoid deadlocks

Use Popen.communicate() instead of Popen.wait()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1080 from ceph/wip-deps

requirements.txt: Add nose and update libvirt-python

Merge pull request #1088 from tchaikov/wip-get-upgrade-version

task/install: should pass `node` as `config` to get_upgrade_version()

task/install: should pass `node` as `config` to get_upgrade_version()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1086 from smithfarm/wip-20392

Drop swift.py task

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1087 from smithfarm/wip-16521-rebase

Pre-upgrade sanity check

install: s/_get_gitbuilder_project/_get_gitbuilder_project/g

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test_upgrade_common: add mock for get_package_version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

install._upgrade_is_downgrade unit test

Signed-off-by: Nathan Cutler <ncutler@suse.com>

install.upgrade: implement pre-upgrade sanity check

Add a pre-upgrade sanity check to the install.upgrade task. This sanity check:

1. determines the version of Ceph installed on the remote ("installed_version")
2. determines the version of Ceph in the target gitbuilder repo
("upgrade_version")
3. if "installed_version" is greater than "upgrade_version", fail immediately

Since the upgrade itself cannot succeed if upgrade_version < installed_version,
it's better to fail before attempting the upgrade.

Before this patch, we were failing in the verify_package_version() sanity check
which is run after attempting (and failing) to upgrade the packages.

Fixes: http://tracker.ceph.com/issues/16521
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Drop swift.py task

Complete the move of This task into ceph/ceph.git

Fixes: http://tracker.ceph.com/issues/20392
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1084 from tchaikov/wip-20171

task/internal/syslog: blacklist log from ceph-create-keys

collect logs when the test fails

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1083 from ceph/wip-ansible-purge

purge cluster during teardown

Merge pull request #1070 from ceph/wip-ansible-mgr

[tasks/ansible] Add support for mgr role

purge cluster during teardown

It achieves 2 things
a) tests out purge-cluster and iron out any issues as we find them
b) can make ceph-ansible tests run on any nodes in lab (smithi, mira etc)

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

run gather facts at the start

since we are using custom playbook inside task, we need
to run gather facts before other playbooks are run.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

task/internal/syslog: blacklist log from ceph-create-keys

the log format of ceph-create-keys is "$level:$name:$message"

Fixes: http://tracker.ceph.com/issues/20171
Signed-off-by: Kefu Chai <kchai@redhat.com>

Add support for mgr role

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

remove any stale /var/run/ceph

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1082 from ceph/wip-stale-improve

remove any stale /var/run/ceph

remove any stale /var/run/ceph

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1081 from ceph/wip-ansible-deb

[tasks/ceph-ansible]: Add debug output for ceph-ansible runs

Add debug output for ceph-ansible runs

As requested by Andrew and Alfredo to debug failed ansible jobs
properly

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Update libvirt-python

Signed-off-by: Zack Cerza <zack@redhat.com>

Add missing nose requirement

We forgot to run pip-compile in PR #1069

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1079 from ceph/wip-kernel-usage

suite: update usage for -k

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

suite: update usage for -k

-k defaults to distro since e1f5a3a76971 ("use distro kernel as default
for all tests").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1078 from ceph/wip-distro-kernel-as-default

scripts/suite.py: use distro kernel as default for all tests

use distro kernel as default for all tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1075 from ceph/wip-syslog-fix

task/internal/syslog: make kern.log and misc.log world-writable

task/internal/syslog: make kern.log and misc.log world-writable

kern.log and misc.log weren't getting written to on ubuntu machines.
On centos chcon is saving the day.

Fixes: http://tracker.ceph.com/issues/16835
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1069 from ceph/wip-nose-for-rgw-multi

rgw: add nose dependency for rgw_multisite_tests task

rgw: add nose dependency for rgw_multisite_tests task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1071 from ceph/wip-19859

Update default Ubuntu version to 16.04

Update default Ubuntu version to 16.04

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1068 from tchaikov/wip-python-in-ceph-mgr

valgrind: suppression for python embedded in ceph-mgr