ceph.conf.template: add cluster name to paths

This is more like the defaults, and makes non-osd daemons from
multiple clusters work on the same host.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test_misc: fix shadowing of cluster

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster param to wait_until_healthy()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster param to wait_until_osds_up()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster filter to num_instances_of_type

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add optional path param to create_simple_monmap()

This will let us provision multiple clusters in parallel by using
different names for the monmap file.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster param to skeleton_config

We need to consider all monitors to allocate unique ips in get_mons(),
but generate the conf file per-cluster.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster-aware roles filter

Unlike roles_of_type, this actually returns roles. We can't change the
semantics of the existing one due to backwards compatibility
requirements for old ceph-qa-suite branches, so make a new function
and implement the old one in terms of the new more general interface.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add optional cluster filter to is_type

reimplement in terms of split_role while we're at it

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common_fs_utils: use common method to extract id from role

This works with multiple clusters too.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: pass cluster through stop_daemons_of_type()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

orchestra/daemon: update resolve_role_list() for multi-cluster setups

Add a parameter for backwards compatibility, to determine whether the
returned roles should include the cluster.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

orchestra/daemon: update simple DaemonGroup methods to handle clusters

Add new kwarg parameters for cluster to stay backwards-compatible with
older ceph-qa-suite branches, and rename role arguments to type_,
since they were never full roles. Default the cluster to ceph, and
normalize role to $cluster.$type.$id internally.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make get_mon_names() and get_first_mon() cluster-aware

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make all_roles_of_type cluster-aware

Reimplement in terms of roles_of_type to simplify it

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make roles_of_type() cluster-aware

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: convert get_clients() to work with multiple clusters

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: convert num_instances_of_type() to use is_type()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: convert skeleton_config to work with clusters in roles

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add helpers to get components of a role

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make get_mons() cluster-aware

Use the is_type() helper to keep the logic in one place.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: allow a prefix for cluster in is_type()

This still matches roles with no prefix first, so it stays compatible
with existing suites.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #842 from dachary/wip-xenial

add ubuntu 16.04 xenial support

add ubuntu 16.04 xenial support

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph.conf: mon warn on no sortbitwise = false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #840 from ceph/wip-osd-25

set debug osd = 25

Reviewed-by: Samuel Just <sjust@redhat.com>

suite: debug osd = 25

This lets me see an important maybe_handle_cache_detail line.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #837 from ceph/wip-rebase

Rebase 2.0 to 10.1.0

Reviewed-by: Dan Mick <dmik@redhat.com>

Rebase 2.0 to 10.1.0

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #834 from dachary/wip-13980-openstack-nfs

openstack: open UDP ports required by NFS

openstack: open UDP ports required by NFS

http://tracker.ceph.com/issues/13980 Fixes: #13980

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #833 from ceph/wip-parallel

parallel: parallel.__exit__ should always wait for greenlets to finish

Merge pull request #831 from ceph/wip-parallel-config

task/parallel: accept lists of tasks in referenced sections

task/parallel: accept lists of tasks in referenced sections

This syntax is consistent with task configuration everywhere else.

The subtley of different syntax for this task made modifying suites
error-prone. Keep the existing syntax so we don't have to change all
the upgrade suites in each ceph-qa-suite branch.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

parallel: parallel.__exit__ should always wait for greenlets to finish

Per the class docstring, one gevent (or the with body) throwing an
exception should not stop the other events.  Instead, wait for them to
finish and then reraise the first exception we find.

A concrete problem caused by this is that with multiple parallel
installs, one failure can cause packaging operations on the other nodes
to be unceremoniously killed and left in an inconsistent state.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #830 from ceph/wip-apt-clean

task/kernel: run apt-get clean before updating to avoid 15090

task/kernel: run apt-get clean before updating to avoid 15090

Fixes: http://tracker.ceph.com/issues/15090
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #828 from smithfarm/wip-debian-symlink-fix

openstack-debian-7.0-user-data.txt: fix symlink

Reviewed-by: Loic Dachary <ldachary@redhat.com>

openstack-debian-7.0-user-data.txt: fix symlink

The file this symlink was pointing to no longer exists.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #827 from smithfarm/wip-opensuse-image

openstack: add wget command for openSUSE Leap 42.1 VM image

Reviewed-by: Loic Dachary <ldachary@redhat.com>

openstack: add wget command for openSUSE Leap 42.1 VM image

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #825 from ceph/wip-kill-doc

Mention that -kill must be run on the master node

Mention that -kill must be run on the master node

There isn't currently a remote kill-a-process feature.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #824 from ceph/wip-ver-update

rh version update

Install ceph-fuse for 2.0 by default for tests to work

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Package name split, ceph is now ceph-mon, ceph-osd

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add ceph-selinux to pkg list for 1.3.2 builds

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #797 from dachary/wip-openstack-override-clone

openstack: allow override of openstack:clone:

Merge pull request #822 from dachary/wip-14997-openstack-stale-keypairs

openstack: protect against stale keypairs

openstack: protect against stale keypairs

http://tracker.ceph.com/issues/14997 Fixes: #14997

Signed-off-by: Loic Dachary <loic@dachary.org>

Update rh_version supported in downstream

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #821 from ceph/wip-yum-priorities

install: Fix non-idempotent check_obsoletes tweak

Reviewed-by: Dan Mick<dmick@redhat.com>

Merge pull request #813 from smithfarm/wip-fix-siteconfig

docs/siteconfig.rst: adapt comments to reality

docs/siteconfig.rst: consistency

Change "count: 0" to "count: 3" for consistency with the comment.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

install: Fix non-idempotent check_obsoletes tweak

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #820 from ceph/wip-nrpe

selinux: Whitelist nrpe

selinux: Whitelist nagios' nrpe

Signed-off-by: Zack Cerza <zack@redhat.com>

selinux: Fix linter complaints

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: do not warn on straw calc version = 0

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #818 from dachary/wip-14799-valgrind

valgrind: add suppression for tcmalloc in libboost_thread-mt.so.1.53.0

valgrind: add suppression for tcmalloc in libboost_thread-mt.so.1.53.0

http://tracker.ceph.com/issues/14799 Fixes: #14799

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #817 from ceph/wip-valgrind-supp

valgrind: ignore boost thread-local warning

Reviewed-by: Dan Mick <dmick@redhat.com>

valgrind: ignore boost thread-local warning

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #814 from ceph/wip-15014

install task: Make package lists configurable

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #816 from ceph/wip-big-vps

lock: Pass canonicalized hostname to create_if_vm

Reviewed-by: Dan Mick <dmick@redhat.com>

lock: Pass canonicalized hostname to create_if_vm

Signed-off-by: Zack Cerza <zack@redhat.com>

install: Remove old "lib deps" cruft

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Use install_task.get_package_list()

(Properly named task.install.get_package_list())

Signed-off-by: Zack Cerza <zack@redhat.com>

install: Add get_package_list()

Move package lists out of the module and into packages.yaml. Enable
overriding package lists via the job config.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #815 from ceph/wip-14855

misc.get_wwn_id_map: not finding wwns is not an error

misc.get_wwn_id_map: not finding wwns is not an error

Fixes: #14855
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #790 from ceph/wip-fixdenials

Add whitelist option and fix for cephtest denials

Add whitelist option and fix for cephtest denials

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #812 from ceph/wip-gevent

Unpin gevent and avoid patching subprocess

Reviewed-by: Dan Mick <dmick@redhat.com>

Unpin gevent and avoid patching subprocess

Recently, gevent updated to 1.1.0, breaking a few things like misc.sh().
PR #810 pinned gevent to 1.0.2, but I wanted to properly investigate the
issue to not be stuck on an old version. Some git bisects later, I
discovered that gevent began patching the subprocess module by default
in:

https://github.com/gevent/gevent/commit/fc73e9afa34d811ede5d62531025d2e3876b8c01

Unpinning gevent and also telling it not to patch subprocess fixes the
issue.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #811 from ceph/wip-prime

mon osd prime pg temp = true

Reviewed-by: Samuel Just <sjust@redhat.com>

ceph.conf: mon osd prime pg temp = true

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #809 from smithfarm/wip_doc_fake_fs

doc: update documentation of make_fake_fstools()

Merge pull request #808 from smithfarm/wip-doc-teuos

README.rst: mention teuthology-openstack in Provided Utilities

Merge pull request #810 from dachary/wip-14990-gevent

pin gevent 1.0.2

pin gevent 1.0.2

http://tracker.ceph.com/issues/14990 Fixes: #14990

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: update documentation of make_fake_fstools()

When the documentation of make_fake_fstools() was written, the function
evidently only returned two values. Now it returns four values. Additional
return values might be added later.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

README.rst: mention teuthology-openstack in Provided Utilities

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #807 from ceph/wip-repo-set-remote

repo_utils: Call git set remote-url

repo_utils: Call git set remote-url

This will cause git repos on-disk to be updated to point at the correct
remote based on config.ceph_git_base_url

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #804 from dachary/wip-openstack-vault

openstack: create default ~/.vault_pass.txt

Merge pull request #806 from dachary/wip-openstack-sitepackage

openstack: do not use sitepackages for openstack-integration tests

Merge pull request #805 from dachary/wip-openstack-test-selection

openstack: test selection for openstack-integration

openstack: do not use sitepackages for openstack-integration tests

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: test selection for openstack-integration

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: create default ~/.vault_pass.txt

It is now required by teuthology.

http://tracker.ceph.com/issues/14914 Fixes: #14914

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: allow override of openstack:clone:

When running from a wip teuthology branch, it is inconvenient to tweak
~/.teuthology.yaml. Add --teuthology-git-url and --teuthology-branch to
take precendence and override the defaults.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #794 from dachary/wip-openstack-wait

openstack: add --wait to block until the suite finishes

Merge pull request #802 from dachary/wip-openstack-neutron

openstack: openstacklient >= 2.1.0 does not depend on neutronclient

Merge pull request #801 from dachary/wip-openstack-centos-7-2

openstack: package CentOS 7.2 data file

Merge pull request #800 from dachary/wip-openstack-cleanup

openstack: only destroy target/ceph- VMs on restart

Merge pull request #799 from dachary/wip-openstack-timeout

openstack: default to 9h job timeout

Merge pull request #796 from dachary/wip-archive-upload

suite/openstack: add --archive-upload-url

Merge pull request #795 from dachary/wip-openstack-suite-exit-code

suite/openstack: implement the exit code

openstack: openstacklient >= 2.1.0 does not depend on neutronclient

Explicitly require it python-neutronclient.

Signed-off-by: Loic Dachary <loic@dachary.org>