task/hadoop: fix for multi cluster is_type() change

This is the only non-ceph task that uses custom roles, and it happens
to put a '.' as a delimiter between hadoop and the rest of the
role. Since the multi-cluster branch, is_type() is ceph-specific and
interprets this as a cluster named hadoop.

To avoid needing to change several branches in ceph-qa-suite, just use
a local is_hadoop_type() function instead of relying on the now
ceph-specific misc.is_type().

Fixes: http://tracker.ceph.com/issues/15862
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #841 from ceph/wip-multi-cluster

support multiple ceph clusters

Merge pull request #855 from dmick/wip-dependencies

bootstrap: Use mariadb-devel on RHEL for mysql-devel; remove mysqlclient dependency

mysqlclient is not needed; MySQLdb-python is for teuthology-coverage

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Use the appropriate package name for mysql/mariadb

Community package download isn't necessary; RHEL ships
mariadb and its -devel package

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #854 from ceph/wip-15711

Always mention name and sha1 when scheduling fails

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Always mention name and sha1 when scheduling fails

http://tracker.ceph.com/issues/15711
Fixes: 15711
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #832 from ceph/wip-fix-nuke

 fix nuke

Merge pull request #851 from ceph/wip-prune-mon

prune: also remove mon data in jobs' 'data' subdir

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #853 from ceph/wip-pcp2

task/selinux: another pcp whitelist

Reviewed-by: Boris Ranto <branto@redhat.com>

task/selinux: another pcp whitelist

SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1462234212.274:85266): avc: denied { read } for pid=1984 comm="pmcd" name="pmlogger_daily.pid" dev="tmpfs" ino=1474542 scontext=system_u:system_r:pcp_pmcd_t:s0 tcontext=system_u:object_r:cron_var_run_t:s0 tclass=file']

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #852 from ceph/wip-pcp-selinux

task.selinux: add PCP's pmlogger to the whitelist

task.selinux: add PCP's pmlogger to the whitelist

Signed-off-by: Zack Cerza <zack@redhat.com>

Prune mon data along with remote logs

Signed-off-by: Zack Cerza <zack@redhat.com>

Move subdir removal logic into separate function

Signed-off-by: Zack Cerza <zack@redhat.com>

Add timeout to commands

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix nuke, redo some cleanup functions

Redo some functions, remove proc waits, Reorder functions
for better cleanup.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Remove ceph and its dependent packages

Remove ceph and its dependent packages using force option

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use timeout feature of paramiko

use timeout feature of paramiko when executing commands

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #836 from ceph/wip-rh-bootstrap

Changes for bootstrap to work on RHEL

Merge pull request #850 from ceph/wip-paramiko

Unpin paramiko

Reviewed-by: Dan Mick <dmick@redhat.com>

Unpin paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #848 from ceph/wip-filter-wusui

Restrict filter to not scan too far up a path.

Restrict filter to not scan too far up a path.
Directory names closer to the root than /suites are not used by the filter.
Fixes: http://tracker.ceph.com/issues/15470
Signed-off-by: Warren Usui <wusui@redhat.com>

Changes for bootstrap to work on RHEL

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #846 from ceph/wip-babel

Work around pip dependency issue w/ nova and Babel

Reviewed-by: Dan Mick <dmick@redhat.com>

Work around pip dependency issue w/ nova and Babel

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #845 from ceph/wip-vault-pass

ansible.CephLab: touch ~/.vault_pass.txt

Reviewed-by: Dan Mick <dmick@redhat.com>

ansible.CephLab: touch ~/.vault_pass.txt

To avoid ansible bailing with an OSError. Normally we don't even need
the vault password for cephlab.yml.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #843 from ceph/wip-gethash

teuthology-suite: Don't allow scheduling when packages are not present

Reviewed-by: Dan Mick <dmick@redhat.com>

Simplify message counting filtered jobs

Signed-off-by: Zack Cerza <zack@redhat.com>

Don't attempt to schedule when missing packages

By default, anyway. It's possible to set
config.suite_allow_missing_packages to continue stick with the old
behavior.

Signed-off-by: Zack Cerza <zack@redhat.com>

get_hash() -> get_gitbuilder_hash()

Signed-off-by: Zack Cerza <zack@redhat.com>

Always use git_ls_remote() for ceph packages

Signed-off-by: Zack Cerza <zack@redhat.com>

git_ls_remote(): Add debug output and docstring

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #826 from dachary/wip-15236-rekey

orchestra: disable re-keying to fix Key-exchange timed out

Merge pull request #835 from dachary/wip-lock-arch

lock: add the --arch option

Merge pull request #839 from ErwanAliasr1/evelu-bootstrap

Bootstrap

ceph.conf.template: add cluster name to paths

This is more like the defaults, and makes non-osd daemons from
multiple clusters work on the same host.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test_misc: fix shadowing of cluster

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster param to wait_until_healthy()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster param to wait_until_osds_up()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster filter to num_instances_of_type

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add optional path param to create_simple_monmap()

This will let us provision multiple clusters in parallel by using
different names for the monmap file.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster param to skeleton_config

We need to consider all monitors to allocate unique ips in get_mons(),
but generate the conf file per-cluster.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add cluster-aware roles filter

Unlike roles_of_type, this actually returns roles. We can't change the
semantics of the existing one due to backwards compatibility
requirements for old ceph-qa-suite branches, so make a new function
and implement the old one in terms of the new more general interface.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add optional cluster filter to is_type

reimplement in terms of split_role while we're at it

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

common_fs_utils: use common method to extract id from role

This works with multiple clusters too.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: pass cluster through stop_daemons_of_type()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

orchestra/daemon: update resolve_role_list() for multi-cluster setups

Add a parameter for backwards compatibility, to determine whether the
returned roles should include the cluster.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

orchestra/daemon: update simple DaemonGroup methods to handle clusters

Add new kwarg parameters for cluster to stay backwards-compatible with
older ceph-qa-suite branches, and rename role arguments to type_,
since they were never full roles. Default the cluster to ceph, and
normalize role to $cluster.$type.$id internally.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make get_mon_names() and get_first_mon() cluster-aware

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make all_roles_of_type cluster-aware

Reimplement in terms of roles_of_type to simplify it

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make roles_of_type() cluster-aware

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: convert get_clients() to work with multiple clusters

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: convert num_instances_of_type() to use is_type()

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: convert skeleton_config to work with clusters in roles

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: add helpers to get components of a role

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: make get_mons() cluster-aware

Use the is_type() helper to keep the logic in one place.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

misc: allow a prefix for cluster in is_type()

This still matches roles with no prefix first, so it stays compatible
with existing suites.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #842 from dachary/wip-xenial

add ubuntu 16.04 xenial support

add ubuntu 16.04 xenial support

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph.conf: mon warn on no sortbitwise = false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #840 from ceph/wip-osd-25

set debug osd = 25

Reviewed-by: Samuel Just <sjust@redhat.com>

suite: debug osd = 25

This lets me see an important maybe_handle_cache_detail line.

Signed-off-by: Sage Weil <sage@redhat.com>

bootstrap: Enforce LC_ALL for the script

This script is expecting rpm to report with english sentences.
To avoid any issues with other locales, let's enforce the LC_ALL to C.

Signed-off-by: Erwan Velu <erwan@redhat.com>

bootstrap: DNF is the default package manager on Fedora > 22

Let's use dnf if the fedora release is 23 or greater unless let's fall
back to yum

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #837 from ceph/wip-rebase

Rebase 2.0 to 10.1.0

Reviewed-by: Dan Mick <dmik@redhat.com>

Rebase 2.0 to 10.1.0

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

lock: add the --arch option

So that it is possible to lock a target of a given arch. The lock_many
function already has the argument, it only needs to be added to the list
of available options.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #834 from dachary/wip-13980-openstack-nfs

openstack: open UDP ports required by NFS

openstack: open UDP ports required by NFS

http://tracker.ceph.com/issues/13980 Fixes: #13980

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #833 from ceph/wip-parallel

parallel: parallel.__exit__ should always wait for greenlets to finish

Merge pull request #831 from ceph/wip-parallel-config

task/parallel: accept lists of tasks in referenced sections

task/parallel: accept lists of tasks in referenced sections

This syntax is consistent with task configuration everywhere else.

The subtley of different syntax for this task made modifying suites
error-prone. Keep the existing syntax so we don't have to change all
the upgrade suites in each ceph-qa-suite branch.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

parallel: parallel.__exit__ should always wait for greenlets to finish

Per the class docstring, one gevent (or the with body) throwing an
exception should not stop the other events.  Instead, wait for them to
finish and then reraise the first exception we find.

A concrete problem caused by this is that with multiple parallel
installs, one failure can cause packaging operations on the other nodes
to be unceremoniously killed and left in an inconsistent state.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #830 from ceph/wip-apt-clean

task/kernel: run apt-get clean before updating to avoid 15090

task/kernel: run apt-get clean before updating to avoid 15090

Fixes: http://tracker.ceph.com/issues/15090
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #828 from smithfarm/wip-debian-symlink-fix

openstack-debian-7.0-user-data.txt: fix symlink

Reviewed-by: Loic Dachary <ldachary@redhat.com>

openstack-debian-7.0-user-data.txt: fix symlink

The file this symlink was pointing to no longer exists.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #827 from smithfarm/wip-opensuse-image

openstack: add wget command for openSUSE Leap 42.1 VM image

Reviewed-by: Loic Dachary <ldachary@redhat.com>

openstack: add wget command for openSUSE Leap 42.1 VM image

Signed-off-by: Nathan Cutler <ncutler@suse.com>

orchestra: disable re-keying to fix Key-exchange timed out

Disable re-keying entirely: it does not work at all and alway times
out. Here is the reproducer:

ceph-workbench ceph-qa-suite --verbose --suite upgrade/infernalis-x --suite-branch jewel --ceph jewel --ceph-git-url https://github.com/ceph/ceph --filter 'upgrade:infernalis-x/stress-split/{0-cluster/start.yaml 1-infernalis-install/infernalis.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 6-next-mon/monb.yaml 7-workload/{radosbench.yaml rbd_api.yaml} 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/centos_7.2.yaml}'

Which always fails after ~4h with the following:

2016-03-17T23:44:32.607 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/ceph_manager.py", line 639, in wrapper
    return func(self)
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/ceph_manager.py", line 693, in do_thrash
    self.choose_action()()
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/ceph_manager.py", line 445, in fix_pgp_num
    self.ceph_manager.set_pool_pgpnum(pool)
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/ceph_manager.py", line 1294, in set_pool_pgpnum
    if self.get_num_creating() > 0:
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/ceph_manager.py", line 1456, in get_num_creating
    pgs = self.get_pg_stats()
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/ceph_manager.py", line 1325, in get_pg_stats
    out = self.raw_cluster_cmd('pg', 'dump', '--format=json')
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/ceph_manager.py", line 810, in raw_cluster_cmd
    stdout=StringIO(),
  File "/home/ubuntu/teuthology/teuthology/orchestra/remote.py", line 196, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 94, in wait
    greenlet.get()
  File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 274, in get
    raise self._exception
SSHException: Key-exchange timed out waiting for key negotiation

http://tracker.ceph.com/issues/15236 Fixes: #15236

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #825 from ceph/wip-kill-doc

Mention that -kill must be run on the master node

Mention that -kill must be run on the master node

There isn't currently a remote kill-a-process feature.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #824 from ceph/wip-ver-update

rh version update

Install ceph-fuse for 2.0 by default for tests to work

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Package name split, ceph is now ceph-mon, ceph-osd

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add ceph-selinux to pkg list for 1.3.2 builds

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #797 from dachary/wip-openstack-override-clone

openstack: allow override of openstack:clone:

Merge pull request #822 from dachary/wip-14997-openstack-stale-keypairs

openstack: protect against stale keypairs

openstack: protect against stale keypairs

http://tracker.ceph.com/issues/14997 Fixes: #14997

Signed-off-by: Loic Dachary <loic@dachary.org>

Update rh_version supported in downstream

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #821 from ceph/wip-yum-priorities

install: Fix non-idempotent check_obsoletes tweak

Reviewed-by: Dan Mick<dmick@redhat.com>

Merge pull request #813 from smithfarm/wip-fix-siteconfig

docs/siteconfig.rst: adapt comments to reality

docs/siteconfig.rst: consistency

Change "count: 0" to "count: 3" for consistency with the comment.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

install: Fix non-idempotent check_obsoletes tweak

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #820 from ceph/wip-nrpe

selinux: Whitelist nrpe

selinux: Whitelist nagios' nrpe

Signed-off-by: Zack Cerza <zack@redhat.com>

selinux: Fix linter complaints

Signed-off-by: Zack Cerza <zack@redhat.com>