ceph.conf.template: drop cephfs configs

This will be set in the cephfs qa suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1187 from SUSE/wip-restart-workers

Restart dead workers

Merge pull request #1188 from SUSE/wip-job-owner

Fix worker can't figure out owner of a job

Fix worker can't figure out owner of a job

The patch fixes uncaught exception RuntimeError:

  I could not figure out the owner of the requested job.
  Please pass --owner <owner>.

Worker dies with unhandled exception in run_with_watchdog
if it can't figure out owner of a job, which it tries
to kill when job runs longer then given limit of time.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Restart dead workers

This patch allows to restart dead workers separately
not stopping the rest of the teuthology components,
and what is more important the beanstalkd service.
That makes it possible to extend the number of workers too.
Also, either of pulpito and paddles can be restarted alone.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1185 from batrick/hidden-file

build_matrix: ignore hidden files

test: add hidden file test

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

build_matrix: ignore hidden files

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1182 from tchaikov/wip-rerun

suite: allow `--rerun` to run full set of failed tests

teuthology-suite: add --seed option for repeatable random test

currently --rerun does not match tests of
'supported-random-distro$/ubuntu_latest.yaml' with
'supported-random-distro$/centos_latest.yaml'. the former could be part
of description of a failed test, the latter is a a part of job
description generated by build_matrix(). because the '$' operator
instructs theuthology to choose a random file under the directory ending
with '$', and we expand the '$' to a randomly picked file *before*
filtering the generated job list with the filter collected from the
failed tests, there is good chance that the job descriptions of the
failed jobs in self.args.filter_in cannot match with the randomly
generated ones.

so, we introduce an argument '--seed' for teuthology-suite for the
repeatable random test. this argument allows user to specify a seed for
tne RNG used by build_matrix().

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

suite: avoid preparing the full list for filtering

speed up the filtering a little bit

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1180 from SUSE/wip-fix-fetch_binaries_for_coredumps

Fix fetch_binaries_for_coredumps

Fix fetch_binaries_for_coredumps

Addresses error message:

  AttributeError: 'tuple' object has no attribute 'split'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1122 from ceph/wip-restart-daemon

[orchestra/systemd]: check status when daemon is restarted

Merge pull request #1176 from cbodley/wip-ansible-requirements

ceph-ansible: add hard-coded notario dependency

ceph-ansible: add hard-coded notario dependency

Fixes: http://tracker.ceph.com/issues/24230
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1175 from ceph/wip-24168

Unpin libvirt-python

Unpin libvirt-python

http://tracker.ceph.com/issues/24168

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1174 from ceph/wip-suite-tests

suite/test/test_run_.py: Don't hit the network!

suite/test/test_run_.py: Don't hit the network!

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1173 from ceph/wip-vault-pass

ansible.py: Write "foo" to ~/.vault_pass.txt instead of touching

ansible.py: Write "foo" to ~/.vault_pass.txt instead of touching

ansible-playbook will not run with an empty vault password file.  It
will run if the password file has something in it even if it doesn't get
used.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1165 from ceph/wip-wusui-23208

Allow both $ and directory$ for random yamls.

Allow both $ and directory$ for random yamls.

If either the diretory contains a magic $ file, or if the directory name
ends with $, then the random selection of a yaml file will occur.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1172 from batrick/slow-ops-whitelist

remove blanket SLOW_OPS whitelist

remove blanket SLOW_OPS whitelist

Should no longer be necessary after [1].

[1] https://github.com/ceph/ceph/pull/21684

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1170 from ceph/wip-bionic

Add Bionic to distro map

Add Bionic to distro map

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1167 from tchaikov/wip-job-404

teuthology/lock: ignore none 200 jobs in node_job_is_active()

Merge pull request #1169 from ceph/wip-23798

task.selinux: Whitelist syslogd_t denials

task.selinux: Whitelist syslogd_t denials

Fixes: http://tracker.ceph.com/issues/23798
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1168 from tchaikov/wip-mds-all-down

placeholder: whitelist MDS_ALL_DOWN by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

placeholder: whitelist MDS_ALL_DOWN, MDS_UP_LESS_THAN_MAX  by default

because, in ceph/qa/tasks/ceph.py, we start mon, mgr, osd, and then mds.
there is a time window where there is no mds around, but mgr is checking
mdsmap for MDS_ALL_DOWN errors. there is no way to disable this check in
this time window. so we just whitelist MDS_ALL_DOWN here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/lock: ignore none 200 jobs in node_job_is_active()

there is chance that we will have 404 when accessing a job's URI.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1166 from batrick/i23662

placeholder: update whitelist for osd slow op wrn

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

placeholder: update whitelist for osd slow op wrn

Caused by: https://github.com/ceph/ceph/pull/20660 (ea97c120d2173f2fc70d979d57a7edb2a6c5da5e)

Fixes: https://tracker.ceph.com/issues/23662
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1160 from ceph/wip-intro-link-new-wiki

Update getting started instructions to link to the new sepia wiki

Merge pull request #1154 from ceph/wip-distro-head

task.kernel: Only show latest kernel when running rpm -q kernel

Merge pull request #1164 from ceph/wip-skip-tags

ansible: Add ability to skip tags during ansible task

ansible: Add ability to skip tags during ansible task

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1161 from ceph/wip-ansible-group-vars

task.ansible: Allow passing in custom group_vars

Overhaul ansible task tests

This fixes failures that have only manifested in Jenkins; it also causes
CephLab to run more of the Ansible tests.

Signed-off-by: Zack Cerza <zack@redhat.com>

Update tox again

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal.syslog: Whitelist a docker warning

Signed-off-by: Zack Cerza <zack@redhat.com>

pytest.ini: Disable cacheprovider

A change in 3.5.0 seems to be buggy; disable the cache rather than
pinning versions.

I believe the offending commit is:
https://github.com/pytest-dev/pytest/commit/dff0500114971b30a7bb9043acb0d0fb6a9e01c4

Signed-off-by: Zack Cerza <zack@redhat.com>

task.ansible: Allow passing in custom group_vars

Up until now, if you wanted to inject vars to a playbook run, you had to
use --extra-vars, which don't behave the same way that group_vars do.
This commit adds that functionality.

We look for a 'group_vars' dict in the task's config object. If it's
there, we create group_vars files with names taken from the keys, and
content taken from the values.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1157 from ceph/wusui-23208

Implement $ option to randomly choose yamls.

Update getting started instructions to link to the new sepia wiki

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Add unit tests for '$' file.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1158 from ceph/wip-wait-fog

Increase timeouts teuthology will wait for FOG provisioning

fog: Wait 10 minutes for machine to be reachable after deploy

A lot's going on in rc.local after a machine is provisioned with FOG.  5
minutes is a little aggressive when taking into account the time it
takes for:
 - The machine to reboot after the FOG task completes
 - BIOS to load
 - DHCP/PXE/TFTP to timeout (double this if NIC order isn't correct)
 - OS to boot and rc.local to do its magic

Signed-off-by: David Galloway <dgallowa@redhat.com>

fog: Wait 15 minutes for FOG task to complete

When there are running jobs and a large run gets scheduled, FOG
provisioning gets backed up due to its built-in rate limiting.  15
minutes should be a little more lenient and prevent large spikes of dead
jobs when a run is scheduled in an idle queue.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Implement $ option to randomly choose yamls.

This implements tracker #23208

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1156 from ceph/wip-libvirt-sdist

Drop libvirt from setup.py

Drop libvirt from setup.py

Having an unversioned libvirt-python in setup.py started causing
problems with tox, since it uses an sdist archive as a basis for
installing teuthology in its virtualenvs. Removing it is consistent with
best practices. We'll still keep it in requirements.txt.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.kernel: Only show latest kernel when running rpm -q kernel

Fixes: http://tracker.ceph.com/issues/23381
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1152 from ceph/wip-rhsm-selinux

task.selinux: Whitelist rhsmd denials

task.selinux: Whitelist rhsmd denials

These started showing up once we added RHEL to Sepia.

Fixes: https://tracker.ceph.com/issues/23343#note-5
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1148 from badone/wip-fedora-community-mysql-devel-depend

bootstrap: Rename mysql-community-devel for fedora

Merge pull request #1150 from ceph/pytest-3.4

Unbreak testing with py.test 3.4

Update tox

Signed-off-by: Zack Cerza <zack@redhat.com>

Unbreak py.test

https://docs.pytest.org/en/latest/logging.html#incompatible-changes-in-pytest-3-4

Signed-off-by: Zack Cerza <zack@redhat.com>

Update ansible and cryptography

via pip-compile -P ansible -P cryptography

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Rename mysql-community-devel for fedora

The name of the package is community-mysql-devel in currently
supported releases.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #1146 from ceph/wip-allow-more-drift

ceph.conf: mon_clock_drift_allowed .5 -> 1.0

ceph.conf: mon_clock_drift_allowed .5 -> 1.0

All of the errors I see seem to be between .5 and .9s.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1145 from jcsp/wip-mgr-valgrind

install: extend python-related valgrind rules

Reviewed-By: Kefu Chai <kchai@redhat.com>

install: extend python-related valgrind rules

These were set for kind 'possible' which was failing
to suppress Leak_DefinitelyLost errors.

Also add several suppressions related to module
initialization.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1144 from tchaikov/wip-22438

teuthology/task/install/valgrind.supp: add suppression for dlopen()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install/valgrind.supp: add suppression for dlopen()

should cover the case of malloc() in addition to calloc()

Fixes: http://tracker.ceph.com/issues/22438
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1143 from tchaikov/wip-22438

teuthology/task/install/valgrind.supp: add suppression for dlopen()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install/valgrind.supp: add suppression for dlopen()

the analysis in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700899
also applies to ceph-common.

Fixes: http://tracker.ceph.com/issues/22438
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1130 from tchaikov/wip-rerun-no-desc

suite: do not rerun jobs w/o description

Merge pull request #1141 from ceph/wip-wusui-22518

Fix installer.0 bug.

Fix installer.0 bug.
Insure that mon node is used for rbd_pools.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1140 from ceph/wip-foryuri-wusui

Implement installer.0 role.

Merge pull request #1139 from ceph/wip-valgrind

valgrind: added suppression for cython constants

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

valgrind: added suppression for cython constants

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Implement installer.0 role.
This allows one to place the ceph ansible installer node on another machine.
It defaults back to the lowest monitor if there is no installer.0 role.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1138 from ceph/wip-rocks-leak

teuthology/task/install/valgrind.supp: new rocksdb leak

teuthology/task/install/valgrind.supp: new rocksdb leak

<error>
  <unique>0x7</unique>
  <tid>1</tid>
  <kind>Leak_StillReachable</kind>
  <xwhat>
    <text>40 bytes in 1 blocks are still reachable in loss record 8 of 21</text>
    <leakedbytes>40</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x9EAF203</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.12.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>334</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>allocate</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/ext</dir>
      <file>new_allocator.h</file>
      <line>111</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>allocate</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>alloc_traits.h</file>
      <line>436</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::__detail::_Hashtable_alloc&lt;std::allocator&lt;std::__detail::_Hash_node&lt;void const*, false&gt; &gt; &gt;::_M_allocate_buckets(unsigned long) [clone .isra.181]</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable_policy.h</file>
      <line>2107</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_allocate_buckets</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>354</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_rehash_aux</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>2098</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::_Hashtable&lt;rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Identity, std::equal_to&lt;rocksdb::ThreadStatusData*&gt;, std::hash&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ra
nged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_rehash(unsigned long, unsigned long const&amp;)</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>2077</line>
    </frame>
    <frame>
      <ip>0xBDDBCA</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::_Hashtable&lt;rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Identity, std::equal_to&lt;rocksdb::ThreadStatusData*&gt;, std::hash&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ra
nged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node&lt;rocksdb::ThreadStatusData*, false&gt;*)</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>1724</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_insert&lt;rocksdb::ThreadStatusData* const&amp;, std::__detail::_AllocNode&lt;std::allocator&lt;std::__detail::_Hash_node&lt;rocksdb::ThreadStatusData*, false&gt; &gt; &gt; &gt;</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>1828</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>insert</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable_policy.h</file>
      <line>843</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>insert</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>unordered_set.h</file>
      <line>420</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::ThreadStatusUpdater::RegisterThread(rocksdb::ThreadStatus::ThreadType, unsigned long)</fn>
      <dir>/usr/src/debug/ceph-13.0.0-3917-g948b47a/src/rocksdb/monitoring</dir>
      <file>thread_status_updater.cc</file>
      <line>25</line>
    </frame>
    <frame>
      <ip>0xBEEE09</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)</fn>
      <dir>/usr/src/debug/ceph-13.0.0-3917-g948b47a/src/rocksdb/util</dir>
      <file>threadpool_imp.cc</file>
      <line>258</line>
    </frame>
    <frame>
      <ip>0xC1AF0E</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>execute_native_thread_routine</fn>
    </frame>
    <frame>
      <ip>0xA8FDE24</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xD75234C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1136 from ceph/wip-fog

nuke: Power off FOG machines instead of reimaging

nuke: Power off FOG machines instead of reimaging

Now that FOG is in master, we don't have to reimage on nuke. Let's shut
them down - the next job will bring them up again during the reimage
process.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1135 from ceph/wip-fog-conslog

Task: Tolerate a missing ctx.config

Merge pull request #1134 from ceph/wip-fog-timeouts

PhysicalConsole fixes for FOG

PhysicalConsole: Correctly report power_off failure

Signed-off-by: Zack Cerza <zack@redhat.com>

Task: Tolerate a missing ctx.config

When we initialize a ConsoleLog task during FOG reimaging, the ctx
object has no config attribute.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: Correctly report power_on failure

We've been logging an error, then unconditionally reporting success. Fix
it.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: replace retry mechanism

It was buggy and unreadable. Use safe_while.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1133 from tchaikov/wip-hadoop-url

task/hadoop: update hadoop 2.5.2 url

task/hadoop: update hadoop 2.5.2 url

hadoop v2.5.2 is pretty old now, and is removed from the hadoop/common
directory. but it can be found in archive.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1131 from ceph/wip-fog-cancel

fog: Cancel stale deploy tasks

fog: Cancel stale deploy tasks

In case, for whatever reason, any active deploy tasks already exist for
a given host, cancel them before we schedule a new one.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Fix TypeError when canceling tasks

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1126 from ceph/wip-fog

Bare-metal reimaging with FOG

update_inventory: Canonicalize hostname

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Reimage instead of nuking as necessary

Once this branch is in master, we'll probably want to switch this
behavior to no-op nodes that would be reimaged, since the next job will
do that anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: If deploy fails, cancel the deploy task

Signed-off-by: Zack Cerza <zack@redhat.com>

lock.ops.lock_many: Log console while reimaging

Signed-off-by: Zack Cerza <zack@redhat.com>

task/console_log: Allow specifying remotes

For use with reimaging; allow passing remotes directly instead of
relying on the ctx.cluster object, which won't exist at that time.

Signed-off-by: Zack Cerza <zack@redhat.com>