teuthology/lock: ignore none 200 jobs in node_job_is_active()

there is chance that we will have 404 when accessing a job's URI.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1166 from batrick/i23662

placeholder: update whitelist for osd slow op wrn

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

placeholder: update whitelist for osd slow op wrn

Caused by: https://github.com/ceph/ceph/pull/20660 (ea97c120d2173f2fc70d979d57a7edb2a6c5da5e)

Fixes: https://tracker.ceph.com/issues/23662
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1160 from ceph/wip-intro-link-new-wiki

Update getting started instructions to link to the new sepia wiki

Merge pull request #1154 from ceph/wip-distro-head

task.kernel: Only show latest kernel when running rpm -q kernel

Merge pull request #1164 from ceph/wip-skip-tags

ansible: Add ability to skip tags during ansible task

ansible: Add ability to skip tags during ansible task

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1161 from ceph/wip-ansible-group-vars

task.ansible: Allow passing in custom group_vars

Overhaul ansible task tests

This fixes failures that have only manifested in Jenkins; it also causes
CephLab to run more of the Ansible tests.

Signed-off-by: Zack Cerza <zack@redhat.com>

Update tox again

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal.syslog: Whitelist a docker warning

Signed-off-by: Zack Cerza <zack@redhat.com>

pytest.ini: Disable cacheprovider

A change in 3.5.0 seems to be buggy; disable the cache rather than
pinning versions.

I believe the offending commit is:
https://github.com/pytest-dev/pytest/commit/dff0500114971b30a7bb9043acb0d0fb6a9e01c4

Signed-off-by: Zack Cerza <zack@redhat.com>

task.ansible: Allow passing in custom group_vars

Up until now, if you wanted to inject vars to a playbook run, you had to
use --extra-vars, which don't behave the same way that group_vars do.
This commit adds that functionality.

We look for a 'group_vars' dict in the task's config object. If it's
there, we create group_vars files with names taken from the keys, and
content taken from the values.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1157 from ceph/wusui-23208

Implement $ option to randomly choose yamls.

Update getting started instructions to link to the new sepia wiki

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Add unit tests for '$' file.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1158 from ceph/wip-wait-fog

Increase timeouts teuthology will wait for FOG provisioning

fog: Wait 10 minutes for machine to be reachable after deploy

A lot's going on in rc.local after a machine is provisioned with FOG.  5
minutes is a little aggressive when taking into account the time it
takes for:
 - The machine to reboot after the FOG task completes
 - BIOS to load
 - DHCP/PXE/TFTP to timeout (double this if NIC order isn't correct)
 - OS to boot and rc.local to do its magic

Signed-off-by: David Galloway <dgallowa@redhat.com>

fog: Wait 15 minutes for FOG task to complete

When there are running jobs and a large run gets scheduled, FOG
provisioning gets backed up due to its built-in rate limiting.  15
minutes should be a little more lenient and prevent large spikes of dead
jobs when a run is scheduled in an idle queue.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Implement $ option to randomly choose yamls.

This implements tracker #23208

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1156 from ceph/wip-libvirt-sdist

Drop libvirt from setup.py

Drop libvirt from setup.py

Having an unversioned libvirt-python in setup.py started causing
problems with tox, since it uses an sdist archive as a basis for
installing teuthology in its virtualenvs. Removing it is consistent with
best practices. We'll still keep it in requirements.txt.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.kernel: Only show latest kernel when running rpm -q kernel

Fixes: http://tracker.ceph.com/issues/23381
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1152 from ceph/wip-rhsm-selinux

task.selinux: Whitelist rhsmd denials

task.selinux: Whitelist rhsmd denials

These started showing up once we added RHEL to Sepia.

Fixes: https://tracker.ceph.com/issues/23343#note-5
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1148 from badone/wip-fedora-community-mysql-devel-depend

bootstrap: Rename mysql-community-devel for fedora

Merge pull request #1150 from ceph/pytest-3.4

Unbreak testing with py.test 3.4

Update tox

Signed-off-by: Zack Cerza <zack@redhat.com>

Unbreak py.test

https://docs.pytest.org/en/latest/logging.html#incompatible-changes-in-pytest-3-4

Signed-off-by: Zack Cerza <zack@redhat.com>

Update ansible and cryptography

via pip-compile -P ansible -P cryptography

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Rename mysql-community-devel for fedora

The name of the package is community-mysql-devel in currently
supported releases.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #1146 from ceph/wip-allow-more-drift

ceph.conf: mon_clock_drift_allowed .5 -> 1.0

ceph.conf: mon_clock_drift_allowed .5 -> 1.0

All of the errors I see seem to be between .5 and .9s.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1145 from jcsp/wip-mgr-valgrind

install: extend python-related valgrind rules

Reviewed-By: Kefu Chai <kchai@redhat.com>

install: extend python-related valgrind rules

These were set for kind 'possible' which was failing
to suppress Leak_DefinitelyLost errors.

Also add several suppressions related to module
initialization.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1144 from tchaikov/wip-22438

teuthology/task/install/valgrind.supp: add suppression for dlopen()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install/valgrind.supp: add suppression for dlopen()

should cover the case of malloc() in addition to calloc()

Fixes: http://tracker.ceph.com/issues/22438
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1143 from tchaikov/wip-22438

teuthology/task/install/valgrind.supp: add suppression for dlopen()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install/valgrind.supp: add suppression for dlopen()

the analysis in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700899
also applies to ceph-common.

Fixes: http://tracker.ceph.com/issues/22438
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1130 from tchaikov/wip-rerun-no-desc

suite: do not rerun jobs w/o description

Merge pull request #1141 from ceph/wip-wusui-22518

Fix installer.0 bug.

Fix installer.0 bug.
Insure that mon node is used for rbd_pools.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1140 from ceph/wip-foryuri-wusui

Implement installer.0 role.

Merge pull request #1139 from ceph/wip-valgrind

valgrind: added suppression for cython constants

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

valgrind: added suppression for cython constants

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Implement installer.0 role.
This allows one to place the ceph ansible installer node on another machine.
It defaults back to the lowest monitor if there is no installer.0 role.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1138 from ceph/wip-rocks-leak

teuthology/task/install/valgrind.supp: new rocksdb leak

teuthology/task/install/valgrind.supp: new rocksdb leak

<error>
  <unique>0x7</unique>
  <tid>1</tid>
  <kind>Leak_StillReachable</kind>
  <xwhat>
    <text>40 bytes in 1 blocks are still reachable in loss record 8 of 21</text>
    <leakedbytes>40</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x9EAF203</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.12.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>334</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>allocate</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/ext</dir>
      <file>new_allocator.h</file>
      <line>111</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>allocate</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>alloc_traits.h</file>
      <line>436</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::__detail::_Hashtable_alloc&lt;std::allocator&lt;std::__detail::_Hash_node&lt;void const*, false&gt; &gt; &gt;::_M_allocate_buckets(unsigned long) [clone .isra.181]</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable_policy.h</file>
      <line>2107</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_allocate_buckets</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>354</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_rehash_aux</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>2098</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::_Hashtable&lt;rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Identity, std::equal_to&lt;rocksdb::ThreadStatusData*&gt;, std::hash&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ra
nged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_rehash(unsigned long, unsigned long const&amp;)</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>2077</line>
    </frame>
    <frame>
      <ip>0xBDDBCA</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::_Hashtable&lt;rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Identity, std::equal_to&lt;rocksdb::ThreadStatusData*&gt;, std::hash&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ra
nged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node&lt;rocksdb::ThreadStatusData*, false&gt;*)</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>1724</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_insert&lt;rocksdb::ThreadStatusData* const&amp;, std::__detail::_AllocNode&lt;std::allocator&lt;std::__detail::_Hash_node&lt;rocksdb::ThreadStatusData*, false&gt; &gt; &gt; &gt;</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>1828</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>insert</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable_policy.h</file>
      <line>843</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>insert</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>unordered_set.h</file>
      <line>420</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::ThreadStatusUpdater::RegisterThread(rocksdb::ThreadStatus::ThreadType, unsigned long)</fn>
      <dir>/usr/src/debug/ceph-13.0.0-3917-g948b47a/src/rocksdb/monitoring</dir>
      <file>thread_status_updater.cc</file>
      <line>25</line>
    </frame>
    <frame>
      <ip>0xBEEE09</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)</fn>
      <dir>/usr/src/debug/ceph-13.0.0-3917-g948b47a/src/rocksdb/util</dir>
      <file>threadpool_imp.cc</file>
      <line>258</line>
    </frame>
    <frame>
      <ip>0xC1AF0E</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>execute_native_thread_routine</fn>
    </frame>
    <frame>
      <ip>0xA8FDE24</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xD75234C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1136 from ceph/wip-fog

nuke: Power off FOG machines instead of reimaging

nuke: Power off FOG machines instead of reimaging

Now that FOG is in master, we don't have to reimage on nuke. Let's shut
them down - the next job will bring them up again during the reimage
process.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1135 from ceph/wip-fog-conslog

Task: Tolerate a missing ctx.config

Merge pull request #1134 from ceph/wip-fog-timeouts

PhysicalConsole fixes for FOG

PhysicalConsole: Correctly report power_off failure

Signed-off-by: Zack Cerza <zack@redhat.com>

Task: Tolerate a missing ctx.config

When we initialize a ConsoleLog task during FOG reimaging, the ctx
object has no config attribute.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: Correctly report power_on failure

We've been logging an error, then unconditionally reporting success. Fix
it.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: replace retry mechanism

It was buggy and unreadable. Use safe_while.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1133 from tchaikov/wip-hadoop-url

task/hadoop: update hadoop 2.5.2 url

task/hadoop: update hadoop 2.5.2 url

hadoop v2.5.2 is pretty old now, and is removed from the hadoop/common
directory. but it can be found in archive.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1131 from ceph/wip-fog-cancel

fog: Cancel stale deploy tasks

fog: Cancel stale deploy tasks

In case, for whatever reason, any active deploy tasks already exist for
a given host, cancel them before we schedule a new one.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Fix TypeError when canceling tasks

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1126 from ceph/wip-fog

Bare-metal reimaging with FOG

update_inventory: Canonicalize hostname

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Reimage instead of nuking as necessary

Once this branch is in master, we'll probably want to switch this
behavior to no-op nodes that would be reimaged, since the next job will
do that anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: If deploy fails, cancel the deploy task

Signed-off-by: Zack Cerza <zack@redhat.com>

lock.ops.lock_many: Log console while reimaging

Signed-off-by: Zack Cerza <zack@redhat.com>

task/console_log: Allow specifying remotes

For use with reimaging; allow passing remotes directly instead of
relying on the ctx.cluster object, which won't exist at that time.

Signed-off-by: Zack Cerza <zack@redhat.com>

task/console_log: Make logfile names customizable

So that we can create separate console logs during the reimage process

Signed-off-by: Zack Cerza <zack@redhat.com>

Reimage machines in parallel

Signed-off-by: Zack Cerza <zack@redhat.com>

Support reimaging with FOG

https://fogproject.org

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: do not rerun jobs w/o description

Signed-off-by: Kefu Chai <kchai@redhat.com>

PhysicalConsole: Add timeout arg to power_cycle()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1129 from ceph/wip-keyfix

Tolerate failure to scan VM host keys

Tolerate failure to scan VM host keys

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1128 from ceph/wip-keyscan

misc: Reimplement host key scanning

Merge pull request #1123 from tchaikov/wip-wait-util-osds-up

misc.wait_until_osds_up: prolong the timeout from 5 min to 9 min

misc: Reimplement host key scanning

We're seeing very intermittent issues with ssh-keyscan; sometimes given
N hostnames, it only returns N-1 keys. Lack of an error message adds to
the confusion. The solution is to call ssh-keyscan once for each host
instead of batching them together - with a few retries - so that we can
easily ensure we get the right amount. If we do not, raise a
RuntimeError.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit daa28ae9210e1a845840ec80776bd211df2e97e9)

Drop pytest-capturelog

It's apparently been replaced by the built-in pytest-catchlog, which has
a very slightly different API.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 939998627e941c2aa06faefae6287dc08e3a0efe)

Merge pull request #1127 from ceph/wip-default-os-version

Update default OS versions

Update default OS versions

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1124 from tchaikov/wip-more-mon-mgr-mkfs-grace

ceph.conf: prolong "mon mgr mkfs grace" to 2 minutes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

ceph.conf: prolong "mon mgr mkfs grace" to 2 minutes

it might take longer than 1 minutes to:
1. create a monmap
2. create the auth keyrings
3. prepare the osd devices and activate OSDs (fdisk,mkfs.xfs,ceph-osd --mkfs)
4. prepare the monitors (mkfs), and start them
5. start monitor

Signed-off-by: Kefu Chai <kchai@redhat.com>

misc.wait_until_osds_up: prolong the timeout from 5 min to 9 min

if an node hosts 6 OSDs, it would take longer to boot. this addresses
the failure of
/a/kchai-2017-11-03_05:56:44-rados-wip-jewel-backports-reloaded-distro-basic-mira/1806380.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1121 from ceph/wip-cephmetrics

Add a cephmetrics task

ceph_ansible: Support custom cluster names

Signed-off-by: Zack Cerza <zack@redhat.com>

Add cephmetrics task

Deploys, then runs integration tests via tox.

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Optional suffix for inventory filenames

So we can support e.g. YAML inventories.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph_ansible: Don't collect logs on non-Ceph hosts

If a job uses machines for something other than to install Ceph, avoid
attempting to collect Ceph logs - the job will fail.

Signed-off-by: Zack Cerza <zack@redhat.com>

Upgrade ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

Upgrade pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1120 from ceph/wip-ansible-purge-fixc

update ansible package from ansible PPA

remove dependency packages the task added from testnode

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
F

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

update ansible package from ansible PPA

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1118 from ceph/wip-21737-shutdown-assert

ceph.conf: enable debug asserts on shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #934 from ceph/wip-daemon-helper-systemd

Changes for orchestra.daemon to work with systemd

DaemonGroup: Avoid using systemd by default

To use systemd, any tasks which create a DaemonGroup object must pass
use_systemd=True. This could be accomplished by looking for a flag in
the job config. Note that this option is mainly for regression testing;
it may be removed in the future, where the default is to use systemd
when possible.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1119 from ceph/wip-dbrst

downburst: Explicitly type our user-data

downburst: Explicitly type our user-data

Something broke for us in cloud-init 0.7.9. Turns out our user-data was
being interpreted as an octet-stream, as opposed to a cloud-config. Mark
it as a cloud-config, so that cloud-init will do the right thing once
more.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: enable debug asserts on shutdown

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1117 from ceph/wip-umount-varlib

More work to allow mounting /var/lib/ceph to a testnode device