Merge pull request #1130 from tchaikov/wip-rerun-no-desc

suite: do not rerun jobs w/o description

Merge pull request #1141 from ceph/wip-wusui-22518

Fix installer.0 bug.

Fix installer.0 bug.
Insure that mon node is used for rbd_pools.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1140 from ceph/wip-foryuri-wusui

Implement installer.0 role.

Merge pull request #1139 from ceph/wip-valgrind

valgrind: added suppression for cython constants

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

valgrind: added suppression for cython constants

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Implement installer.0 role.
This allows one to place the ceph ansible installer node on another machine.
It defaults back to the lowest monitor if there is no installer.0 role.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1138 from ceph/wip-rocks-leak

teuthology/task/install/valgrind.supp: new rocksdb leak

teuthology/task/install/valgrind.supp: new rocksdb leak

<error>
  <unique>0x7</unique>
  <tid>1</tid>
  <kind>Leak_StillReachable</kind>
  <xwhat>
    <text>40 bytes in 1 blocks are still reachable in loss record 8 of 21</text>
    <leakedbytes>40</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x9EAF203</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.12.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>334</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>allocate</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/ext</dir>
      <file>new_allocator.h</file>
      <line>111</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>allocate</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>alloc_traits.h</file>
      <line>436</line>
    </frame>
    <frame>
      <ip>0xBDB237</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::__detail::_Hashtable_alloc&lt;std::allocator&lt;std::__detail::_Hash_node&lt;void const*, false&gt; &gt; &gt;::_M_allocate_buckets(unsigned long) [clone .isra.181]</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable_policy.h</file>
      <line>2107</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_allocate_buckets</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>354</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_rehash_aux</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>2098</line>
    </frame>
    <frame>
      <ip>0xBDDA87</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::_Hashtable&lt;rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Identity, std::equal_to&lt;rocksdb::ThreadStatusData*&gt;, std::hash&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ra
nged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_rehash(unsigned long, unsigned long const&amp;)</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>2077</line>
    </frame>
    <frame>
      <ip>0xBDDBCA</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>std::_Hashtable&lt;rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Identity, std::equal_to&lt;rocksdb::ThreadStatusData*&gt;, std::hash&lt;rocksdb::ThreadStatusData*&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ra
nged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;false, true, true&gt; &gt;::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node&lt;rocksdb::ThreadStatusData*, false&gt;*)</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>1724</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>_M_insert&lt;rocksdb::ThreadStatusData* const&amp;, std::__detail::_AllocNode&lt;std::allocator&lt;std::__detail::_Hash_node&lt;rocksdb::ThreadStatusData*, false&gt; &gt; &gt; &gt;</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable.h</file>
      <line>1828</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>insert</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>hashtable_policy.h</file>
      <line>843</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>insert</fn>
      <dir>/opt/rh/devtoolset-7/root/usr/include/c++/7/bits</dir>
      <file>unordered_set.h</file>
      <line>420</line>
    </frame>
    <frame>
      <ip>0xBDC6BE</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::ThreadStatusUpdater::RegisterThread(rocksdb::ThreadStatus::ThreadType, unsigned long)</fn>
      <dir>/usr/src/debug/ceph-13.0.0-3917-g948b47a/src/rocksdb/monitoring</dir>
      <file>thread_status_updater.cc</file>
      <line>25</line>
    </frame>
    <frame>
      <ip>0xBEEE09</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)</fn>
      <dir>/usr/src/debug/ceph-13.0.0-3917-g948b47a/src/rocksdb/util</dir>
      <file>threadpool_imp.cc</file>
      <line>258</line>
    </frame>
    <frame>
      <ip>0xC1AF0E</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>execute_native_thread_routine</fn>
    </frame>
    <frame>
      <ip>0xA8FDE24</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xD75234C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1136 from ceph/wip-fog

nuke: Power off FOG machines instead of reimaging

nuke: Power off FOG machines instead of reimaging

Now that FOG is in master, we don't have to reimage on nuke. Let's shut
them down - the next job will bring them up again during the reimage
process.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1135 from ceph/wip-fog-conslog

Task: Tolerate a missing ctx.config

Merge pull request #1134 from ceph/wip-fog-timeouts

PhysicalConsole fixes for FOG

PhysicalConsole: Correctly report power_off failure

Signed-off-by: Zack Cerza <zack@redhat.com>

Task: Tolerate a missing ctx.config

When we initialize a ConsoleLog task during FOG reimaging, the ctx
object has no config attribute.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: Correctly report power_on failure

We've been logging an error, then unconditionally reporting success. Fix
it.

Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole: replace retry mechanism

It was buggy and unreadable. Use safe_while.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1133 from tchaikov/wip-hadoop-url

task/hadoop: update hadoop 2.5.2 url

task/hadoop: update hadoop 2.5.2 url

hadoop v2.5.2 is pretty old now, and is removed from the hadoop/common
directory. but it can be found in archive.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1131 from ceph/wip-fog-cancel

fog: Cancel stale deploy tasks

fog: Cancel stale deploy tasks

In case, for whatever reason, any active deploy tasks already exist for
a given host, cancel them before we schedule a new one.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Fix TypeError when canceling tasks

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1126 from ceph/wip-fog

Bare-metal reimaging with FOG

update_inventory: Canonicalize hostname

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: Reimage instead of nuking as necessary

Once this branch is in master, we'll probably want to switch this
behavior to no-op nodes that would be reimaged, since the next job will
do that anyway.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: If deploy fails, cancel the deploy task

Signed-off-by: Zack Cerza <zack@redhat.com>

lock.ops.lock_many: Log console while reimaging

Signed-off-by: Zack Cerza <zack@redhat.com>

task/console_log: Allow specifying remotes

For use with reimaging; allow passing remotes directly instead of
relying on the ctx.cluster object, which won't exist at that time.

Signed-off-by: Zack Cerza <zack@redhat.com>

task/console_log: Make logfile names customizable

So that we can create separate console logs during the reimage process

Signed-off-by: Zack Cerza <zack@redhat.com>

Reimage machines in parallel

Signed-off-by: Zack Cerza <zack@redhat.com>

Support reimaging with FOG

https://fogproject.org

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: do not rerun jobs w/o description

Signed-off-by: Kefu Chai <kchai@redhat.com>

PhysicalConsole: Add timeout arg to power_cycle()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1129 from ceph/wip-keyfix

Tolerate failure to scan VM host keys

Tolerate failure to scan VM host keys

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1128 from ceph/wip-keyscan

misc: Reimplement host key scanning

Merge pull request #1123 from tchaikov/wip-wait-util-osds-up

misc.wait_until_osds_up: prolong the timeout from 5 min to 9 min

misc: Reimplement host key scanning

We're seeing very intermittent issues with ssh-keyscan; sometimes given
N hostnames, it only returns N-1 keys. Lack of an error message adds to
the confusion. The solution is to call ssh-keyscan once for each host
instead of batching them together - with a few retries - so that we can
easily ensure we get the right amount. If we do not, raise a
RuntimeError.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit daa28ae9210e1a845840ec80776bd211df2e97e9)

Drop pytest-capturelog

It's apparently been replaced by the built-in pytest-catchlog, which has
a very slightly different API.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 939998627e941c2aa06faefae6287dc08e3a0efe)

Merge pull request #1127 from ceph/wip-default-os-version

Update default OS versions

Update default OS versions

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1124 from tchaikov/wip-more-mon-mgr-mkfs-grace

ceph.conf: prolong "mon mgr mkfs grace" to 2 minutes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

ceph.conf: prolong "mon mgr mkfs grace" to 2 minutes

it might take longer than 1 minutes to:
1. create a monmap
2. create the auth keyrings
3. prepare the osd devices and activate OSDs (fdisk,mkfs.xfs,ceph-osd --mkfs)
4. prepare the monitors (mkfs), and start them
5. start monitor

Signed-off-by: Kefu Chai <kchai@redhat.com>

misc.wait_until_osds_up: prolong the timeout from 5 min to 9 min

if an node hosts 6 OSDs, it would take longer to boot. this addresses
the failure of
/a/kchai-2017-11-03_05:56:44-rados-wip-jewel-backports-reloaded-distro-basic-mira/1806380.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1121 from ceph/wip-cephmetrics

Add a cephmetrics task

ceph_ansible: Support custom cluster names

Signed-off-by: Zack Cerza <zack@redhat.com>

Add cephmetrics task

Deploys, then runs integration tests via tox.

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible: Optional suffix for inventory filenames

So we can support e.g. YAML inventories.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph_ansible: Don't collect logs on non-Ceph hosts

If a job uses machines for something other than to install Ceph, avoid
attempting to collect Ceph logs - the job will fail.

Signed-off-by: Zack Cerza <zack@redhat.com>

Upgrade ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

Upgrade pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1120 from ceph/wip-ansible-purge-fixc

update ansible package from ansible PPA

remove dependency packages the task added from testnode

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
F

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

update ansible package from ansible PPA

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1118 from ceph/wip-21737-shutdown-assert

ceph.conf: enable debug asserts on shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #934 from ceph/wip-daemon-helper-systemd

Changes for orchestra.daemon to work with systemd

DaemonGroup: Avoid using systemd by default

To use systemd, any tasks which create a DaemonGroup object must pass
use_systemd=True. This could be accomplished by looking for a flag in
the job config. Note that this option is mainly for regression testing;
it may be removed in the future, where the default is to use systemd
when possible.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1119 from ceph/wip-dbrst

downburst: Explicitly type our user-data

downburst: Explicitly type our user-data

Something broke for us in cloud-init 0.7.9. Turns out our user-data was
being interpreted as an octet-stream, as opposed to a cloud-config. Mark
it as a cloud-config, so that cloud-init will do the right thing once
more.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.conf: enable debug asserts on shutdown

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1117 from ceph/wip-umount-varlib

More work to allow mounting /var/lib/ceph to a testnode device

internal: Be okay with an empty /var/lib/ceph dir

Signed-off-by: David Galloway <dgallowa@redhat.com>

install: Always try to unmount /var/lib/ceph

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1116 from ceph/wip-var-lib-ceph

install: Unmount /var/lib/ceph itself if needed

install: Unmount /var/lib/ceph itself if needed

We plan to mount a small logical volume to /var/lib/ceph.

Fixes: http://tracker.ceph.com/issues/20910
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1113 from ceph/wip-debug-shutdown

ceph.conf: osd_debug_shutdown=true

Merge pull request #1110 from ceph/wip-ceph-ansible-rgw-interface

[task/ceph-ansible]: Add required radosgw_interface due to ceph-ansible upstream changes

Merge pull request #1112 from ceph/wip-set-ansible-lib

task/ceph-ansible: copy purge-cluster to top-level directory for it to work

copy purge-cluster to top level dir for the playbook to work

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph.conf: osd_debug_shutdown=true

The extra logging during osd shutdown is becoming optional; enable it in
qa.

Signed-off-by: Sage Weil <sage@redhat.com>

SystemDState: Use correct output_cmd for rgw

We had this right for systemctl commands, but not for the journalctl
command.

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup.add_daemon(): Properly start daemons

For some reason we weren't calling restart() inside add_daemons() for
systemd.

Signed-off-by: Zack Cerza <zack@redhat.com>

SystemDState: Tweak logging

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup: Don't use systemd with valgrind

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup: Detect and use systemd

Don't rely on a configuration item to tell us whether or not to use
systemd.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remote: add init_system property

So far, this only exists to answer the question: "does this host use
systemd?" - in which case it will return 'systemd'. Else it will return
None.

Signed-off-by: Zack Cerza <zack@redhat.com>

SystemDState: Treat PIDs as strings when killing

Signed-off-by: Zack Cerza <zack@redhat.com>

Split DaemonState in two

Put systemd stuff in SystemDState

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/daemon: Convert to a subpackage

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Raise on startup failure

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Use consistent daemon IDs

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState.check_status(): support systemd

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Drop use of self.id

... we already had self.id_

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonGroup: Simplify setting self.use_init

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/daemon: Tweak logging

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Factor out PID discovery

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: Don't needlessly reset self.id

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/daemon: Fix linter errors

Signed-off-by: Zack Cerza <zack@redhat.com>

DaemonState: factor out command generation

Signed-off-by: Zack Cerza <zack@redhat.com>

Changes for daemon-helper to work with systemd

when cluster is setup using ceph-ansible or ceph-deploy
use systemd commands to kill/revive deamons.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add required radosgw_interface due to ceph-ansible upstream changes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1111 from ceph/wip-pg-limit

ceph.conf: set new pg max per osd limit

ceph.conf: set new pg max per osd limit

This is replacing the old one, but we need to keep the old one for
upgrades etc.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1091 from ceph/wip-fix-cephlab-overrides

[ansible-doc-fix]: correct the doc to fix the overrides name

Merge pull request #1109 from ceph/wip-openstack-log

task/internal/syslog: blacklist openstack datasource log

task/internal/syslog: blacklist openstack datasource log

Caused OVH node jobs to fail when it shouldn't have

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1107 from ceph/wip-prune-fail

prune: Optionally remove failed jobs entirely

prune: Use summary.yaml to determine age of job

If a job has been partially pruned, its directory's mtime will get
bumped, causing total removal to be delayed by potentially months.
Looking at the mtime of summary.yaml should provide results that are
more intuitive.

Signed-off-by: Zack Cerza <zack@redhat.com>

prune: Optionally remove failed jobs entirely

This works exactly like --pass, except it defaults to being disabled

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1101 from ceph/wip-ansible-fixes

[tasks/ceph-ansible]: use site.yml from the ceph-ansible repo