internal/syslog: whitelist ceph-crash daemon messages

None of these should cause test failures.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #1046 from smithfarm/wip-run-py

run.py: Provide --os-{type,version} values to get_initial_tasks

Merge pull request #1001 from SUSE/wip-18141

platform.linux_distribution() is deprecated; stop using it

Merge pull request #1200 from tchaikov/wip-rerun-fix

suite,report: initialize seed with None

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #1199 from tchaikov/pytest-requires-pluggy-v0.7

setup,py,requirements.txt: add pytest

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

suite,report: initialize seed with None

so we can set it if it is not initialized

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup,py,requirements.txt: add pytest

pytest requires pluggy >= 0.7, while we always use pluggy 0.6, as
specified by requirements.txt. as this version is good enough for
tox. but in tox.ini, we do use pytest, and no version is specified,
so we have good chance running into https://github.com/pytest-dev/pytest/issues/3753

also, remove pytest from tox.ini, as this dependency has been
added in requirements.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1197 from tchaikov/wip-str.replace

task/install: str.replace() does not keyword args

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

task/install: str.replace() does not keyword args

In [8]: ?s.replace
Docstring:
S.replace(old, new[, count]) -> str

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1196 from tchaikov/wip-rerun-fix

report: strip space and quotes from parsed message

report: strip space and qutoes from parsed message

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1195 from ceph/revert-1090-wip-upgrade-equal

Revert "[DNM] upgrade check: upgrade to same version is a noop"

Revert "[DNM] upgrade check: upgrade to same version is a noop"

Merge pull request #1090 from smithfarm/wip-upgrade-equal

[DNM] upgrade check: upgrade to same version is a noop

Merge pull request #1193 from adamwg/handle-branches-with-slashes

Handle branch names containing slashes

Handle branch names containing slashes

A branch name containing a slash is perfectly legal in git, but
teuthology uses branch names verbatim in run names, which causes POSTs
to fail when submitting runs to paddles. Replace all '/' in run names
with ':' to allow for branches with slashes in their names.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

orchestra/opsys: support openSUSE Leap

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests/opsys: add more OS.from_os_release test cases

Signed-off-by: Nathan Cutler <ncutler@suse.com>

orchestra/opsys: add new distros to codenames map

Signed-off-by: Nathan Cutler <ncutler@suse.com>

platform.linux_distribution() is deprecated; stop using it

Fixes: http://tracker.ceph.com/issues/18141
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1191 from ceph/wip-ansible-default

default ansible to 2.5 as per ceph-ansible requirements

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

default ansible to 2.5 as per ceph-ansible requirements for ceph master branch testing.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1189 from tchaikov/wip-fix-rerun

results,schedule,woker: persist --seed and --subset in results.log

Reviewed-by: Warren Usui <wusui@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1190 from ceph/install_nfs_ganesha_with_ceph-ansible

added nfs-ganesha to roles

added nfs-ganesha to roles

test: '$' is not removed from path even if triggers random tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

results,schedule,woker: persist --seed and --subset in results.log

to create a repeatable test suite, in addition to `--seed <SEED>`, we also
need to pass the same `--subset <SUBSET>` to teuthology-suite when
rerunning the failed tests. but it would be handy if teuthology-suite
could remember these settings and recall them when `--rerun <RUN>`.

in this change, we repurpose the last job sending the email to report
the test result to note down the subset and seed used for scheduling the
test suite. these variables are stored in results.log at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #1186 into master

* refs/pull/1186/head:
ceph.conf.template: drop cephfs configs

Reviewed-by: Zheng Yan <zyan@redhat.com>

ceph.conf.template: drop cephfs configs

This will be set in the cephfs qa suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1187 from SUSE/wip-restart-workers

Restart dead workers

Merge pull request #1188 from SUSE/wip-job-owner

Fix worker can't figure out owner of a job

Fix worker can't figure out owner of a job

The patch fixes uncaught exception RuntimeError:

  I could not figure out the owner of the requested job.
  Please pass --owner <owner>.

Worker dies with unhandled exception in run_with_watchdog
if it can't figure out owner of a job, which it tries
to kill when job runs longer then given limit of time.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Restart dead workers

This patch allows to restart dead workers separately
not stopping the rest of the teuthology components,
and what is more important the beanstalkd service.
That makes it possible to extend the number of workers too.
Also, either of pulpito and paddles can be restarted alone.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1185 from batrick/hidden-file

build_matrix: ignore hidden files

test: add hidden file test

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

build_matrix: ignore hidden files

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1182 from tchaikov/wip-rerun

suite: allow `--rerun` to run full set of failed tests

teuthology-suite: add --seed option for repeatable random test

currently --rerun does not match tests of
'supported-random-distro$/ubuntu_latest.yaml' with
'supported-random-distro$/centos_latest.yaml'. the former could be part
of description of a failed test, the latter is a a part of job
description generated by build_matrix(). because the '$' operator
instructs theuthology to choose a random file under the directory ending
with '$', and we expand the '$' to a randomly picked file *before*
filtering the generated job list with the filter collected from the
failed tests, there is good chance that the job descriptions of the
failed jobs in self.args.filter_in cannot match with the randomly
generated ones.

so, we introduce an argument '--seed' for teuthology-suite for the
repeatable random test. this argument allows user to specify a seed for
tne RNG used by build_matrix().

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

suite: avoid preparing the full list for filtering

speed up the filtering a little bit

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1180 from SUSE/wip-fix-fetch_binaries_for_coredumps

Fix fetch_binaries_for_coredumps

Fix fetch_binaries_for_coredumps

Addresses error message:

  AttributeError: 'tuple' object has no attribute 'split'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1122 from ceph/wip-restart-daemon

[orchestra/systemd]: check status when daemon is restarted

Merge pull request #1176 from cbodley/wip-ansible-requirements

ceph-ansible: add hard-coded notario dependency

ceph-ansible: add hard-coded notario dependency

Fixes: http://tracker.ceph.com/issues/24230
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1175 from ceph/wip-24168

Unpin libvirt-python

Unpin libvirt-python

http://tracker.ceph.com/issues/24168

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1174 from ceph/wip-suite-tests

suite/test/test_run_.py: Don't hit the network!

suite/test/test_run_.py: Don't hit the network!

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1173 from ceph/wip-vault-pass

ansible.py: Write "foo" to ~/.vault_pass.txt instead of touching

ansible.py: Write "foo" to ~/.vault_pass.txt instead of touching

ansible-playbook will not run with an empty vault password file.  It
will run if the password file has something in it even if it doesn't get
used.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1165 from ceph/wip-wusui-23208

Allow both $ and directory$ for random yamls.

Allow both $ and directory$ for random yamls.

If either the diretory contains a magic $ file, or if the directory name
ends with $, then the random selection of a yaml file will occur.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1172 from batrick/slow-ops-whitelist

remove blanket SLOW_OPS whitelist

remove blanket SLOW_OPS whitelist

Should no longer be necessary after [1].

[1] https://github.com/ceph/ceph/pull/21684

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1170 from ceph/wip-bionic

Add Bionic to distro map

Add Bionic to distro map

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1167 from tchaikov/wip-job-404

teuthology/lock: ignore none 200 jobs in node_job_is_active()

Merge pull request #1169 from ceph/wip-23798

task.selinux: Whitelist syslogd_t denials

task.selinux: Whitelist syslogd_t denials

Fixes: http://tracker.ceph.com/issues/23798
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1168 from tchaikov/wip-mds-all-down

placeholder: whitelist MDS_ALL_DOWN by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

placeholder: whitelist MDS_ALL_DOWN, MDS_UP_LESS_THAN_MAX  by default

because, in ceph/qa/tasks/ceph.py, we start mon, mgr, osd, and then mds.
there is a time window where there is no mds around, but mgr is checking
mdsmap for MDS_ALL_DOWN errors. there is no way to disable this check in
this time window. so we just whitelist MDS_ALL_DOWN here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/lock: ignore none 200 jobs in node_job_is_active()

there is chance that we will have 404 when accessing a job's URI.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1166 from batrick/i23662

placeholder: update whitelist for osd slow op wrn

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

placeholder: update whitelist for osd slow op wrn

Caused by: https://github.com/ceph/ceph/pull/20660 (ea97c120d2173f2fc70d979d57a7edb2a6c5da5e)

Fixes: https://tracker.ceph.com/issues/23662
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1160 from ceph/wip-intro-link-new-wiki

Update getting started instructions to link to the new sepia wiki

Merge pull request #1154 from ceph/wip-distro-head

task.kernel: Only show latest kernel when running rpm -q kernel

Merge pull request #1164 from ceph/wip-skip-tags

ansible: Add ability to skip tags during ansible task

ansible: Add ability to skip tags during ansible task

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1161 from ceph/wip-ansible-group-vars

task.ansible: Allow passing in custom group_vars

Overhaul ansible task tests

This fixes failures that have only manifested in Jenkins; it also causes
CephLab to run more of the Ansible tests.

Signed-off-by: Zack Cerza <zack@redhat.com>

Update tox again

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal.syslog: Whitelist a docker warning

Signed-off-by: Zack Cerza <zack@redhat.com>

pytest.ini: Disable cacheprovider

A change in 3.5.0 seems to be buggy; disable the cache rather than
pinning versions.

I believe the offending commit is:
https://github.com/pytest-dev/pytest/commit/dff0500114971b30a7bb9043acb0d0fb6a9e01c4

Signed-off-by: Zack Cerza <zack@redhat.com>

task.ansible: Allow passing in custom group_vars

Up until now, if you wanted to inject vars to a playbook run, you had to
use --extra-vars, which don't behave the same way that group_vars do.
This commit adds that functionality.

We look for a 'group_vars' dict in the task's config object. If it's
there, we create group_vars files with names taken from the keys, and
content taken from the values.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1157 from ceph/wusui-23208

Implement $ option to randomly choose yamls.

Update getting started instructions to link to the new sepia wiki

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Add unit tests for '$' file.

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1158 from ceph/wip-wait-fog

Increase timeouts teuthology will wait for FOG provisioning

fog: Wait 10 minutes for machine to be reachable after deploy

A lot's going on in rc.local after a machine is provisioned with FOG.  5
minutes is a little aggressive when taking into account the time it
takes for:
 - The machine to reboot after the FOG task completes
 - BIOS to load
 - DHCP/PXE/TFTP to timeout (double this if NIC order isn't correct)
 - OS to boot and rc.local to do its magic

Signed-off-by: David Galloway <dgallowa@redhat.com>

fog: Wait 15 minutes for FOG task to complete

When there are running jobs and a large run gets scheduled, FOG
provisioning gets backed up due to its built-in rate limiting.  15
minutes should be a little more lenient and prevent large spikes of dead
jobs when a run is scheduled in an idle queue.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Implement $ option to randomly choose yamls.

This implements tracker #23208

Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1156 from ceph/wip-libvirt-sdist

Drop libvirt from setup.py

Drop libvirt from setup.py

Having an unversioned libvirt-python in setup.py started causing
problems with tox, since it uses an sdist archive as a basis for
installing teuthology in its virtualenvs. Removing it is consistent with
best practices. We'll still keep it in requirements.txt.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.kernel: Only show latest kernel when running rpm -q kernel

Fixes: http://tracker.ceph.com/issues/23381
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1152 from ceph/wip-rhsm-selinux

task.selinux: Whitelist rhsmd denials

task.selinux: Whitelist rhsmd denials

These started showing up once we added RHEL to Sepia.

Fixes: https://tracker.ceph.com/issues/23343#note-5
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1148 from badone/wip-fedora-community-mysql-devel-depend

bootstrap: Rename mysql-community-devel for fedora

Merge pull request #1150 from ceph/pytest-3.4

Unbreak testing with py.test 3.4

Update tox

Signed-off-by: Zack Cerza <zack@redhat.com>

Unbreak py.test

https://docs.pytest.org/en/latest/logging.html#incompatible-changes-in-pytest-3-4

Signed-off-by: Zack Cerza <zack@redhat.com>

Update ansible and cryptography

via pip-compile -P ansible -P cryptography

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Rename mysql-community-devel for fedora

The name of the package is community-mysql-devel in currently
supported releases.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #1146 from ceph/wip-allow-more-drift

ceph.conf: mon_clock_drift_allowed .5 -> 1.0

ceph.conf: mon_clock_drift_allowed .5 -> 1.0

All of the errors I see seem to be between .5 and .9s.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1145 from jcsp/wip-mgr-valgrind

install: extend python-related valgrind rules

Reviewed-By: Kefu Chai <kchai@redhat.com>

install: extend python-related valgrind rules

These were set for kind 'possible' which was failing
to suppress Leak_DefinitelyLost errors.

Also add several suppressions related to module
initialization.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1144 from tchaikov/wip-22438

teuthology/task/install/valgrind.supp: add suppression for dlopen()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install/valgrind.supp: add suppression for dlopen()

should cover the case of malloc() in addition to calloc()

Fixes: http://tracker.ceph.com/issues/22438
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1143 from tchaikov/wip-22438

teuthology/task/install/valgrind.supp: add suppression for dlopen()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install/valgrind.supp: add suppression for dlopen()

the analysis in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700899
also applies to ceph-common.

Fixes: http://tracker.ceph.com/issues/22438
Signed-off-by: Kefu Chai <kchai@redhat.com>