tasks/reg11184.py: fix rados get syntax

no -o

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7b3ea31efeb0da81b3a5ed486cebce72edbf2fec)

thrasher: Can't test ceph-objectstore-tool if nodes turned off (powercycle)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b255db820ffd1d3e1253a9a60c7192696c226490)

reg11184: Add regression test for 11184

Create divergent priors and a split and then move a pg using
ceph-objectstore-tool export/import
Add yaml file to run the reg11184 task

Fixes: #11343
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 38160ecfaaca114dfe2d0e605a4368ef92ec30f6)

divergent_priors2: Do divergent priors and ceph-objectstore-tool export/import

Based on tasks/divergent_priors.py but also do simple export/remove/import on
same osd.
Add yaml file to run the divergent_priors2 task

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b10558021fb3ce14627fedcae992453e4a43d458)

divergent_priors: Fix divergent_priors task

Flake8 fixes
Use new set_recovery_delay admin socket command
Fix bad value set for filestore_blackhole
Make sure log trims and only require 100 objects
Use kick_recovery_wq to properly set osd_recovery_delay_start to 0
Write and remove divergent and verify removal was undone
Fix to make compatible with wip-10809-11135-10290
Make sure to set_recovery_delay in a non-racey way (while osd running but down)
Leave divergent "in" so its PGs aren't treated as strays
Add yaml file to run the divergent_priors task

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a6a4aaf3d3cc6677423ee6d2181254ff52e8b76a)

Thrasher: Fix log message

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ca04dc65c197f54fd6f71806059c4323cc8ee86c)

Merge pull request #825 from ceph/wip-14652-hammer

Fixes #14652 (moved rbd and rgw tests on separate clients, forced s3tests branch)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #14652

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

suites/hammer/older: do not upgrade client while running RBD import/export

Between the Giant and Hammer releases, the symbols exported from librados/librbd
were cleaned up.  This creates a short window of time between installing
upgraded packages where symbol lookup errors could occur.

Fixes: #12563
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit cb7eb2ad6f6c56ac1801c1f032faaf40d42cc955)

Merge pull request #816 from ceph/wip-14618-hammer

Whitelisted soft lockup, detected stalls on CPUs to avoid failures in ov

Whitelisted soft lockup, detected stalls on CPUs to avoid failures in ovh

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #815 from ceph/wip-14618-hammer

Whitelisted

Whitelisted
  soft lockup
  detected stalls on CPUs
to avoide failures in ovh

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #804 from ceph/wip-14388-hammer

Added v0.94.3 to the mix

Added v0.94.3 to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #808 from dachary/wip-smithi-disks-hammer

smithi disks are too fast and fill quickly

rados: 5m radosbench (not 10m)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 04e41a608bf5f271bd062718051597977678d2db)

rados: run radosbench for 10m (not 15m)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6110421bcef2d0d1d4f921822a7161a423d763ca)

rados: run radosbench for 15m instead of 30m

The smithi disks fill after about 20m.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e2f9cedd6426b90af5cf9b04d181e88104849581)

suites/rbd: disable pool validation for pool snaps test

Fixes: #14113
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 46dddfc54b8d650f111007ae812cc81de0e4456c)

Fixed symblink for os configs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 0915747895b20b856a173d58f515af6b03381406)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Change supported ditros to use latest CentOS 7.2

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 5a79aa649f6c55ae003f4a99147a0f70fa1e8659)

Suites/jewel in the smithi lab will be on centos 7.1

Signed-off-by: yuriw <yuri.weinstein@gmail.com>
(cherry picked from commit d4d3787609f351b7efa0d52120858d090d1cf80e)

Merge pull request #773 from ceph/wip-14124-hammer

Fixes issue 14124

Fixes issue 14124

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #771 from dachary/wip-buildpackages-centos-7.1-hammer

buildpackages: support CentOS 7.1 and CentOS 7.2

buildpackages: support CentOS 7.1 and CentOS 7.2

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 834a38cf90774802c92ba34ae28aab6987a5d4d6)

Merge pull request #766 from dachary/wip-buildpackages-log-hammer

Wip buildpackages log hammer

buildpackages: no need to sudo in bootcmd

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 64d8a586f72c71834d71a8147ae12bf395d5319a)

buildpackages: log the output of make

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0c1e7f1f6b6137d573984305d4f7519e97a5cdd6)

buildpackages: pkgrepo host also needs user-data setup.

The packages repo host fails in environments where networking setup is
needed in VMs. Use the same user-data as the buildhosts to ensure this
is the case.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit ebf8508c083b4a1bf39354597a6472034081962c)

buildpackages: copy teuthology user-data.txt changes

7b27e1db7: openstack: support /etc/network/intefaces injection
2358562cf: ensure VMs always have /etc/hosts set up
4378a505d: always allow unsigned deb packages
50b2db521: openstack: encode instance name with the full IP
6e828a33b: openstack: add 8.8.8.8 as a last resort resolver

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 89bde0b8fe9361131bf86ae4f07ae2e4587a00b2)

buildpackages: catch instance creation failure.

Split the sleep from the server creation, so we catch 'server create'
failures (eg due to quota):
> Quota exceeded for cores: Requested 16, but already used 10 of 20 cores
> (HTTP 403) (Request-ID: req-6467934e-db50-4479-995c-4d44dedf553a)

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 3c6954642a3d25db5f5814261c0a4c21a0178607)

buildpackages: Be robust about addresses

OpenStack could tell us the VM has multiple networks, and offers no
guarantee about the order of addresses either (the old code failed if
the v4 IP was first).

For now, take the first listed network, and the first listed IPv4
address therein. Comments contain more detailed examples of possible
output from openstack tool.

Also remove the need for using jq to parse the output.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 0240f5764399e3cea6d9ebe722df2d542e54428b)

Merge pull request #762 from dachary/wip-13624-workunit-git-hammer

workunit: fetch from --ceph-git-url

workunit: fetch from --ceph-git-url

The commit from which workunits are fetched must be retrieved
from --ceph-git-url via teuth_config.get_ceph_git_url() instead of
assuming it is available via git://git.ceph.com/ceph.git.

Using git://git.ceph.com/ceph.git is convenient because it supports git
archive. In the general case, some git servers such as github do not
support git archive and a full git clone must be done instead.

Although it would be possible to

    git clone --branch=master --depth=1 --single-branch

to reduce the amount of data being retrieved, it would require a

    git fetch origin SHA1

but git version >= 1.7 do not support fetching a commit.

http://tracker.ceph.com/issues/13624 Fixes: #13624

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b0a770a3e7754f51c18245cedfff607938631d3e)

workunit: keep the path to the list of tests in a variable

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 28dc5c0c3347b41bae24a4fa7db491002c4fe2ec)

workunit: allow tag to override sha1

The sha1 for the workunit task is always set by the suite.py task. The
tag must be checked before the sha1 othewise it cannot be used to
override the sha1.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7de33f082e7d7108036912d9ce2076fd309d6cd2)

Merge pull request #753 from ceph/wip_fix_fuse_hammer

Fixed missing ceph-fuse

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixed missing ceph-fuse

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #750 from dachary/wip-13910-buildpackages-images-hammer

buildpackages: create the image if it does not exist

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

buildpackages: create the image if it does not exist

Since buildpackages runs before target provisioning, it is possible that
the desired image does not yet exist on a newly provisionned tenant (or
region).

http://tracker.ceph.com/issues/13910 Fixes: #13910

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit ab9a3d5a88c321a4e3dfd71e0263bc984d4dd6db)

Merge pull request #745 from ceph/wip-ceph-whitespace-hammer

tasks/ceph: fix up whitespace

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/ceph: fix up whitespace

...because otherwise it lights up like a christmas
tree in pycharm.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f05d9776281d27df03e74e10085e691669532e95)

Merge pull request #720 from dachary/wip-12748-hammer

rados/singleton-nomsgr/all: add export-after-evict.yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #724 from dachary/wip-openstack-rbd-hammer

openstack: rbd/thrash: allocate three disks, always

openstack: rbd/{thrash,qemu}: allocate three disks, always

The thrasher and qemu need disk attached to run against xfs or btrfs,
otherwise it will use the local file system which is always ext4. It
will succeed but this is a kind of false positive since no xfs or btrfs
related tests were run.

http://tracker.ceph.com/issues/13270 Refs: #13270

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 29973822f8f21044026055a5fd216c3a96ba7eaf)

Merge pull request #722 from dachary/wip-13581-openstack-hints-hammer

openstack: convert to array hints

openstack: convert to array hints (hammer specific)

So that it is more flexible.

http://tracker.ceph.com/issues/13581 Fixes: #13581

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: convert to array hints

So that it is more flexible.

http://tracker.ceph.com/issues/13581 Fixes: #13581

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8f2af5949c8ed4e24cedd8a6df0d169733e14926)

Conflicts:
suites/smoke/1node/clusters/openstack.yaml (does not exist)

rados/singleton-nomsgr/all: add export-after-evict.yaml

add a test for http://tracker.ceph.com/issues/12748

Fixes: #12748
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a3442cf07841a7269713e30ca270f2f9c356a598)

Merge pull request #717 from dachary/wip-dumpling-parallel-hammer

buildpackages: do not -jX on dumpling

buildpackages: do not -jX on dumpling

http://tracker.ceph.com/issues/13845 Refs: #13845

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f398b71e339b4617db180c8140f23d004f445a24)

Merge pull request #713 from dachary/wip-buildpackages-refactor-hammer

  buildpackages: implement get_pkg_type & avoid duplicate packages-repository

buildpackages: avoid duplicate packages-repository

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 87e50d394a26682614ab3ec8f566bda9f30e8416)

buildpackages: implement get_pkg_type

It can be deprectated when it is implemented in teuthology.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 13fb23ed69e6b7dfa5c005e81abc6b983c019ba4)

Merge pull request #711 from dachary/wip-buildpackages-refactor-hammer

buildpackages: refactor to not require remote

buildpackages: refactor to not require remote

Most of the flavor, sha1, tag etc. selection logic as implemented in the
packaging module of teuthology relies on remote hosts. This is complex
to tests and inconvenient because hosts must be provisionned even before
trying to figure out which packages need to be installed.

Using remote hosts is necessary when bare metal targets are used because
teuthology must adapt to the operating system already installed. The
selection logic in the context of dynamically provisionned targets is
simpler because it is defined by the job being run.

The buildpackages is refactored to use only the job configuration to
figure out which packages must be built. It makes it specific to targets
that are dynamically provisionned. It would have to be modified to query
the remote host in the case of bare metal targets.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 69e9b22e4baa06705119cc818f3b2c1a472c31f6)

Merge pull request #705 from dachary/wip-buildpackages-packaging-hammer

buildpackages: protect packages-repository

buildpackages: protect packages-repository

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4825f64a7c22320b034b020093489bf86a953fca)

Merge pull request #702 from dachary/wip-buildpackages-giant-hammer

buildpackages: do not fails for unpackaged files

buildpackages: do not fails for unpackaged files

Older versions of Ceph have that problem failing would break all upgrade suites.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8d798c4eec508a5e2433fc55ab072c9105855a18)

Merge pull request #696 from dachary/wip-buildpackages-flavor-hammer

buildpackages: 10GB is too small most of the time

buildpackages: 10GB is too small most of the time

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 129c5db6c9bc41faa5c52df340776f00ab2db2b6)

Merge pull request #685 from dachary/wip-12973-hammer

failed to encode map e428 with expected crc

buildpackages: giant has some unpackaged files

And there really is nothing we can do about it expect ignore them.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados: 'failed to encode ...' warnings are normal on upgrades (part 2)

http://tracker.ceph.com/issues/12973 Fixes: #12973

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4cbf1770397e4c97aa1d0486ac7f7a3e1da98771)

Conflicts:
suites/upgrade/hammer/point-to-point/point-to-point.yaml does not exist in hammer

rados: 'failed to encode ...' warnings are normal on upgrades

Fixes: #12973
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3084453b047ca8b6bd0ce56e4e5a7ecce71753bc)

Conflicts:
suites/rados/upgrade/hammer-x-singleton/3-thrash/default.yaml does not exist in hammer

Merge pull request #683 from dachary/wip-13451-hammer

rados/objectstore/alloc-hint: need three attached disks

rados/objectstore/alloc-hint: need three attached disks

So they can be mkfs xfs.

http://tracker.ceph.com/issues/13451 Fixes: #13451

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1a63fd73de3c3de8cff16d486496b1fc5c4cd3b4)

Merge pull request #673 from dachary/wip-buildpackage-ipv6-hammer

buildpackages: silently ignore IPv6 addresses

buildpackages: silently ignore IPv6 addresses

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7b8f205952d5e49609efbd77a1becae3aff4e531)

Merge pull request #671 from ceph/wip-13234-hammer

rados: add test for 13234.yaml

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #653 from ceph/wip-13567-hammer

suites/rbd/qemu: excercise CoW support during QEMU testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #606 from dachary/wip-12320-ceph-fuse-hammer

tasks/ceph_fuse.py: virtual machines need flexible mount timeout

Merge pull request #659 from dachary/wip-buildpackages-tags-hammer

buildpackages: fetch tags from the official Ceph repository

buildpackages: fetch tags from the official Ceph repository

A clone of Ceph is not automagically updated with the tags from the
official Ceph repository. For a pull request based on master, git
describe will use whatever tags existed at the time the clone was made,
unless the author pull them from the official Ceph repository and later
git push --tags them.

The output of git describe is used to name the packages and if the
official tags are not present, the packages will be incorrectly
named. For instance instead of 9.0.3-34 the packages could be named
0.87-8433 because the v0.87 tag is the most recent tag in the
repository. That confuses the install task that will fail with:

   'ceph version 0.87 was not installed, found 9.0.3.'

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 970a37226f884c7f109e48c90373b7fd2697f6d0)

Merge pull request #593 from dachary/wip-13000-centos-activate-hammer

ceph-deploy: always zap disk before creating an osd

suites/rbd/qemu: excercise CoW support during QEMU testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 59e768bb29960b4ca411a71342ef982bb2499354)

Merge pull request #651 from ceph/wip-fix-hammer-stable

Added v0.94.4 to the mix

Added v0.94.4 to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #650 from ceph/wip-openstack-fixes-fs_hammer

Added openstack.yaml config

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #626 from ceph/wip-13450-openstack-rados-thrash

openstack: rados/thrash: allocate three disks, always

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #627 from dachary/wip-13449-admin-socket-hammer

tasks/admin_socket.py: wait 120 seconds instead of 60

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #634 from dachary/wip-11260-openstack-singleton-nomsgr-hammer

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

Reviewed-by: Kefu Chai <kchai@redhat.com>

Added openstack.yaml config
Moved it to tasks/cfuse_workunit_kernel_untar_build.yaml

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #637 from dachary/wip-openstack-ceph-deploy-hammer

ceph-deploy: hints for OpenStack

ceph-deploy: hints for OpenStack

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 3da86724c3db305f395bce5ba2175258294dadb6)

Merge pull request #636 from dachary/wip-distro-supported

hammer now only supports CentOS 7 and Ubuntu 14.04

hammer now only supports CentOS 7 and Ubuntu 14.04

Signed-off-by: Loic Dachary <loic@dachary.org>

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

If given only 8GB RAM,  ceph_test_msgr may abort with buffer::bad_alloc.

http://tracker.ceph.com/issues/11260 Fixes: #11260

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit eac791c664ceed4bd3ac41b0d81ffc337b55e047)

Merge pull request #631 from dachary/wip-isa-distro-hammer

erasure-code: isolate isa plugin arch constraint

erasure-code: isolate isa plugin arch constraint

Link the distro directory to the directory containing all supported
distros. Add the x86_64 arch contraint required by the isa plugin to an
isolated file that is combined with all jobs.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 65565de50dbc841cc2a8254317140131348b65d9)

Merge pull request #620 from ceph/wip-13366

Removed centos 6.5 from config, fixes #13366

Merge pull request #629 from dachary/wip-13454-openstack-flavor-hammer

buildpackages: implement notcmalloc flavor

buildpackages: implement notcmalloc flavor

http://tracker.ceph.com/issues/13454 Fixes: #13454

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit baa8ea327e136197072faffedbe23087c9db2cdb)

tasks/admin_socket.py: wait 120 seconds instead of 60

When running on virtual machines, it may take more than one minute for a
daemon to create the admin socket.

http://tracker.ceph.com/issues/13449 Fixes: #13449

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c0828cae19a49d73de02e5cc4a1ac51403b98dec)

openstack: rados/thrash: allocate three disks, always

The thrasher needs disk attached to run against xfs, ext4 or btrfs. And
some jobs use more disks and do not fit in 40GB.

http://tracker.ceph.com/issues/13450 Fixes: #13450

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #625 from dachary/wip-buildpackages-submodules-hammer

buildpackages: get ceph submodules

buildpackages: get ceph submodules

Prior to v0.80.9, autogen.sh did not get submodules. Copy/paste the
submodule initialization from newer autogen.sh in common.sh so that
v0.80.8 and below can be rebuilt from sources. It does not hurt to
update the submodules twice.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 81f5c3ad0374039280030fba239bf46934584829)