buildpackages: catch VM instances in ERROR state.

It is possible for 'server create' to return success, but then NOT have
a good VM afterwards. The instance will be in state ERROR.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 785801c2e9fec3fe81b0a1759d137367c9716dcb)

buildpackages: wait for the build machine deletion

When the quotas are low, it matters to block until the build machine is
actually deleted. Otherwise target provisionning may fail because the
they exceed the quota. For instance the default on OVH is to have 32
cores and the build machine uses 16. The packages-repository machine
uses two, the teuthology cluster uses one and that leaves only 13 cores
for the targets which may be too low when running jobs that require
large instances.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit cf6bac279e86d97c234dca6fe23238a484c9f0ac)

Merge pull request #940 from dachary/wip-archs-hammer

buildpackages: add missing arm64 to make-deb.sh

buildpackages: discard google-perftools on trusty aarch64

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c655de7ef322986481cb7c1c452ddb38bf304002)

arch: teuthology must use aarch64, not arm64

teuthology / gitbuilder / ansible_architecture arch is

   dpkg-architecture --query DEB_HOST_GNU_CPU (x86_64 and aarch64)

deb package Architectures: (conf/distributions) is

   dpkg-architecture --query DEB_HOST_GNU_ARCH (amd64 and arm64)

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 5ab482e82c020b9949f362b1aad37431d68332be)

buildpackages: runabove has 30GB root disk max

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 42836aaf2fa0cdcdff9086a406e0c691f8ec6166)

buildpackages: install rsync

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 3e7f311b761d584866614a5227d34d4732de94b3)

buildpackages: add missing arm64 to make-deb.sh

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4f1029102a75e1d3708335474bc57f94a4b3ddb9)

Merge pull request #931 from ceph/wip-thrash-cache-mode-hammer

suites/rados: remove cache mode thrashing test

Reviewed-by: Samuel Just <sjust@redhat.com>

suites/rados: remove cache mode thrashing test

This is known to be unreliable (because the 'forward' mode does
not keep client ops ordered properly).  Disable it.

(Upstream we've replaced 'forward' with 'proxy'.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #929 from dachary/wip-buildpackages-debian-hammer

buildpackages: add debian 8.0 support

buildpackages: add debian 8.0 support

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #604 from dachary/wip-radosbench-hammer

tasks/radosbench: wait just a bit longer

Merge pull request #918 from Vicente-Cheng/wip-15301-hammer

hammer: min_write_recency_for_promote does not exist in hammer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

hammer: min_write_recency_for_promote does not exist in hammer

Fixes #15301

`cache min_write_recency_for_promote` which do not exist
in hammer is introduced as part of proxy write.

This config just remove simply

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #913 from dachary/wip-archs-hammer

archs: add files for each known architecture

buildpackages: do not try tcmalloc on arm64

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b976563af252a63b646a01d3baada4d8064ff1ff)

buildpackages: desambiguate nic on server create

When there are more than one network, it is required to select the one
to be attached to the nic of the instance.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b9b00a5ddb66cf25a133b3476ac6e7ee96dd1923)

dummy: reduce run time, run user.yaml playbook

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d4e3cec1851ae35889127b90912e133178085bc6)

buildpackages: build will work on a range of flavors

When selecting a flavor to build, we prefer the largest we can find and
fallback to smaller ones, down to the minimum acceptable flavor, if
there is no other available.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit a8f4321f5cdbe1bd61a53ebfec6e19541afcee86)

buildpackages: do not hardcode x86_64 in build scripts

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c5182f257c86321e6cb6ba2f1d50432c0b81e6f6)

buildpackages: use architecture suffixed images

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f9c4858a2693abffffc9fd3ca3e8ac40f973a024)

archs: add files for each known architecture

So that they can be prepend to teuthology-suite to run a job on a
designated architecture.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 40277ca6a9943f27aa56ce748ece53e4b69c7473)

Merge pull request #901 from dachary/wip-buildpackages-hammer

buildpackages: provision a small build machine for tests

buildpackages: provision a small build machine for tests

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1cc657798aa0fc7eeebc6de06074e7caca42b24e)

Merge pull request #897 from smithfarm/wip-checkout-sha1-hammer

workunit: git checkout sha1/branch/tag

Reviewed-by: Loic Dachary <ldachary@redhat.com>

workunit: git checkout sha1/branch/tag

git clone --branch refspec only accepts branches and tags but not a sha1
git reset --hard refspec only accepts tags and sha1 but not branches

Use git checkout because it interprets branches, tags and sha1 as
expected.

$ git checkout jewel
Branch jewel set up to track remote branch jewel from origin.
Switched to a new branch 'jewel'
$ git checkout v0.94.6
HEAD is now at e832001... 0.94.6
$ git checkout b2503b0e15c0b13f480f0835060479717b9cf935
HEAD is now at 95cefea... 0.94.3

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d1f456850c6297ca3fbb3ec0daddfddafdeff784)

Merge pull request #889 from ceph/wip-15171-hammer

suites/rados: vary min_{read,write}_recency_for_promote

Reviewed-by: Samuel Just <sjust@redhat.com>

suites/rados: vary min_{read,write}_recency_for_promote

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #884 from dachary/wip-buildpackages-hammer

buildpackages: allow for packages to be re-used for releases

buildpackages: ensure clone to a ceph directory

In case the repository from which the clone is not named ceph.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 63c1a6f9ed1a2c60b26b484b36b8f3a6593249aa)

buildpackages: for source inclusion in debian

We do not build debian versions (i.e. -2 -3 etc). Even if the debian
detection logic infers that the previous debian version already had the
source package, it is incorrect.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit df3eba0f6bc96139bcb8df2e05bfa5eea04f7610)

buildpackages: allow for more parallel builds

Builds are in parallel if the OS type / version and the SHA1 are
different. Having just one build at a time for a given SHA1 slows thing
down when there are more than a few supported OS.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 33b4397da8914205eaecaffc9b4bfb1c692309d0)

buildpackages: enable autoindex in the packages repository

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c60ace31eae65b4fba8c51b4017aa0498b4102f6)

buildpackages: do not requires the long tag version

So that v10.1.0 is used instead of 10.1.0-0-g35ac827 when possible.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b30655eeb673df447dc736e10b040b227911bbbc)

buildpackages: also pull tags from the user ceph fork

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 5e145c1a2f2a650575dc7481be37f5020853a6fd)

buildpackages: create release packages

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit ddd5f7c329327a94c0761276aa9c0073a617f8f3)

Merge pull request #879 from dachary/wip-workunit-hammer

workunit: git-reset --hard does not accept a branch

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

workunit: git-reset --hard does not accept a branch

Uses git clone --branch refspec because it agrees to use either a tag or
a branch.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 96ef6ab6991ae7f6ffff7a2a0d92bcd2de88f497)

Merge pull request #856 from SUSE/wip-opensuse-distros-hammer

distros: add yaml fragments for openSUSE 13.2 and 42.1

Reviewed-by: Loic Dachary <ldachary@redhat.com>

distros: add yaml fragments for openSUSE 13.2 and 42.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 3458adb5d3b749453848fc5edbaeedaeff84853c)

Merge pull request #834 from ceph/wip-14077-hammer

add divergent priors tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #845 from dachary/wip-14931-cram-hammer

cram: pin to version 0.6 because 0.7 handles \r differently

cram: pin to version 0.6 because 0.7 handles \r differently

http://tracker.ceph.com/issues/14931 Fixes: #14931

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 70fb7493f18c45b93d6065c03403c5dd478295f3)

Merge pull request #838 from ceph/wip-14905-hammer

Added v0.94.6 to stable upgrade suite

Added v0.94.6 to stable upgrade suite

Fixed #14905

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rados/singleton/all/pg-removal-interruption: ensure osd.0 is marked down

We were waiting for teh dameon to fail, not for the mons to notice,
which meant that after we restart the daemon the next tell could
try to talk to the old osd.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 07232b1e2bed5911453f30397a5a9de6c9d3125d)

Reduced runtime for radosbench to 600

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit a72bba29103cd287131eb7b56953c417649fed5d)

divergent_priors2: give divergent time to come up

Fixes: #13322
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit c0b0ec2831f410d89b695a1d08122334d8ee907a)

tasks/divergent_priors2: wait for osd to start before tell

Otherwise we can get

2015-09-24T19:22:15.191 INFO:teuthology.orchestra.run.mira080.stderr:Error ENXIO: problem getting command descriptions from osd.1

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ecc504fea0f1c67b768c52d95a41163e25bd065a)

tasks/divergent_prior[2]: fix rados get syntax

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6064443ad1e0dd87801a4508025906d5e26c02c5)

tasks/reg11184.py: fix rados get syntax

no -o

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7b3ea31efeb0da81b3a5ed486cebce72edbf2fec)

thrasher: Can't test ceph-objectstore-tool if nodes turned off (powercycle)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b255db820ffd1d3e1253a9a60c7192696c226490)

reg11184: Add regression test for 11184

Create divergent priors and a split and then move a pg using
ceph-objectstore-tool export/import
Add yaml file to run the reg11184 task

Fixes: #11343
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 38160ecfaaca114dfe2d0e605a4368ef92ec30f6)

divergent_priors2: Do divergent priors and ceph-objectstore-tool export/import

Based on tasks/divergent_priors.py but also do simple export/remove/import on
same osd.
Add yaml file to run the divergent_priors2 task

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b10558021fb3ce14627fedcae992453e4a43d458)

divergent_priors: Fix divergent_priors task

Flake8 fixes
Use new set_recovery_delay admin socket command
Fix bad value set for filestore_blackhole
Make sure log trims and only require 100 objects
Use kick_recovery_wq to properly set osd_recovery_delay_start to 0
Write and remove divergent and verify removal was undone
Fix to make compatible with wip-10809-11135-10290
Make sure to set_recovery_delay in a non-racey way (while osd running but down)
Leave divergent "in" so its PGs aren't treated as strays
Add yaml file to run the divergent_priors task

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a6a4aaf3d3cc6677423ee6d2181254ff52e8b76a)

Thrasher: Fix log message

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ca04dc65c197f54fd6f71806059c4323cc8ee86c)

Merge pull request #825 from ceph/wip-14652-hammer

Fixes #14652 (moved rbd and rgw tests on separate clients, forced s3tests branch)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #14652

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

suites/hammer/older: do not upgrade client while running RBD import/export

Between the Giant and Hammer releases, the symbols exported from librados/librbd
were cleaned up.  This creates a short window of time between installing
upgraded packages where symbol lookup errors could occur.

Fixes: #12563
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit cb7eb2ad6f6c56ac1801c1f032faaf40d42cc955)

Merge pull request #816 from ceph/wip-14618-hammer

Whitelisted soft lockup, detected stalls on CPUs to avoid failures in ov

Whitelisted soft lockup, detected stalls on CPUs to avoid failures in ovh

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #815 from ceph/wip-14618-hammer

Whitelisted

Whitelisted
  soft lockup
  detected stalls on CPUs
to avoide failures in ovh

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #804 from ceph/wip-14388-hammer

Added v0.94.3 to the mix

Added v0.94.3 to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #808 from dachary/wip-smithi-disks-hammer

smithi disks are too fast and fill quickly

rados: 5m radosbench (not 10m)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 04e41a608bf5f271bd062718051597977678d2db)

rados: run radosbench for 10m (not 15m)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6110421bcef2d0d1d4f921822a7161a423d763ca)

rados: run radosbench for 15m instead of 30m

The smithi disks fill after about 20m.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e2f9cedd6426b90af5cf9b04d181e88104849581)

suites/rbd: disable pool validation for pool snaps test

Fixes: #14113
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 46dddfc54b8d650f111007ae812cc81de0e4456c)

Fixed symblink for os configs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 0915747895b20b856a173d58f515af6b03381406)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Change supported ditros to use latest CentOS 7.2

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 5a79aa649f6c55ae003f4a99147a0f70fa1e8659)

Suites/jewel in the smithi lab will be on centos 7.1

Signed-off-by: yuriw <yuri.weinstein@gmail.com>
(cherry picked from commit d4d3787609f351b7efa0d52120858d090d1cf80e)

Merge pull request #773 from ceph/wip-14124-hammer

Fixes issue 14124

Fixes issue 14124

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #771 from dachary/wip-buildpackages-centos-7.1-hammer

buildpackages: support CentOS 7.1 and CentOS 7.2

buildpackages: support CentOS 7.1 and CentOS 7.2

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 834a38cf90774802c92ba34ae28aab6987a5d4d6)

Merge pull request #766 from dachary/wip-buildpackages-log-hammer

Wip buildpackages log hammer

buildpackages: no need to sudo in bootcmd

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 64d8a586f72c71834d71a8147ae12bf395d5319a)

buildpackages: log the output of make

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0c1e7f1f6b6137d573984305d4f7519e97a5cdd6)

buildpackages: pkgrepo host also needs user-data setup.

The packages repo host fails in environments where networking setup is
needed in VMs. Use the same user-data as the buildhosts to ensure this
is the case.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit ebf8508c083b4a1bf39354597a6472034081962c)

buildpackages: copy teuthology user-data.txt changes

7b27e1db7: openstack: support /etc/network/intefaces injection
2358562cf: ensure VMs always have /etc/hosts set up
4378a505d: always allow unsigned deb packages
50b2db521: openstack: encode instance name with the full IP
6e828a33b: openstack: add 8.8.8.8 as a last resort resolver

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 89bde0b8fe9361131bf86ae4f07ae2e4587a00b2)

buildpackages: catch instance creation failure.

Split the sleep from the server creation, so we catch 'server create'
failures (eg due to quota):
> Quota exceeded for cores: Requested 16, but already used 10 of 20 cores
> (HTTP 403) (Request-ID: req-6467934e-db50-4479-995c-4d44dedf553a)

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 3c6954642a3d25db5f5814261c0a4c21a0178607)

buildpackages: Be robust about addresses

OpenStack could tell us the VM has multiple networks, and offers no
guarantee about the order of addresses either (the old code failed if
the v4 IP was first).

For now, take the first listed network, and the first listed IPv4
address therein. Comments contain more detailed examples of possible
output from openstack tool.

Also remove the need for using jq to parse the output.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 0240f5764399e3cea6d9ebe722df2d542e54428b)

Merge pull request #762 from dachary/wip-13624-workunit-git-hammer

workunit: fetch from --ceph-git-url

workunit: fetch from --ceph-git-url

The commit from which workunits are fetched must be retrieved
from --ceph-git-url via teuth_config.get_ceph_git_url() instead of
assuming it is available via git://git.ceph.com/ceph.git.

Using git://git.ceph.com/ceph.git is convenient because it supports git
archive. In the general case, some git servers such as github do not
support git archive and a full git clone must be done instead.

Although it would be possible to

    git clone --branch=master --depth=1 --single-branch

to reduce the amount of data being retrieved, it would require a

    git fetch origin SHA1

but git version >= 1.7 do not support fetching a commit.

http://tracker.ceph.com/issues/13624 Fixes: #13624

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b0a770a3e7754f51c18245cedfff607938631d3e)

workunit: keep the path to the list of tests in a variable

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 28dc5c0c3347b41bae24a4fa7db491002c4fe2ec)

workunit: allow tag to override sha1

The sha1 for the workunit task is always set by the suite.py task. The
tag must be checked before the sha1 othewise it cannot be used to
override the sha1.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7de33f082e7d7108036912d9ce2076fd309d6cd2)

Merge pull request #753 from ceph/wip_fix_fuse_hammer

Fixed missing ceph-fuse

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixed missing ceph-fuse

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #750 from dachary/wip-13910-buildpackages-images-hammer

buildpackages: create the image if it does not exist

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

buildpackages: create the image if it does not exist

Since buildpackages runs before target provisioning, it is possible that
the desired image does not yet exist on a newly provisionned tenant (or
region).

http://tracker.ceph.com/issues/13910 Fixes: #13910

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit ab9a3d5a88c321a4e3dfd71e0263bc984d4dd6db)

Merge pull request #745 from ceph/wip-ceph-whitespace-hammer

tasks/ceph: fix up whitespace

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/ceph: fix up whitespace

...because otherwise it lights up like a christmas
tree in pycharm.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f05d9776281d27df03e74e10085e691669532e95)

Merge pull request #720 from dachary/wip-12748-hammer

rados/singleton-nomsgr/all: add export-after-evict.yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #724 from dachary/wip-openstack-rbd-hammer

openstack: rbd/thrash: allocate three disks, always

openstack: rbd/{thrash,qemu}: allocate three disks, always

The thrasher and qemu need disk attached to run against xfs or btrfs,
otherwise it will use the local file system which is always ext4. It
will succeed but this is a kind of false positive since no xfs or btrfs
related tests were run.

http://tracker.ceph.com/issues/13270 Refs: #13270

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 29973822f8f21044026055a5fd216c3a96ba7eaf)

Merge pull request #722 from dachary/wip-13581-openstack-hints-hammer

openstack: convert to array hints

openstack: convert to array hints (hammer specific)

So that it is more flexible.

http://tracker.ceph.com/issues/13581 Fixes: #13581

Signed-off-by: Loic Dachary <loic@dachary.org>