Merge pull request #1125 from ceph/wip_fix_hammer_points

Removed old giant install step

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed old giant istall step

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1102 from SUSE/wip-osd-recovery-hammer

osd_recovery: add delay to ensure maps propogate

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #1104 from smithfarm/wip-16847

Wip 16847

Reviewed-by: Samuel Just <sjust@redhat.com>

fix indentation error in docstring

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit af973259eff9ffbbbe9aa209c3b3dd2d80d7219b)

tox.ini: exclude .tox dir from flake8

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 50f32756a472c29cc6c18a128e5b6cc99c9371e5)

do not ommit syntax errors in linter checks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit db69142b9078fd9ed35fb3a542fc340dae9be665)

repair_test: Disable scheduled scrubbing to not mess up do_pg_scrub()

Fixes: http://tracker.ceph.com/issues/15679
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5c0edbae3eb35654b4efb2e6eb097014953aea12)

ceph_manager: do_pg_scrub: keep scrubbing until it's done

The ceph pg scrub ... command isn't really guarranteed to
start a scrub, keep reissuing it until the scrub actually
happens.

Related: #12746
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 89dcc0daf31eabc04810f759e0694dc8303fee4d)

osd_recovery: add delay to ensure maps propogate

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit ddabdb7f722b192435b87bc31fe587b93cc53924)

Merge pull request #1089 from smithfarm/wip-832-hammer

Wip 832 hammer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

divergent_priors: add sleep after reviving divergent

Basically same thing as
c0b0ec2831f410d89b695a1d08122334d8ee907a
client is getting ENXIO when trying to do osd tell because
the osd isn't quite up yet.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit af3af30418902fb0ff436d7207582bae0226a05a)

rados/singleton: use xfs instead

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 7e0fb5dd617c2f87ef4bc952f7f5a26b4205db85)

ceph_manager: use time before mon command for timeout

Slow mon commands can cause a false failure.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 8cf25611fb572b2f0db4b5670a4b9d2ea7944b5a)

tasks/ceph_manager: dump pgs when recover times out

It is really hard to map a stuck recovery back to the pgs that
are stuck.  This will make it easy.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a328e3e60d0c686333d06b86841d02de27a873be)

tasks/ceph_manager: dump pgs if other peering timeouts expire

We were doing this for one of the recovery timeouts but not all of them.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6deba7c6497cc19f3142faf785274a086f779013)

Merge pull request #1023 from smithfarm/wip-buildpackages-hammer

openstack: ovh renamed flavors : s/eg/hg/

Reviewed-by: Loic Dachary <ldachary@redhat.com>

openstack: ovh renamed flavors : s/eg/hg/

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4d47b2d1b0fae2efcc331fbf2a6f8c576bb4aad3)

Merge pull request #998 from smithfarm/wip-opensuse-make-rpm-hammer

make-rpm.sh: support openSUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

make-rpm.sh: support openSUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit b7cbe2f96fa2ebb08f48f1fcb89fa591ec7ea1e2)

Merge pull request #995 from smithfarm/wip-opensuse-user-data-hammer

add user-data for openSUSE 42.1

Reviewed-by: Loic Dachary <ldachary@redhat.com>

add user-data for openSUSE 42.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 209ce61788117a9c9b7fa8333504f1e63538f0c2)

Merge pull request #952 from dachary/wip-buildpackages-xenial-hammer

buildpackages: add ubuntu 16.04 xenial support

buildpackages: add ubuntu 16.04 xenial support

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #949 from dachary/wip-pr-763-hammer

buildpackages: wait for the build machine deletion

buildpackages: add build timeouts.

The build/upload process CAN get stuck sometimes. As we are paying
utility rates for the high-power build server, we should timeout when
it's likely the build is stuck.

Use a default timeout just short of 4 hours, as some providers round up
partial instance-hours for billing.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit a33517a88f27df9c13a013ad3378aab050ddbea7)

buildpackages: support ubuntu 12.04.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit df5d147094281b0cb77646e529c5f1a597c72d38)

buildpackages: fix configure on Ubuntu 12.04.

Ubuntu 12.04/precise has a buggy lttng; it fails our lttng-is-sane
check, as it leaves out headers.

This should not stop building of Ceph, as the automatic build handling
from debian/rules will correctly omit LTTNG on Precise.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit cd4b7ad6428523a968b38dc9aa1edaa35be96c62)

buildpackages: catch VM instances in ERROR state.

It is possible for 'server create' to return success, but then NOT have
a good VM afterwards. The instance will be in state ERROR.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>
(cherry picked from commit 785801c2e9fec3fe81b0a1759d137367c9716dcb)

buildpackages: wait for the build machine deletion

When the quotas are low, it matters to block until the build machine is
actually deleted. Otherwise target provisionning may fail because the
they exceed the quota. For instance the default on OVH is to have 32
cores and the build machine uses 16. The packages-repository machine
uses two, the teuthology cluster uses one and that leaves only 13 cores
for the targets which may be too low when running jobs that require
large instances.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit cf6bac279e86d97c234dca6fe23238a484c9f0ac)

Merge pull request #940 from dachary/wip-archs-hammer

buildpackages: add missing arm64 to make-deb.sh

buildpackages: discard google-perftools on trusty aarch64

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c655de7ef322986481cb7c1c452ddb38bf304002)

arch: teuthology must use aarch64, not arm64

teuthology / gitbuilder / ansible_architecture arch is

   dpkg-architecture --query DEB_HOST_GNU_CPU (x86_64 and aarch64)

deb package Architectures: (conf/distributions) is

   dpkg-architecture --query DEB_HOST_GNU_ARCH (amd64 and arm64)

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 5ab482e82c020b9949f362b1aad37431d68332be)

buildpackages: runabove has 30GB root disk max

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 42836aaf2fa0cdcdff9086a406e0c691f8ec6166)

buildpackages: install rsync

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 3e7f311b761d584866614a5227d34d4732de94b3)

buildpackages: add missing arm64 to make-deb.sh

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4f1029102a75e1d3708335474bc57f94a4b3ddb9)

Merge pull request #931 from ceph/wip-thrash-cache-mode-hammer

suites/rados: remove cache mode thrashing test

Reviewed-by: Samuel Just <sjust@redhat.com>

suites/rados: remove cache mode thrashing test

This is known to be unreliable (because the 'forward' mode does
not keep client ops ordered properly).  Disable it.

(Upstream we've replaced 'forward' with 'proxy'.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #929 from dachary/wip-buildpackages-debian-hammer

buildpackages: add debian 8.0 support

buildpackages: add debian 8.0 support

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #604 from dachary/wip-radosbench-hammer

tasks/radosbench: wait just a bit longer

Merge pull request #918 from Vicente-Cheng/wip-15301-hammer

hammer: min_write_recency_for_promote does not exist in hammer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

hammer: min_write_recency_for_promote does not exist in hammer

Fixes #15301

`cache min_write_recency_for_promote` which do not exist
in hammer is introduced as part of proxy write.

This config just remove simply

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

Merge pull request #913 from dachary/wip-archs-hammer

archs: add files for each known architecture

buildpackages: do not try tcmalloc on arm64

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b976563af252a63b646a01d3baada4d8064ff1ff)

buildpackages: desambiguate nic on server create

When there are more than one network, it is required to select the one
to be attached to the nic of the instance.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b9b00a5ddb66cf25a133b3476ac6e7ee96dd1923)

dummy: reduce run time, run user.yaml playbook

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d4e3cec1851ae35889127b90912e133178085bc6)

buildpackages: build will work on a range of flavors

When selecting a flavor to build, we prefer the largest we can find and
fallback to smaller ones, down to the minimum acceptable flavor, if
there is no other available.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit a8f4321f5cdbe1bd61a53ebfec6e19541afcee86)

buildpackages: do not hardcode x86_64 in build scripts

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c5182f257c86321e6cb6ba2f1d50432c0b81e6f6)

buildpackages: use architecture suffixed images

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f9c4858a2693abffffc9fd3ca3e8ac40f973a024)

archs: add files for each known architecture

So that they can be prepend to teuthology-suite to run a job on a
designated architecture.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 40277ca6a9943f27aa56ce748ece53e4b69c7473)

Merge pull request #901 from dachary/wip-buildpackages-hammer

buildpackages: provision a small build machine for tests

buildpackages: provision a small build machine for tests

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1cc657798aa0fc7eeebc6de06074e7caca42b24e)

Merge pull request #897 from smithfarm/wip-checkout-sha1-hammer

workunit: git checkout sha1/branch/tag

Reviewed-by: Loic Dachary <ldachary@redhat.com>

workunit: git checkout sha1/branch/tag

git clone --branch refspec only accepts branches and tags but not a sha1
git reset --hard refspec only accepts tags and sha1 but not branches

Use git checkout because it interprets branches, tags and sha1 as
expected.

$ git checkout jewel
Branch jewel set up to track remote branch jewel from origin.
Switched to a new branch 'jewel'
$ git checkout v0.94.6
HEAD is now at e832001... 0.94.6
$ git checkout b2503b0e15c0b13f480f0835060479717b9cf935
HEAD is now at 95cefea... 0.94.3

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d1f456850c6297ca3fbb3ec0daddfddafdeff784)

Merge pull request #889 from ceph/wip-15171-hammer

suites/rados: vary min_{read,write}_recency_for_promote

Reviewed-by: Samuel Just <sjust@redhat.com>

suites/rados: vary min_{read,write}_recency_for_promote

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #884 from dachary/wip-buildpackages-hammer

buildpackages: allow for packages to be re-used for releases

buildpackages: ensure clone to a ceph directory

In case the repository from which the clone is not named ceph.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 63c1a6f9ed1a2c60b26b484b36b8f3a6593249aa)

buildpackages: for source inclusion in debian

We do not build debian versions (i.e. -2 -3 etc). Even if the debian
detection logic infers that the previous debian version already had the
source package, it is incorrect.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit df3eba0f6bc96139bcb8df2e05bfa5eea04f7610)

buildpackages: allow for more parallel builds

Builds are in parallel if the OS type / version and the SHA1 are
different. Having just one build at a time for a given SHA1 slows thing
down when there are more than a few supported OS.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 33b4397da8914205eaecaffc9b4bfb1c692309d0)

buildpackages: enable autoindex in the packages repository

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c60ace31eae65b4fba8c51b4017aa0498b4102f6)

buildpackages: do not requires the long tag version

So that v10.1.0 is used instead of 10.1.0-0-g35ac827 when possible.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b30655eeb673df447dc736e10b040b227911bbbc)

buildpackages: also pull tags from the user ceph fork

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 5e145c1a2f2a650575dc7481be37f5020853a6fd)

buildpackages: create release packages

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit ddd5f7c329327a94c0761276aa9c0073a617f8f3)

Merge pull request #879 from dachary/wip-workunit-hammer

workunit: git-reset --hard does not accept a branch

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

workunit: git-reset --hard does not accept a branch

Uses git clone --branch refspec because it agrees to use either a tag or
a branch.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 96ef6ab6991ae7f6ffff7a2a0d92bcd2de88f497)

Merge pull request #856 from SUSE/wip-opensuse-distros-hammer

distros: add yaml fragments for openSUSE 13.2 and 42.1

Reviewed-by: Loic Dachary <ldachary@redhat.com>

distros: add yaml fragments for openSUSE 13.2 and 42.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 3458adb5d3b749453848fc5edbaeedaeff84853c)

Merge pull request #834 from ceph/wip-14077-hammer

add divergent priors tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #845 from dachary/wip-14931-cram-hammer

cram: pin to version 0.6 because 0.7 handles \r differently

cram: pin to version 0.6 because 0.7 handles \r differently

http://tracker.ceph.com/issues/14931 Fixes: #14931

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 70fb7493f18c45b93d6065c03403c5dd478295f3)

Merge pull request #838 from ceph/wip-14905-hammer

Added v0.94.6 to stable upgrade suite

Added v0.94.6 to stable upgrade suite

Fixed #14905

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rados/singleton/all/pg-removal-interruption: ensure osd.0 is marked down

We were waiting for teh dameon to fail, not for the mons to notice,
which meant that after we restart the daemon the next tell could
try to talk to the old osd.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 07232b1e2bed5911453f30397a5a9de6c9d3125d)

Reduced runtime for radosbench to 600

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit a72bba29103cd287131eb7b56953c417649fed5d)

divergent_priors2: give divergent time to come up

Fixes: #13322
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit c0b0ec2831f410d89b695a1d08122334d8ee907a)

tasks/divergent_priors2: wait for osd to start before tell

Otherwise we can get

2015-09-24T19:22:15.191 INFO:teuthology.orchestra.run.mira080.stderr:Error ENXIO: problem getting command descriptions from osd.1

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ecc504fea0f1c67b768c52d95a41163e25bd065a)

tasks/divergent_prior[2]: fix rados get syntax

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6064443ad1e0dd87801a4508025906d5e26c02c5)

tasks/reg11184.py: fix rados get syntax

no -o

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7b3ea31efeb0da81b3a5ed486cebce72edbf2fec)

thrasher: Can't test ceph-objectstore-tool if nodes turned off (powercycle)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b255db820ffd1d3e1253a9a60c7192696c226490)

reg11184: Add regression test for 11184

Create divergent priors and a split and then move a pg using
ceph-objectstore-tool export/import
Add yaml file to run the reg11184 task

Fixes: #11343
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 38160ecfaaca114dfe2d0e605a4368ef92ec30f6)

divergent_priors2: Do divergent priors and ceph-objectstore-tool export/import

Based on tasks/divergent_priors.py but also do simple export/remove/import on
same osd.
Add yaml file to run the divergent_priors2 task

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b10558021fb3ce14627fedcae992453e4a43d458)

divergent_priors: Fix divergent_priors task

Flake8 fixes
Use new set_recovery_delay admin socket command
Fix bad value set for filestore_blackhole
Make sure log trims and only require 100 objects
Use kick_recovery_wq to properly set osd_recovery_delay_start to 0
Write and remove divergent and verify removal was undone
Fix to make compatible with wip-10809-11135-10290
Make sure to set_recovery_delay in a non-racey way (while osd running but down)
Leave divergent "in" so its PGs aren't treated as strays
Add yaml file to run the divergent_priors task

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a6a4aaf3d3cc6677423ee6d2181254ff52e8b76a)

Thrasher: Fix log message

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ca04dc65c197f54fd6f71806059c4323cc8ee86c)

Merge pull request #825 from ceph/wip-14652-hammer

Fixes #14652 (moved rbd and rgw tests on separate clients, forced s3tests branch)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #14652

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

suites/hammer/older: do not upgrade client while running RBD import/export

Between the Giant and Hammer releases, the symbols exported from librados/librbd
were cleaned up.  This creates a short window of time between installing
upgraded packages where symbol lookup errors could occur.

Fixes: #12563
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit cb7eb2ad6f6c56ac1801c1f032faaf40d42cc955)

Merge pull request #816 from ceph/wip-14618-hammer

Whitelisted soft lockup, detected stalls on CPUs to avoid failures in ov

Whitelisted soft lockup, detected stalls on CPUs to avoid failures in ovh

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #815 from ceph/wip-14618-hammer

Whitelisted

Whitelisted
  soft lockup
  detected stalls on CPUs
to avoide failures in ovh

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #804 from ceph/wip-14388-hammer

Added v0.94.3 to the mix

Added v0.94.3 to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #808 from dachary/wip-smithi-disks-hammer

smithi disks are too fast and fill quickly

rados: 5m radosbench (not 10m)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 04e41a608bf5f271bd062718051597977678d2db)

rados: run radosbench for 10m (not 15m)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6110421bcef2d0d1d4f921822a7161a423d763ca)

rados: run radosbench for 15m instead of 30m

The smithi disks fill after about 20m.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e2f9cedd6426b90af5cf9b04d181e88104849581)

suites/rbd: disable pool validation for pool snaps test

Fixes: #14113
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 46dddfc54b8d650f111007ae812cc81de0e4456c)

Fixed symblink for os configs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 0915747895b20b856a173d58f515af6b03381406)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>