rbd fio tests using rbd engine, tests on clones as well using features 1 and 13.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rbd io tests using fio, various fio options can be specified in yaml

test rbd or krbd using fio, can also run io on rbd clones if option is specified in yaml
various options like image-size, rbd format/features, fio io size, readwrite options can be provided in yaml.
check the docstring for exact usage.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #586 from ceph/wip-mdscap

tasks/cephfs: mds allow

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #575 from ceph/wip-32bits-ceph-fuse

suites/fs: test ceph-fuse with 32-bits ino_t

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rados/singleton-nomsgr/all/11429: grep osd.0 only

Restart can be slow enough that osd.1 and 2 finish deleting the
pgs.  Verifying one osd sees the instance is sufficient.

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/cephfs: mds allow

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-infernalis'

Merge pull request #583 from dachary/wip-buildpackages-flock

buildpackages: flock must --close

Merge pull request #584 from dachary/wip-buildpackages-centos-6

buildpackages: user-data must be per os-type/os-version

buildpackages: user-data must be per os-type/os-version

CentOS 6.5 needs to install a package and reboot to grow the root file
system. Instead of assuming a common user-data.txt file can fit all
Operating Systems, make one user data per os-type/os-version combination.

Signed-off-by: Loic Dachary <loic@dachary.org>

add cache tiering test for hammer-x

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rados/singleton-nomsgr/all/11439: fix greps

Signed-off-by: Sage Weil <sage@redhat.com>

buildpackages: flock must --close

The process run by flock must not inherit the file descriptor because
this will cause the lock to be held forever should the command survive
the call to flock. This is for instance the case for the ssh-agent.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #581 from dachary/wip-11883-git-url

buildpackages: get Ceph git url via get_ceph_git_url()

buildpackages: get Ceph git url via get_ceph_git_url()

Instead of relying on git_base_url, use the get_ceph_git_url() to obtain
the URL of the Ceph repository to use with git clone. This allows the
user to override it via the git_ceph_url configuration option and the
--git-ceph-url command line option to teuthology-openstack.

http://tracker.ceph.com/issues/11883 Refs: #11883

Signed-off-by: Loic Dachary <loic@dachary.org>

upgrade/hammer-x/parallel: set infernalis settings after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

releases/infernalis.yaml: final steps after infernalis upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/divergent_priors2: wait for osd to start before tell

Otherwise we can get

2015-09-24T19:22:15.191 INFO:teuthology.orchestra.run.mira080.stderr:Error ENXIO: problem getting command descriptions from osd.1

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr/all/11429: more reliably create orphan pgs

Make sure osds have the latest map with a pg ls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #574 from ceph/wip-13098

rados/singleton-nomsgr/all/full-tiering: test full cache tier

rados/singleton-nomsgr/all/full-tiering: test full cache tier

Verify fix for #13098

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #577 from ceph/wip-get-prop

tolerate pg_num get failure

tasks/ceph_manager: ignore failure getting pg_num

Otherwise, we may fail while racing with a workload that deletes a pool:

2015-09-23T15:01:52.855 INFO:tasks.workunit.client.1.vpm128.stdout:[ RUN      ] LibRadosTwoPoolsPP.PromoteSnapTrimRace
2015-09-23T15:01:53.892 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw pg_num'
2015-09-23T15:01:54.206 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.gc pg_num'
2015-09-23T15:01:54.462 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users.uid pg_num'
2015-09-23T15:01:54.696 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users.email pg_num'
2015-09-23T15:01:55.006 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users pg_num'
2015-09-23T15:01:55.296 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.buckets.index pg_num'
2015-09-23T15:01:55.523 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .log pg_num'
2015-09-23T15:01:55.752 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .usage pg_num'
2015-09-23T15:01:56.188 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.buckets.extra pg_num'
2015-09-23T15:01:56.625 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get test-rados-api-vpm128-17360-6 pg_num'
2015-09-23T15:01:56.928 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get test-rados-api-vpm128-17360-13 pg_num'
2015-09-23T15:01:57.193 INFO:teuthology.orchestra.run.vpm176.stderr:Error ENOENT: unrecognized pool 'test-rados-api-vpm128-17360-13'
2015-09-23T15:01:57.206 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #576 from ceph/wip-13031-packages

buildpackages: make rpm packages

Reviewed-by: Joe Julian <jjulian@io.com>

buildpackages: make rpm packages

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #573 from ceph/wip-hammer-x

hammer-x fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

upgrade/hammer-x/stress-split-erasure-code-x86_64: specify arch

Signed-off-by: Sage Weil <sage@redhat.com>

suites/fs: test ceph-fuse with 32-bits ino_t

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #571 from dachary/wip-isa-arch

erasure-code: isolate isa plugin arch constraint

upgrade/hammer-x: use supported distros, not hard-coded rhel7+trusty

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: use rgw civetweb

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/parallel: clients are hammer to for final workload

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/point-to-point: fix x client

- install hammer for hammer->x upgrade
- use hammer branch for workunit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #453 from ceph/wip-11835

tasks/cephfs: time out on ceph-fuses that don't die

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph: wait for MDS to be active when creating a cluster

This is the correct implementation of 685d76a77ca16ca601a99148ef507cfde1fb3593,
merged while broken in ff1655cb57793f18698919f5284b1f207893ff8f and
reverted in 4cccde634fb10b58e4eb050286b945ed218e6f62.

Signed-off-by: John Spray <john.spray@redhat.com>

upgrade/hammer-x/point-to-point: upgrade client for final test

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/point-to-point: use proper hammer test.sh

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/parallel: upgrade client before final workload test

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: fix full_sequential indentation

Broken by caa92358572c2a226976adc6488542b409d1b484

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: use eastern tz for all upgrades

Hammer doesn't handle mixed timezones and hit_sets (cache pools)
correctly.  Avoid it.

Signed-off-by: Sage Weil <sage@redhat.com>

timezone: add eastern, pacific, random timezone snippets

These need to be early tasks, probably. Random will pick either
eastern or pacific tz randomly.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer: do ubuntu for point to point

centos packages don't (git)build for these old
releases

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: only ubuntu

the centos packages don't (git)build

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #572 from dachary/wip-duplicate-centos7

do not duplicate CentOS 7 runs

do not duplicate CentOS 7 runs

The a-supported-distro.yaml file must be outside of the supported distro
directory otherwise it will have all suites including the distro
directory run CentOS 7 jobs twice.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: isolate isa plugin arch constraint

Link the distro directory to the directory containing all supported
distros. Add the x86_64 arch contraint required by the isa plugin to an
isolated file that is combined with all jobs.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #566 from ceph/wip-s3-branch-fix

Use old branches as is without 'ceph-' for firefly, hammer etc

upgrade/hammer-x: run rados.py to completion before moving on

Leave the final workloads alone.. those can run in parallel
with other stuff.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "Merge pull request #567 from ceph/ceph_fuse-timeout"

This reverts commit ff1655cb57793f18698919f5284b1f207893ff8f, reversing
changes made to 2b25080d4faf0ae9ddb0000808830cd4d5c7bea1.

Since we haven't actually started the MDS daemons yet, this code is broken.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #569 from dachary/wip-13031-packages

buildpackages: fix flake / pep8

tasks/cephfs: switch unmount timeout to 15 minutes

Our ffsb and fsync tests contain so many small writes at random offsets
that it can take >10 minutes to commit all of them to disk if we get
a slower OSD cluster. 15 minutes is still a plenty-fast timeout for
this stage compared to just hanging and losing the logs!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #567 from ceph/ceph_fuse-timeout

ceph: wait for CephFS to be healthy before proceeding

Reviewed-by: John Spray <john.spray@redhat.com>

buildpackages: build the repository and the packages

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph: wait for CephFS to be healthy before proceeding

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Use old branches as is without 'ceph-' for firefly, hammer etc

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

buildpackages: re-use config from install task

So that the packages built match what the install task expects.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: pep8 compliance

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: build ceph packages and upload them

Import install from task.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: build ceph packages and upload them

Build Ceph packages from source for the required revision, os_type,
os_version and architecture and upload them to the gitbuilder
repository.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #560 from dachary/wip-13026-hammer-stop

hammer-x: test CEPH_FEATURE_HAMMER_0_94_4

Merge pull request #564 from ceph/wip-hammer-x

fixes for hammer-x

upgrade/hammer-x/split-*: ignore osdmap encode errors

This is normal during upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: hammer test.sh before and during upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

hammer-x: test CEPH_FEATURE_HAMMER_0_94_4

http://tracker.ceph.com/issues/13026 Refs: #13026

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #563 from ceph/wip-13060

fix test for 11429 (and now 13060)

suites/rados/singleton-nomsgr/all/11429.yaml: double-hop and fix

- simplify this.. lots of extra cruft we don't need
- restart twice at hammer to ensure that we can still load pgs
  post-upgrade
- do the same for the final version.

Fixes: #11429 (again, for ~infernalis)
Fixes: #13060
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #562 from ceph/wip-radosbench-cleanup

tasks/radosbench: allow cleanup: false

tasks/radosbench: allow cleanup: false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #561 from ceph/wip-sudo

sudo ceph

Merge pull request #559 from ceph/wip-10328

leaks

suites: sudo ceph

Signed-off-by: Sage Weil <sage@redhat.com>

tasks: sudo ceph for cli

/var/run/ceph is 770.  This is mainly necessary for any
interaction with the daemon sockets, but it is what users do
and it may avoid log noise.

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr/all/valgrind-leaks: verify leak checking works

Fixes: #10328
Signed-off-by: Sage Weil <sage@redhat.com>

ceph: add option to expect valgind errors and fail if there are none

See #10328
Signed-off-by: Sage Weil <sage@redhat.com>

distros: alias a-supported-distro.yaml to CentOS 7

Some jobs are distribution agnostic and it is meaningless to run them on
all supported distributions. For such jobs it is convient to require
than they run on a supported distribution but we don't care which. This
can be done by including a-supported-distro.yaml in the suite sub
directory instead of linking to distros/supported.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #558 from ceph/wip-12973

rados: 'failed to encode ...' warnings are normal on upgrades

Tested-by: Kefu Chai <kchai@redhat.com>

rados: 'failed to encode ...' warnings are normal on upgrades

Fixes: #12973
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #542 from ceph/wip-12551-cephfs-truncate-sec

Adds test for ceph/pull/5442

Reviewed-by: Sage Weil <sage@redhat.com>

suites/rados/singleton-nomsgr/all/cache-fs-trunc.yaml

Verify the cache tier doesn't clobber truncate_{seq,size}

Merge pull request #557 from ceph/wip-12563

suites/hammer/older: do not upgrade client while running RBD import/export

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #550 from ceph/wip-12806

tasks/qemu: avoid adding duplicated entries to /etc/exports

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suites/hammer/older: do not upgrade client while running RBD import/export

Between the Giant and Hammer releases, the symbols exported from librados/librbd
were cleaned up.  This creates a short window of time between installing
upgraded packages where symbol lookup errors could occur.

Fixes: #12563
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #556 from ceph/wip-stuck-pg-dump

tasks/ceph_manager: dump pgs when recover times out

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/ceph_manager: dump pgs when recover times out

It is really hard to map a stuck recovery back to the pgs that
are stuck.  This will make it easy.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #541 from XinzeChi/wip-rebug

fix rebug rados

Reviewed-by: Loic Dachary <ldachary@redhat.com>

distros/supported: remove jessie until paramiko issue is resolved

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-ceph-deploy'

Merge pull request #552 from ceph/wip-12878

tasks/scrub_test: sudo ls ...

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks/divergent_prior[2]: fix rados get syntax

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/reg11184.py: fix rados get syntax

no -o

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #553 from ceph/wip-s3tests

s3tests: use ceph-$foo branches

tasks/s3tests: use ceph-$foo branches

We need to be able to merge things into s3-tests master that
break rgw.  Create ceph-foo branches (ceph-master,
ceph-infernalis, etc.) and use those instead.

Signed-off-by: Sage Weil <sage@redhat.com>

kcephfs: include debug mds 20 everywhere

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

tasks/scrub_test: sudo ls ...

/var/lib/ceph/osd/* is owned by ceph now; ubuntu user can't read
it.

Fixes: #12878
Signed-off-by: Sage Weil <sage@redhat.com>

tasks/ceph_deploy: install ceph-test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #503 from guangyy/wip-12316-n

erasure coding: LRC thrasher & fast read thrasher

Reviewed-by: David Zafman <dzafman@redhat.com>

suites/rados/singleton-nomsgr/all/11429: upgrade to hammer first

Merge pull request #551 from ceph/wip-12820

tasks/cephfs: fix mount_timeout config loading

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: fix mount_timeout config loading

Previously were defaulting to a string, which
always compared greater than elapsed, so never
timed out.

Fixes: #12820
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph_deploy: work with systemd

This is ugly, but I'm not going to rewrite this right now.

Signed-off-by: Sage Weil <sage@redhat.com>