rados/singleton-nomsgr/all/11429: more reliably create orphan pgs

Make sure osds have the latest map with a pg ls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #574 from ceph/wip-13098

rados/singleton-nomsgr/all/full-tiering: test full cache tier

rados/singleton-nomsgr/all/full-tiering: test full cache tier

Verify fix for #13098

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #577 from ceph/wip-get-prop

tolerate pg_num get failure

tasks/ceph_manager: ignore failure getting pg_num

Otherwise, we may fail while racing with a workload that deletes a pool:

2015-09-23T15:01:52.855 INFO:tasks.workunit.client.1.vpm128.stdout:[ RUN      ] LibRadosTwoPoolsPP.PromoteSnapTrimRace
2015-09-23T15:01:53.892 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw pg_num'
2015-09-23T15:01:54.206 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.gc pg_num'
2015-09-23T15:01:54.462 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users.uid pg_num'
2015-09-23T15:01:54.696 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users.email pg_num'
2015-09-23T15:01:55.006 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users pg_num'
2015-09-23T15:01:55.296 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.buckets.index pg_num'
2015-09-23T15:01:55.523 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .log pg_num'
2015-09-23T15:01:55.752 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .usage pg_num'
2015-09-23T15:01:56.188 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.buckets.extra pg_num'
2015-09-23T15:01:56.625 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get test-rados-api-vpm128-17360-6 pg_num'
2015-09-23T15:01:56.928 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get test-rados-api-vpm128-17360-13 pg_num'
2015-09-23T15:01:57.193 INFO:teuthology.orchestra.run.vpm176.stderr:Error ENOENT: unrecognized pool 'test-rados-api-vpm128-17360-13'
2015-09-23T15:01:57.206 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #576 from ceph/wip-13031-packages

buildpackages: make rpm packages

Reviewed-by: Joe Julian <jjulian@io.com>

buildpackages: make rpm packages

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #573 from ceph/wip-hammer-x

hammer-x fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

upgrade/hammer-x/stress-split-erasure-code-x86_64: specify arch

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #571 from dachary/wip-isa-arch

erasure-code: isolate isa plugin arch constraint

upgrade/hammer-x: use supported distros, not hard-coded rhel7+trusty

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: use rgw civetweb

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/parallel: clients are hammer to for final workload

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/point-to-point: fix x client

- install hammer for hammer->x upgrade
- use hammer branch for workunit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #453 from ceph/wip-11835

tasks/cephfs: time out on ceph-fuses that don't die

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph: wait for MDS to be active when creating a cluster

This is the correct implementation of 685d76a77ca16ca601a99148ef507cfde1fb3593,
merged while broken in ff1655cb57793f18698919f5284b1f207893ff8f and
reverted in 4cccde634fb10b58e4eb050286b945ed218e6f62.

Signed-off-by: John Spray <john.spray@redhat.com>

upgrade/hammer-x/point-to-point: upgrade client for final test

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/point-to-point: use proper hammer test.sh

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/parallel: upgrade client before final workload test

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: fix full_sequential indentation

Broken by caa92358572c2a226976adc6488542b409d1b484

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: use eastern tz for all upgrades

Hammer doesn't handle mixed timezones and hit_sets (cache pools)
correctly.  Avoid it.

Signed-off-by: Sage Weil <sage@redhat.com>

timezone: add eastern, pacific, random timezone snippets

These need to be early tasks, probably. Random will pick either
eastern or pacific tz randomly.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer: do ubuntu for point to point

centos packages don't (git)build for these old
releases

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: only ubuntu

the centos packages don't (git)build

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #572 from dachary/wip-duplicate-centos7

do not duplicate CentOS 7 runs

do not duplicate CentOS 7 runs

The a-supported-distro.yaml file must be outside of the supported distro
directory otherwise it will have all suites including the distro
directory run CentOS 7 jobs twice.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: isolate isa plugin arch constraint

Link the distro directory to the directory containing all supported
distros. Add the x86_64 arch contraint required by the isa plugin to an
isolated file that is combined with all jobs.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #566 from ceph/wip-s3-branch-fix

Use old branches as is without 'ceph-' for firefly, hammer etc

upgrade/hammer-x: run rados.py to completion before moving on

Leave the final workloads alone.. those can run in parallel
with other stuff.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "Merge pull request #567 from ceph/ceph_fuse-timeout"

This reverts commit ff1655cb57793f18698919f5284b1f207893ff8f, reversing
changes made to 2b25080d4faf0ae9ddb0000808830cd4d5c7bea1.

Since we haven't actually started the MDS daemons yet, this code is broken.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #569 from dachary/wip-13031-packages

buildpackages: fix flake / pep8

tasks/cephfs: switch unmount timeout to 15 minutes

Our ffsb and fsync tests contain so many small writes at random offsets
that it can take >10 minutes to commit all of them to disk if we get
a slower OSD cluster. 15 minutes is still a plenty-fast timeout for
this stage compared to just hanging and losing the logs!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #567 from ceph/ceph_fuse-timeout

ceph: wait for CephFS to be healthy before proceeding

Reviewed-by: John Spray <john.spray@redhat.com>

buildpackages: build the repository and the packages

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph: wait for CephFS to be healthy before proceeding

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Use old branches as is without 'ceph-' for firefly, hammer etc

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

buildpackages: re-use config from install task

So that the packages built match what the install task expects.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: pep8 compliance

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: build ceph packages and upload them

Import install from task.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: build ceph packages and upload them

Build Ceph packages from source for the required revision, os_type,
os_version and architecture and upload them to the gitbuilder
repository.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #560 from dachary/wip-13026-hammer-stop

hammer-x: test CEPH_FEATURE_HAMMER_0_94_4

Merge pull request #564 from ceph/wip-hammer-x

fixes for hammer-x

upgrade/hammer-x/split-*: ignore osdmap encode errors

This is normal during upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: hammer test.sh before and during upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

hammer-x: test CEPH_FEATURE_HAMMER_0_94_4

http://tracker.ceph.com/issues/13026 Refs: #13026

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #563 from ceph/wip-13060

fix test for 11429 (and now 13060)

suites/rados/singleton-nomsgr/all/11429.yaml: double-hop and fix

- simplify this.. lots of extra cruft we don't need
- restart twice at hammer to ensure that we can still load pgs
  post-upgrade
- do the same for the final version.

Fixes: #11429 (again, for ~infernalis)
Fixes: #13060
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #562 from ceph/wip-radosbench-cleanup

tasks/radosbench: allow cleanup: false

tasks/radosbench: allow cleanup: false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #561 from ceph/wip-sudo

sudo ceph

Merge pull request #559 from ceph/wip-10328

leaks

suites: sudo ceph

Signed-off-by: Sage Weil <sage@redhat.com>

tasks: sudo ceph for cli

/var/run/ceph is 770.  This is mainly necessary for any
interaction with the daemon sockets, but it is what users do
and it may avoid log noise.

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr/all/valgrind-leaks: verify leak checking works

Fixes: #10328
Signed-off-by: Sage Weil <sage@redhat.com>

ceph: add option to expect valgind errors and fail if there are none

See #10328
Signed-off-by: Sage Weil <sage@redhat.com>

distros: alias a-supported-distro.yaml to CentOS 7

Some jobs are distribution agnostic and it is meaningless to run them on
all supported distributions. For such jobs it is convient to require
than they run on a supported distribution but we don't care which. This
can be done by including a-supported-distro.yaml in the suite sub
directory instead of linking to distros/supported.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #558 from ceph/wip-12973

rados: 'failed to encode ...' warnings are normal on upgrades

Tested-by: Kefu Chai <kchai@redhat.com>

rados: 'failed to encode ...' warnings are normal on upgrades

Fixes: #12973
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #542 from ceph/wip-12551-cephfs-truncate-sec

Adds test for ceph/pull/5442

Reviewed-by: Sage Weil <sage@redhat.com>

suites/rados/singleton-nomsgr/all/cache-fs-trunc.yaml

Verify the cache tier doesn't clobber truncate_{seq,size}

Merge pull request #557 from ceph/wip-12563

suites/hammer/older: do not upgrade client while running RBD import/export

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #550 from ceph/wip-12806

tasks/qemu: avoid adding duplicated entries to /etc/exports

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suites/hammer/older: do not upgrade client while running RBD import/export

Between the Giant and Hammer releases, the symbols exported from librados/librbd
were cleaned up.  This creates a short window of time between installing
upgraded packages where symbol lookup errors could occur.

Fixes: #12563
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #556 from ceph/wip-stuck-pg-dump

tasks/ceph_manager: dump pgs when recover times out

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/ceph_manager: dump pgs when recover times out

It is really hard to map a stuck recovery back to the pgs that
are stuck.  This will make it easy.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #541 from XinzeChi/wip-rebug

fix rebug rados

Reviewed-by: Loic Dachary <ldachary@redhat.com>

distros/supported: remove jessie until paramiko issue is resolved

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-ceph-deploy'

Merge pull request #552 from ceph/wip-12878

tasks/scrub_test: sudo ls ...

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks/divergent_prior[2]: fix rados get syntax

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/reg11184.py: fix rados get syntax

no -o

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #553 from ceph/wip-s3tests

s3tests: use ceph-$foo branches

tasks/s3tests: use ceph-$foo branches

We need to be able to merge things into s3-tests master that
break rgw.  Create ceph-foo branches (ceph-master,
ceph-infernalis, etc.) and use those instead.

Signed-off-by: Sage Weil <sage@redhat.com>

kcephfs: include debug mds 20 everywhere

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

tasks/scrub_test: sudo ls ...

/var/lib/ceph/osd/* is owned by ceph now; ubuntu user can't read
it.

Fixes: #12878
Signed-off-by: Sage Weil <sage@redhat.com>

tasks/ceph_deploy: install ceph-test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #503 from guangyy/wip-12316-n

erasure coding: LRC thrasher & fast read thrasher

Reviewed-by: David Zafman <dzafman@redhat.com>

suites/rados/singleton-nomsgr/all/11429: upgrade to hammer first

Merge pull request #551 from ceph/wip-12820

tasks/cephfs: fix mount_timeout config loading

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: fix mount_timeout config loading

Previously were defaulting to a string, which
always compared greater than elapsed, so never
timed out.

Fixes: #12820
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph_deploy: work with systemd

This is ugly, but I'm not going to rewrite this right now.

Signed-off-by: Sage Weil <sage@redhat.com>

distros: add debian 8 jessie, remove wheezy from supported list

Wheezy only has experiemntal c++11 support; it's not new enough.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-deploy: install w/ ceph-deploy, not install.py

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #546 from dachary/wip-11881-multipath

ceph-disk workunit

tasks/qemu: avoid adding duplicated entries to /etc/exports

try deleting leftover entries for previous job

Fixes: #12806
Signed-off-by: Yan, Zheng <zyan@redhat.com>

rados/basic/msgr: enable all experimental features if random/async

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "rados/basic/msgr: enable all experimental features if random/async"

This reverts commit 40db8376ab42756e7c2f45469153099183439744.

Getting errors trying to schedule with this commit.

Merge remote-tracking branch 'gh/next'

Merge pull request #548 from tchaikov/wip-enable-exp-features

rados/basic/msgr: enable all experimental features if random/async

Reviewed-by: Samuel Just <sjust@redhat.com>

rados/basic/msgr: enable all experimental features if random/async

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #547 from ceph/wip-12822

tasks/cephfs: timeout on wait in test_network_death

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: timeout on wait in test_network_death

So that we can get a failure instead of a hang in

Signed-off-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #545 from ceph/wip-12777-next

tasks/cephfs: fix test_pool_perm teardown

implement the ceph-disk suite

Run the ceph-disk workunit with a mon setup with ceph-deploy. It is
prefered to the install done by the ceph tasks because it relies on the
init system and udev rules which are going to be tested by the ceph-disk
workunit.

http://tracker.ceph.com/issues/11881 Refs: #11881

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-deploy: implement only_mon

If only_mon is true, do not attempt to deploy osds.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-deploy: implement keep_running

When keep_running is true, do not shutdown the cluster, leave it as it
is for other workunits or tasks to use. This effectively allows the
ceph-deploy task to be used as a helper to deploy clusters.

The call to build_ceph_cluster is simplified by giving it the whole
configuration dictionary instead of re-building one with selected arguments.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-deploy: ceph report on failure

When ceph-deploy fails, run ceph report to get more information about
the state of the cluster at the time of the failure.

Signed-off-by: Loic Dachary <loic@dachary.org>

Update rados_api.yaml(cherry picked from commit 5b9d41e3ea0f07171ce59f0bb1f5636733452062)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Initial check in to add firefly-hammer-x suite #12641

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 998ba8b58c5c70e8a110c3e1cf29cd24ae9692b9)