upgrade/infernalis-x: allocate OpenStack volumes for each OSD

The intention is to run with xfs which requires a separate disk for each
OSD. In addition the tests will exhaust the available root disk space
otherwise.

http://tracker.ceph.com/issues/15236 Fixes: #15236

Signed-off-by: Loic Dachary <loic@dachary.org>

suites/upgrade/hammer-x: do upgrade before staritng working

Otherwise crushtool test will fail while creating pools.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #853 from ceph/wip-cephfs-test-python

suites: invoke libcephfs python tests

Merge pull request #904 from dachary/wip-rbd-upgrade-use-jewel

upgrade: use infernalis instead of hammer for rbd tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

upgrade: use infernalis instead of hammer for rbd tests

There are only few differences at the moment but it may be confusing
later on if a bug is fixed in the infernalis workunits later on.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #903 from ceph/wip-15204

tasks: fix ceph_deploy call to Filesystem()

Merge remote-tracking branch 'gh/jewel'

Merge pull request #895 from ceph/wip-rbd-mirror-test

suites/rbd: small cleanup and add ceph_test_rbd_mirror workloads

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

upgrade/infernalis-x/point-to-point: do upgrade before test workload

Test creates pools, that can fail while crushtool is being
overwritten.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/f-h-x-offline: fix mon statup

OSDs are down, so we can't wait for healthy.

Signed-off-by: Sage Weil <sage@redhat.com>

tasks: fix ceph_deploy call to Filesystem()

Fixes: #15204
Signed-off-by: John Spray <john.spray@redhat.com>

suites/rbd: add ceph_test_rbd_mirror workloads

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

upgrade/infernalis-x/point-to-point: run rgw + s3tests in sequential

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis-x/parallel: do upgrade before workload

This avoids upgrades while pools are being created (which may run
crushtool and fail).

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/f-h-x-offline: stop mons during upgrade

Mons also can't go firefly -> hammer and form a quorum; stop them
first.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis-x/point-to-point: upgrade before workload

Upgrading the mons can make pool creation fail.  Don't do it in
parallel with the rados api test workload (which does a lot of
that).

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis-x/point-to-point: run steps in series

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis-x/point-to-point: remove stray upgrade steps

This looks like an accidental cut and paste.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis-x/parallel: do workload in series

Same fix as b483793f6774db5c12ec348da3804f9adb44b810.  This makes
the cls test run later so that it doesn't race with the package
upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #899 from dachary/wip-buildpackages

buildpackages: provision a small build machine for tests

buildpackages: provision a small build machine for tests

Signed-off-by: Loic Dachary <loic@dachary.org>

suites/rbd/maintenance: use symlinks for cluster config

This is the same 3-node cluster as most tests use, and qemu with
xfstests already has an openstack config in the qemu sub-suite.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #893 from ceph/wip-rbd-maint-ops

suite/rbd: test dynamic features and rebuild object map

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

upgrade/hammer-x/parallel: run workloads in sequence, not parallel

Signed-off-by: Sage Weil <sage@redhat.com>

suite/rbd: test dynamic features and rebuild object map

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #888 from ceph/wip-15171

suites/rados/thrash: vary min_{read,write}_recency_for_promote

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #892 from ceph/wip-rbd-qemu-journaling

suites/rbd: add qemu tests with journaling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

suites/rbd: add qemu tests with journaling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Changed run time for radosbench to 200

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 0ad1e4006b1f1fd0a06bfb13e428e16312c99173)

Merge pull request #890 from ceph/wip-fix-powercycle

Changed run time for radosbench to 200

Changed run time for radosbench to 200

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge remote-tracking branch 'gh/jewel'

upgrade/hammer-x: fix ec portion of workload

This should go in the workload sequence.

*also*, it'll be the first item, which pushes the many pool
creations in the api tests off until after packages install,
which avoids a problem with ceph-mon running crushtool while it
is being extracted.

Signed-off-by: Sage Weil <sage@redhat.com>

suites/rados/thrash: vary min_{read,write}_recency_for_promote

Hopefully this will shake out #15171

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #887 from dmick/master

tasks/ceph.py: Remove *.pid at end of run

tasks/ceph.py: Remove *.pid at end of run

http://tracker.ceph.com/issues/15162
Fixes: #15162
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #885 from ceph/wip-15139

rados/singleton-nomsgr: run singleton upgrade tests on ubuntu

Merge pull request #886 from ceph/wip-15095

upgrade/infernalis-x: do not run *Whiteout tests against post-infernalis cluster

Reviewed-by: Josh Durgin <jdurgin@redhat.com

upgrade/infernalis-x: do not to tier *Whiteout tests against post-infernalis cluster

Fixes: #15095
Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr: run singleton upgrade tests on ubuntu

Fixes: #15139
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #882 from dachary/wip-buildpackages

buildpackages: allow for packages to be re-used for releases

buildpackages: ensure clone to a ceph directory

In case the repository from which the clone is not named ceph.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: for source inclusion in debian

We do not build debian versions (i.e. -2 -3 etc). Even if the debian
detection logic infers that the previous debian version already had the
source package, it is incorrect.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: allow for more parallel builds

Builds are in parallel if the OS type / version and the SHA1 are
different. Having just one build at a time for a given SHA1 slows thing
down when there are more than a few supported OS.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: enable autoindex in the packages repository

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: do not requires the long tag version

So that v10.1.0 is used instead of 10.1.0-0-g35ac827 when possible.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: also pull tags from the user ceph fork

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: create release packages

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #880 from ceph/wip-rados-vs-mds

suites: remove unused mdss from RADOS tests

Merge pull request #881 from dachary/wip-workunit

workunit: sha1/branch/tag reset method

Reviewed-by: Nathan Cutler <ncutler@suse.com>

workunit: git checkout sha1/branch/tag

git clone --branch refspec only accepts branches and tags but not a sha1
git reset --hard refspec only accepts tags and sha1 but not branches

Use git checkout because it interprets branches, tags and sha1 as
expected.

$ git checkout jewel
Branch jewel set up to track remote branch jewel from origin.
Switched to a new branch 'jewel'
$ git checkout v0.94.6
HEAD is now at e832001... 0.94.6
$ git checkout b2503b0e15c0b13f480f0835060479717b9cf935
HEAD is now at 95cefea... 0.94.3

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #875 from ceph/wip-15124

tasks/cephfs: support old mdsmap command during setup

Merge pull request #836 from ceph/wip-fix_cdtest

fix ceph-deploy test

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #861 from ceph/wip-vstart-fixes

tasks/cephfs: vstart & fuse improvements

Merge pull request #867 from ceph/jewel-15050

tasks/cephfs: cover fancy layout in test_strays

suites: fix assumed filesystem pool name

Tests shouldn't really assume the names of
cephfs pools, but because it's hardcoded in
the yaml here I'll just update it.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: support old mdsmap command during setup

While Filesystem at large requires the new commands, for
use from the `ceph` task we must support old style commands,
as the ceph task is used to instantiate old clusters during
upgrade testing.

Fixes: #15124, #15049, #15106
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #877 from dachary/wip-workunit

workunit: git-reset --hard does not accept a branch

Reviewed by: Nathan Cutler <ncutler@suse.com>

workunit: git-reset --hard does not accept a branch

Uses git clone --branch refspec because it agrees to use either a tag or
a branch.

Signed-off-by: Loic Dachary <loic@dachary.org>

suites: remove unused mdss from RADOS tests

These daemons were apparently unused in tests that
created an MDS daemon but never use a fuse task
to access it.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #871 from ceph/wip-rbd-client-upgrade

upgrade/client-upgrade: only use supported features for RBD tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

upgrade/client-upgrade: only use supported features for RBD tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #870 from ceph/wip-rbd-mirror

rbd: add some missing workunits

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Fix review comments

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #868 from ceph/wip-fio-ver

Update fio version to 2.7

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: add some missing workunits

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #863 from ceph/wip-15049

rados/singleton-nomsgr/all/11429: no mds

Merge pull request #824 from ceph/wip-multi-filesystems

Test updates for multi-filesystem support

Update fio version to 2.7

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge branch 'master' into jewel

tasks/cephfs: cover fancy layout in test_strays

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tasks/cephfs: enable multi fs flag

Cluster won't let you create more than without
setting this flag explicitly.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: debug in wait_for_daemons

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: switch off extra daemons in TestStrays

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: further thrasher fixes

Move the thrasher-specific methods out of CephManager
into MDSThrasher and plumb them into MDSCluster.

Signed-off-by: John Spray <john.spray@redhat.com

tasks/cephfs: simplify recreate()

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: update test_journal_repair for multi fs

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add TestStandbyReplay

In addition to the multi-fs tests, more localised
standby-replay cases.

Signed-off-by: John Spray <john.spray@redhat.com>

suites: add fs/multifs

Separate because it wants a larger MDS count.  Points
to the new tests in test_failover.py.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix datascan for multiple filesystems

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: move the journaltool smoke test into python land

This is one of those things that had a convoluted .yaml file
because it predated the ability to write explicit procedural
python tests.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: rename Filesystem.reset to recreate

Name was ambiguous wrt "fs reset"

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: fix mds_thrash for multi filesystems

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: update test_runner for mds_cluster

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: generalise cephfs classes for multi-fs

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: add cephfs TestMultiFilesystems

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #865 from ceph/wip-dbg-packages-1

packages: update for all the new debug packages we have in jewel

packages: update for all the new debug packages we have in jewel

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #859 from ceph/wip-15014

Add packages.yaml

Removed branch: firefly from 4-after.yaml

Fixes #15012

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 872aed9e72d0bac1fc21152544eef3c42abbd0bb)

Merge pull request #864 from ceph/wip-15012-jewel

Removed branch: firefly from 4-after.yaml

Removed branch: firefly from 4-after.yaml

Fixes #15012

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rados/singleton-nomsgr/all/11429: no mds

The newer cephfs logic wants to use the 'status' command on the
daemon, but that doesn't exist on firefly.  Luckly we don't need
it for this test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #862 from ceph/wip-odsync

rados: Enable odsync write for pggrow thrasher

rados: Enable odsync write for pggrow thrasher

Signed-off-by: Samuel Just <sjust@redhat.com>

suites: invoke libcephfs python tests

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: vstart & fuse improvements

A couple of corner cases that would get tripped up sometimes
in a vstart_runner.py environment.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #812 from ceph/jewel-10436

suites/fs: enable all snapshost test scripts

Merge remote-tracking branch 'origin/greg-fs-testing-3-7-damage'

Merge pull request #731 from ceph/wip-scrub-repair

Wip scrub repair