ceph_manager: factorize with get_pool_dump

Replace code that duplicates get_osd_dump_json with a call to the
function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: factorize with get_osd_dump_json

Replace code that duplicates get_osd_dump_json with a call to the
function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add get_pool_dump to return the json pool dump

Get the part of ceph osd dump specific to the pool given in argument.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add get_osd_dump_json to get the full output

Converts ceph osd dump_json into a python object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

fs/basic/tasks/cfuse_workunit_misc: timeout at 6h

The multiple_rsync can take a while.

Signed-off-by: Sage Weil <sage@redhat.com>

fs/multiclient: force mpi tests onto trusty

This will ensure we have a matching version of MPI.

Signed-off-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #234 from ceph/wip-dzaddscrub

Add scrub_test and repair_test to rados basic suite

Reviewed-by: Loic Dachary <ldachary@redhat.com>

smoke/basic/tasks/libcephfs_interface_tests.yaml: debug client

Hopefully we can catch #9864

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

tasks/radosbench: no log to stderr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #221 from ceph/wip-forward-scrub

Wip forward scrub

Reviewed-by: John Spray <john.spray@redhat.com>

mds_scrub_checks: Run scrub and flush commands against the MDS.

We mostly do a variety of successful ones, but we also corrupt the store
using the rados tool and make sure we get the expected error codes. Includes
a yaml fragment so the task gets run as part of the fs/basic suite.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #228 from ceph/wip-9995

tasks/cephfs/mount: use seperate for testing flock and posix lock

Reviewed-by: John Spray <john.spray@redhat.com>

repair_test: Wait for OSDs to come up before proceeding with test

Signed-off-by: David Zafman <dzafman@redhat.com>

Add scrub_test and repair_test to rados basic suite

Signed-off-by: David Zafman <david.zafman@inktank.com>

tasks/cephfs/mount: use seperate for testing flock and posix lock

Old version libfuse treats both flock and posix lock requests as posix
lock request. This is a workaround for the bug.

Fixes: #9995
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'gh/giant' into m

Conflicts:
tasks/ceph_manager.py

upgrade/dumpling-firefly-x: sync iwth giant

upgrade/firefly-x: sync with giant

Merge pull request #223 from ceph/wip-9672

Added timeout: 6h to one more workunit

Added timeout: 6h to one more workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #211 from ceph/wip-apache-worker

apache: switch to use the apache worker mpm

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-apache-worker'

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>

Smarter s3tests branch selection

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #165 from dachary/wip-erasure-code-isa

erasure-code: isa plugin thrashods suite

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #189 from ceph/wip-apache-max-requests

apache: set MaxRequestsPerChild to 0

Merge pull request #199 from ceph/wip-rgw-ec-slow-backend

rgw: mark ec related rgw suites as slow backend

Merge pull request #201 from dachary/wip-9841-no-lrc

erasure-code: enclose no-lrc in a task

erasure-code: enclose no-lrc in a task

http://tracker.ceph.com/issues/9841 Fixes: #9841

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Reviwed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

s/thrashosds/workload for now in dumpling-firefly-x/stress-split suite

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

replace thrashosd with a workload.
avoid running 2 thrashosds in the same job.

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

s/branch:dumpling/

no need for a dumpling workload here.

do not upgrade client when workload is in progress
do not run 2 rgw tasks in a row from the same client

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

Merge pull request #216 from ceph/wip-cleanup-giant

Cleaned up confusing upgrade-sequence numbering

Cleaned up confusing upgrade-sequence numbering

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #214 from ceph/revert-213-wip-s3tests-branch

Revert "Changed s3tests to run off "branch: giant-noreason" per Yehuda's...

Revert "Changed s3tests to run off "branch: giant-noreason" per Yehuda's fix for #9169"

Merge pull request #213 from ceph/wip-s3tests-branch

Changed s3tests to run off "branch: giant-noreason" per Yehuda's fix for #9169

Changed s3tests to run off "branch: giant-noreason" per Yehuda's fix for #9169

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge branch 'giant' of github.com:ceph/ceph-qa-suite into giant

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit c3b53c326558dd3dd435c9e3fdbc07d8c15f4e68)

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 35c9cae84c7964ad96381c7455c40e43dd6b4a8d)

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Don't need to explicitly turn off the test during some upgrades
Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>

tasks/thrashosds: support overrides

e.g.,

overrides:
  thrashosds:
    thrash_primary_affinity: false
...
tasks:
- install:
- ceph:
- thrashosds:
- workunit:
...

Needed for #9865

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>

Smarter s3tests branch selection

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
(cherry picked from commit 1b8d31986a5a8bc4cd9fea90eb680db3ca1866c8)

Merge pull request #206 from ceph/wip-9672

Added  timeout: 6h to workunit

Added  timeout: 6h to workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #205 from dachary/wip-erasure-code-isa-giant

erasure-code: isa plugin thrashods suite

Reviewed-by: Samuel Just <sam.just@inktank.com>

erasure-code: isa plugin thrashods suite

Add the distros directory which enumerates the os_type and
os_version on which the suite is to run.

http://tracker.ceph.com/issues/9660 Fixes: #9660

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 16e564ec8dea1649888847102ea39fa082e4d3f2)

Merge pull request #204 from dachary/wip-spurious-erasure-code-workload

erasure-code: dumpling-firefly-x/parallel/6-final-workload is incorrect

erasure-code: dumpling-firefly-x/parallel/6-final-workload is incorrect

6-final-workload/ec-rados-plugin=jerasure-k=2-m=1.yaml was added to the
wrong directory.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #202 from dachary/wip-9841-no-lrc-giant

erasure-code: enclose no-lrc in a task (giant)

erasure-code: enclose no-lrc in a task

http://tracker.ceph.com/issues/9841 Fixes: #9841

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 98ef86a202a26682f74c6d7b7aef4616fcc4b128)

fixed dumpling-firefly-x suite to upgrade to firefly branch during upgrade-sequence0

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Reviwed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 85221bc3258b27f0a6b86a74aa1734dac16afe1d)

Merge pull request #200 from dachary/wip-erasure-code-giant

backport erasure code fixes to giant

upgrade/dumpling-x: do not do ec test on mixed cluster

dumpling doens't support it

Signed-off-by: Sage Weil <sage@redhat.com>

erasure-code: upgrade/firefly-x/stress-split-erasure-code

Mimic upgrade/firefly-x/stress-split only with thrashosd configured with
min_in: 4 for k=3,m=1 . All erasure code workloads are moved in this
dedicated suite.

http://tracker.ceph.com/issues/9652 Fixes: #9652

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Conflicts:
suites/upgrade/firefly-x/stress-split-erasure-code/3-thrash/default.yaml

erasure-code: do not include k=3,m=1 in dumpling-x

Because thrashosd does not have min_in: 4 and that we don't want to set
it because other workloads may be influenced.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add thrash-erasure-code to suites/rados

Because it needs the thrasher to have min_in: 4 otherewise k=3,m=1 will
block.

http://tracker.ceph.com/issues/9648 Fixes: #9648

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: unfound test needs a non empty file

Other rados put will fail as follows

$ touch /tmp/bar
$ ./rados -p rbd put existing_3 /tmp/bar
$ ./rados -p rbd put existing_3 /tmp/bar
WARNING: could not create object: existing_3
error putting rbd/existing_3: (17) File exists

it should be considered a bug in the rados command line but needs to be
addressed separately.

http://tracker.ceph.com/issues/9387 Fixes: #9387

Signed-off-by: Loic Dachary loic-201408@dachary.org

erasure-code: fix bugous ec-rados-default.yaml in some suites

The ec-rados-default.yaml was linked to suites as if it was a task
although it is a workload intended for parallel upgrade tests.

The ec-rados-plugin=jerasure-k=2-m=1.yaml task is defined and used
instead, where relevant.

http://tracker.ceph.com/issues/9549 Fixes: #9549

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Conflicts:
suites/upgrade/dumpling-firefly-giant/parallel/6-giant-workload/ec-rados-plugin=jerasure-k=3-m=1.yaml
suites/upgrade/dumpling-firefly-x/parallel/7-final-workload/ec-rados-plugin=jerasure-k=3-m=1.yaml
suites/upgrade/firefly-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml

erasure-code: CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2 integration tests

http://tracker.ceph.com/issues/9343 Refs: #9343

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: assert that firefly-x/stress-split cannot load lrc

Because it would crash existing plugins.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

fixed firefly-x suite to not upgrade mon.c twice and optimized the suite as well.
included another upgrade combination to dumpling-firefly-x suite

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

optimized the suite and added tests to upgrade client first

Signed-off-by: tamil <tmuthami@redhat.com>

optimized the suite and added ceph_objectstore_tool: false to thrashosds wherever applicable

Signed-off-by: tamil <tmuthami@redhat.com>

Merge pull request #196 from ceph/wip-9805

Added ceph_objectstore_tool: false, fixes #9805

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Added ceph_objectstore_tool: false, fixes #9805

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

removed upgrading clients in parallel to workloads
and also do not run rgw workloads on mixed osds.

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

Merge pull request #195 from ceph/wip-9729_master

Wip 9729 master

Added print

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added client.0 upgrade to firefly step

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Fixes #9729, running incompatible tests while the cluster is mixed between firefly and dumpling
Changed to rados/test-upgrade-firefly.sh version

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #181 from ceph/wip-client-flock

tasks/mds_client_recovery: file lock test

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #192 from ceph/wip-9729

Wip 9729

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Added print

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added client.0 upgrade to firefly step

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Fixes #9729, running incompatible tests while the cluster is mixed between firefly and dumpling
Changed to rados/test-upgrade-firefly.sh version

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

apache: set MaxRequestsPerChild to 0

Otherwise the default is 10k.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

tasks/s3tests: add slow backend configurable

Adding this so that we can modify the clients' conf file as needed with slow backend.
This can be achieved by:

overrides:
  s3tests:
    slow_backend: true

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 61409179dfdd6969c624dd741813291332644003)

Merge pull request #184 from ceph/wip-s3tests-backend

tasks/s3tests: add slow backend configurable

Merge pull request #188 from ceph/wip-multi-version

multiversion/dumpling-x: remove rgw

multiversion/dumpling-x: remove rgw

firefly radosgw + dumpling osds won't work.  (Nor will anything newer
than firefly).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #187 from ceph/wip-multi-version

multi-version: do not run new ceph_test_rados against dumpling

multi-version: do not run new ceph_test_rados against dumpling

Dumpling does not properly order reads (w/ RWORDERED) vs writes.  The code
controlling that was totally rewritten in firefly.  It's not worth backporting.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #177 from ceph/wip_9691

Wip 9691 - (stable giant releases upgrades suite)

Bring in file to fix missng symlink.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Removed secod ceph-fuse

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge branch 'giant' of https://github.com/ceph/ceph-qa-suite into wip_9691

Merge pull request #183 from ceph/update_upgrade_suits2

Update upgrade suits2