tasks/ceph: conservative PG count in FS pools

This was tripping over the recent commit 42c85e80
in Ceph master, which tightens the limits on
acceptable PG counts per OSD, and was making
teuthology runs fail due to never going clean.

Rather than put in a new hardcoded count, infer
it from config.  Move some code around so that
the ceph task can get at a Filesystem object
to use in FS setup (this already has conf-getting
methods).

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #249 from ceph/wip-9557

tasks: add mds_auto_repair

Reviewed-by: John Spray <john.spray@redhat.com>

tasks: add mds_auto_repair

New CephFS tests for MDS's auto repair functions. (So far the only
test case is verify/repair backtrace on fetch dirfrag)

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Update vps.yaml

Fixed syntax

Fix for #10178

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #244 from dachary/wip-objectstore-tool

add erasure coded pool tests to objectstore tool

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: minor fixes

Remove development comment
nullfd isn't used anymore
s/log.errors/log.error/

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: fix flake8 issues

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: add assert so that teuthology knows when this test fails

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: skip unused osds

When the number of objects is low, some OSDs may not be used at all.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: run tests on erasure coded pools

Create an erasure coded pool and run tests on it. The list of PGs is
adapted to contain the shard id.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: only run get/set bytes for replicated pools

It needs to be adapted to run with erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: use CephManager::get_pool_dump

Instead of custom made get_pool_id function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: add pgnum option

Instead of hardcoding 12 use a configuration option that defaults to
12. It is handy during development to lower the number to 4 and speed up
the test cycle.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: encapsulate into a function

Move code out of the task into function. Also remove the "REP" specifics
from helper functions that could also be used for erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: add hinfo_key tests for erasure coded objects

When an hinfo_key attribute is found, assume an erasure coded object and
verify set-attr/get-attr works as expected by removing its content and
restoring it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: keep all json object representation

For erasure coded pools to be tested, the json object representation
must be preserved for all PG because they are all different. The
internal representation is changed from

   db[name]["pgid"] = pg
   db[name]["json"] = objjson

to a per pg map:

   db[name].setdefault("pg2json", {})[pg] = objjson

and the rest of the code is modified to adapt accordingly.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: tests only needs 1 machine

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add pool type constants

The osd dump command displays pool types using numerics instead of
symbolic names. Create constants in the CephManager class to use instead
of numbers.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: fix typo in get_pgid docstring

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: factorize with get_pool_dump

Replace code that duplicates get_osd_dump_json with a call to the
function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: factorize with get_osd_dump_json

Replace code that duplicates get_osd_dump_json with a call to the
function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #240 from ceph/wip-10057

ms die on skipped message = false for all krbd, kclient tasks

ceph_manager: add get_pool_dump to return the json pool dump

Get the part of ceph osd dump specific to the pool given in argument.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add get_osd_dump_json to get the full output

Converts ceph osd dump_json into a python object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #197 from ceph/wip-calateuth-wusui

Wip calateuth wusui

Further changes for calamari_setup.py

Made suggestions from code reviews.
Added no_epel option.
Merged Dan Mick's changes that add the ability to get
iceballs from http URL.
Remove duplicate assignment and added some log.debugs

Signed-off-by: Warren Usui <warren.usui@inktank.com>

Add calamari_setup

Calamari_setup can be used to set up a calamari gui for manual testing,
or be run in a suite to test the calamari setup and calamari ceph
installation code.

Fixes: 9759
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #246 from ceph/revert-245-wip-next-yuriw

Revert "Copied giant-x suite to next branch"

Revert "Copied giant-x suite to next branch"

Merge pull request #245 from ceph/wip-next-yuriw

Copied giant-x suite to next branch

Copied giant-x suite to next branch

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

ms die on skipped message = false for all krbd, kclient tasks

Signed-off-by: Sage Weil <sage@redhat.com>

fs/basic/tasks/cfuse_workunit_misc: timeout at 6h

The multiple_rsync can take a while.

Signed-off-by: Sage Weil <sage@redhat.com>

fs/multiclient: force mpi tests onto trusty

This will ensure we have a matching version of MPI.

Signed-off-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #234 from ceph/wip-dzaddscrub

Add scrub_test and repair_test to rados basic suite

Reviewed-by: Loic Dachary <ldachary@redhat.com>

smoke/basic/tasks/libcephfs_interface_tests.yaml: debug client

Hopefully we can catch #9864

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

tasks/radosbench: no log to stderr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #221 from ceph/wip-forward-scrub

Wip forward scrub

Reviewed-by: John Spray <john.spray@redhat.com>

mds_scrub_checks: Run scrub and flush commands against the MDS.

We mostly do a variety of successful ones, but we also corrupt the store
using the rados tool and make sure we get the expected error codes. Includes
a yaml fragment so the task gets run as part of the fs/basic suite.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #228 from ceph/wip-9995

tasks/cephfs/mount: use seperate for testing flock and posix lock

Reviewed-by: John Spray <john.spray@redhat.com>

repair_test: Wait for OSDs to come up before proceeding with test

Signed-off-by: David Zafman <dzafman@redhat.com>

Add scrub_test and repair_test to rados basic suite

Signed-off-by: David Zafman <david.zafman@inktank.com>

tasks/cephfs/mount: use seperate for testing flock and posix lock

Old version libfuse treats both flock and posix lock requests as posix
lock request. This is a workaround for the bug.

Fixes: #9995
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'gh/giant' into m

Conflicts:
tasks/ceph_manager.py

upgrade/dumpling-firefly-x: sync iwth giant

upgrade/firefly-x: sync with giant

Merge pull request #223 from ceph/wip-9672

Added timeout: 6h to one more workunit

Added timeout: 6h to one more workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #211 from ceph/wip-apache-worker

apache: switch to use the apache worker mpm

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-apache-worker'

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>

Smarter s3tests branch selection

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #165 from dachary/wip-erasure-code-isa

erasure-code: isa plugin thrashods suite

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #189 from ceph/wip-apache-max-requests

apache: set MaxRequestsPerChild to 0

Merge pull request #199 from ceph/wip-rgw-ec-slow-backend

rgw: mark ec related rgw suites as slow backend

Merge pull request #201 from dachary/wip-9841-no-lrc

erasure-code: enclose no-lrc in a task

erasure-code: enclose no-lrc in a task

http://tracker.ceph.com/issues/9841 Fixes: #9841

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Reviwed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

s/thrashosds/workload for now in dumpling-firefly-x/stress-split suite

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

replace thrashosd with a workload.
avoid running 2 thrashosds in the same job.

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

s/branch:dumpling/

no need for a dumpling workload here.

do not upgrade client when workload is in progress
do not run 2 rgw tasks in a row from the same client

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

Merge pull request #216 from ceph/wip-cleanup-giant

Cleaned up confusing upgrade-sequence numbering

Cleaned up confusing upgrade-sequence numbering

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #214 from ceph/revert-213-wip-s3tests-branch

Revert "Changed s3tests to run off "branch: giant-noreason" per Yehuda's...

Revert "Changed s3tests to run off "branch: giant-noreason" per Yehuda's fix for #9169"

Merge pull request #213 from ceph/wip-s3tests-branch

Changed s3tests to run off "branch: giant-noreason" per Yehuda's fix for #9169

Changed s3tests to run off "branch: giant-noreason" per Yehuda's fix for #9169

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge branch 'giant' of github.com:ceph/ceph-qa-suite into giant

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit c3b53c326558dd3dd435c9e3fdbc07d8c15f4e68)

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 35c9cae84c7964ad96381c7455c40e43dd6b4a8d)

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Don't need to explicitly turn off the test during some upgrades
Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>

tasks/thrashosds: support overrides

e.g.,

overrides:
  thrashosds:
    thrash_primary_affinity: false
...
tasks:
- install:
- ceph:
- thrashosds:
- workunit:
...

Needed for #9865

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>

Smarter s3tests branch selection

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
(cherry picked from commit 1b8d31986a5a8bc4cd9fea90eb680db3ca1866c8)

Merge pull request #206 from ceph/wip-9672

Added  timeout: 6h to workunit

Added  timeout: 6h to workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #205 from dachary/wip-erasure-code-isa-giant

erasure-code: isa plugin thrashods suite

Reviewed-by: Samuel Just <sam.just@inktank.com>

erasure-code: isa plugin thrashods suite

Add the distros directory which enumerates the os_type and
os_version on which the suite is to run.

http://tracker.ceph.com/issues/9660 Fixes: #9660

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 16e564ec8dea1649888847102ea39fa082e4d3f2)

Merge pull request #204 from dachary/wip-spurious-erasure-code-workload

erasure-code: dumpling-firefly-x/parallel/6-final-workload is incorrect

erasure-code: dumpling-firefly-x/parallel/6-final-workload is incorrect

6-final-workload/ec-rados-plugin=jerasure-k=2-m=1.yaml was added to the
wrong directory.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #202 from dachary/wip-9841-no-lrc-giant

erasure-code: enclose no-lrc in a task (giant)

erasure-code: enclose no-lrc in a task

http://tracker.ceph.com/issues/9841 Fixes: #9841

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 98ef86a202a26682f74c6d7b7aef4616fcc4b128)

fixed dumpling-firefly-x suite to upgrade to firefly branch during upgrade-sequence0

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Reviwed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 85221bc3258b27f0a6b86a74aa1734dac16afe1d)

Merge pull request #200 from dachary/wip-erasure-code-giant

backport erasure code fixes to giant

upgrade/dumpling-x: do not do ec test on mixed cluster

dumpling doens't support it

Signed-off-by: Sage Weil <sage@redhat.com>

erasure-code: upgrade/firefly-x/stress-split-erasure-code

Mimic upgrade/firefly-x/stress-split only with thrashosd configured with
min_in: 4 for k=3,m=1 . All erasure code workloads are moved in this
dedicated suite.

http://tracker.ceph.com/issues/9652 Fixes: #9652

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Conflicts:
suites/upgrade/firefly-x/stress-split-erasure-code/3-thrash/default.yaml

erasure-code: do not include k=3,m=1 in dumpling-x

Because thrashosd does not have min_in: 4 and that we don't want to set
it because other workloads may be influenced.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add thrash-erasure-code to suites/rados

Because it needs the thrasher to have min_in: 4 otherewise k=3,m=1 will
block.

http://tracker.ceph.com/issues/9648 Fixes: #9648

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: unfound test needs a non empty file

Other rados put will fail as follows

$ touch /tmp/bar
$ ./rados -p rbd put existing_3 /tmp/bar
$ ./rados -p rbd put existing_3 /tmp/bar
WARNING: could not create object: existing_3
error putting rbd/existing_3: (17) File exists

it should be considered a bug in the rados command line but needs to be
addressed separately.

http://tracker.ceph.com/issues/9387 Fixes: #9387

Signed-off-by: Loic Dachary loic-201408@dachary.org

erasure-code: fix bugous ec-rados-default.yaml in some suites

The ec-rados-default.yaml was linked to suites as if it was a task
although it is a workload intended for parallel upgrade tests.

The ec-rados-plugin=jerasure-k=2-m=1.yaml task is defined and used
instead, where relevant.

http://tracker.ceph.com/issues/9549 Fixes: #9549

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Conflicts:
suites/upgrade/dumpling-firefly-giant/parallel/6-giant-workload/ec-rados-plugin=jerasure-k=3-m=1.yaml
suites/upgrade/dumpling-firefly-x/parallel/7-final-workload/ec-rados-plugin=jerasure-k=3-m=1.yaml
suites/upgrade/firefly-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml