Merge pull request #271 from dachary/wip-cot-rename

Reviewed-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: fix pep8 errors

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: --op list now prints [pg,object]

The format of the output of --op list was changed to include the PG to
which the object belong. It simplifies the loop in
ceph_objectstore_tool.py.

http://tracker.ceph.com/issues/10376 Fixes: #10376

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: Changes for rename of binary ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #257 from ceph/wip-fs-creation

Fix PG counts and refactor FS creation during setup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #266 from ceph/wip-9892-2

radosgw-admin: wait before trimming usage

radosgw-admin: wait before trimming usage

Fixes: #9892
Need to wait through the usage interval before trimming usage, otherwise we might not
remove all pending usage info.

Backport: dumpling, firefly, giant

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit dd09ecbfab8a659f3faaf879a52849caab5e8e8e)

Merge pull request #265 from ceph/wip-test-huge-tickets

krbd: add huge_tickets workunit

krbd: add huge_tickets workunit

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #260 from ceph/wip-mds-readonly

tasks: add test case for readonly MDS

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #264 from ceph/wip-wn

tasks/watch_notify_same_primary: fix test for new watch/notify

Reviewed-by: Samuel Just <sjust@redhat.com>

tasks/watch_notify_same_primary: fix test for new watch/notify

It now checks for 'notify1' and 'notify2' strings, allowing it to work
on both old and new versions of rados watch command.

Signed-off-by: Sage Weil <sage@redhat.com>

tasks: add test case for readonly MDS

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'gh/next'

Conflicts:
machine_types/vps.yaml

Merge pull request #261 from ceph/wip-10223

Fixes #10269 http://tracker.ceph.com/issues/10269

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fixes #10269 http://tracker.ceph.com/issues/10269

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

radosgw_admin: test full sync using existing tests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #259 from ceph/wip-10223

Changed workloads in 2-workload to run in parallel

Changed workloads in 2-workload to run in parallel

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #212 from ceph/wip-9117-wusui

Move output in task/s3readwrite

Update vps.yaml

See http://tracker.ceph.com/issues/10178

Update vps.yaml

tasks/ceph: move FS creation inside Filesystem

Leave the legacy handling out in cephfs_setup, move
the filesystem creation stuff into Filesystem.  I
anticipate this being the right place for it if/when
we have tests that want to do 'fs rm' 'fs new' type
cycles within themselves.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph: conservative PG count in FS pools

This was tripping over the recent commit 42c85e80
in Ceph master, which tightens the limits on
acceptable PG counts per OSD, and was making
teuthology runs fail due to never going clean.

Rather than put in a new hardcoded count, infer
it from config.  Move some code around so that
the ceph task can get at a Filesystem object
to use in FS setup (this already has conf-getting
methods).

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #249 from ceph/wip-9557

tasks: add mds_auto_repair

Reviewed-by: John Spray <john.spray@redhat.com>

tasks: add mds_auto_repair

New CephFS tests for MDS's auto repair functions. (So far the only
test case is verify/repair backtrace on fetch dirfrag)

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Move output in task/s3readwrite

The s3readwrite.py task formerly wrote too much output while excuting.
It now saves the data on the local machine in either the archive
directory or in /tmp if no archive directory is specified.
The new file contains a client name and timestamp in its name.
Once all processing has completed, that file is saved locally.

Fixes: 9117
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Update vps.yaml

Fixed syntax

Update vps.yaml

Fixed syntax

Fix fro #10178

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Fix for #10178

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #244 from dachary/wip-objectstore-tool

add erasure coded pool tests to objectstore tool

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: minor fixes

Remove development comment
nullfd isn't used anymore
s/log.errors/log.error/

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: fix flake8 issues

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: add assert so that teuthology knows when this test fails

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: skip unused osds

When the number of objects is low, some OSDs may not be used at all.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: run tests on erasure coded pools

Create an erasure coded pool and run tests on it. The list of PGs is
adapted to contain the shard id.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: only run get/set bytes for replicated pools

It needs to be adapted to run with erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: use CephManager::get_pool_dump

Instead of custom made get_pool_id function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: add pgnum option

Instead of hardcoding 12 use a configuration option that defaults to
12. It is handy during development to lower the number to 4 and speed up
the test cycle.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: encapsulate into a function

Move code out of the task into function. Also remove the "REP" specifics
from helper functions that could also be used for erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: add hinfo_key tests for erasure coded objects

When an hinfo_key attribute is found, assume an erasure coded object and
verify set-attr/get-attr works as expected by removing its content and
restoring it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: keep all json object representation

For erasure coded pools to be tested, the json object representation
must be preserved for all PG because they are all different. The
internal representation is changed from

   db[name]["pgid"] = pg
   db[name]["json"] = objjson

to a per pg map:

   db[name].setdefault("pg2json", {})[pg] = objjson

and the rest of the code is modified to adapt accordingly.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: tests only needs 1 machine

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add pool type constants

The osd dump command displays pool types using numerics instead of
symbolic names. Create constants in the CephManager class to use instead
of numbers.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: fix typo in get_pgid docstring

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: factorize with get_pool_dump

Replace code that duplicates get_osd_dump_json with a call to the
function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: factorize with get_osd_dump_json

Replace code that duplicates get_osd_dump_json with a call to the
function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #243 from ceph/wip-10057-next

ms die on skipped message = false for all krbd, kclient tasks

Merge pull request #240 from ceph/wip-10057

ms die on skipped message = false for all krbd, kclient tasks

ceph_manager: add get_pool_dump to return the json pool dump

Get the part of ceph osd dump specific to the pool given in argument.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add get_osd_dump_json to get the full output

Converts ceph osd dump_json into a python object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #197 from ceph/wip-calateuth-wusui

Wip calateuth wusui

Further changes for calamari_setup.py

Made suggestions from code reviews.
Added no_epel option.
Merged Dan Mick's changes that add the ability to get
iceballs from http URL.
Remove duplicate assignment and added some log.debugs

Signed-off-by: Warren Usui <warren.usui@inktank.com>

Add calamari_setup

Calamari_setup can be used to set up a calamari gui for manual testing,
or be run in a suite to test the calamari setup and calamari ceph
installation code.

Fixes: 9759
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #247 from ceph/wip-next-yuriw

Copied giant-x suite to next branch

Merge pull request #246 from ceph/revert-245-wip-next-yuriw

Revert "Copied giant-x suite to next branch"

Revert "Copied giant-x suite to next branch"

Merge pull request #245 from ceph/wip-next-yuriw

Copied giant-x suite to next branch

Copied giant-x suite to next branch

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

ms die on skipped message = false for all krbd, kclient tasks

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 219b72b6fdd5a020f266b3675fb149cce6cbac8e)

ms die on skipped message = false for all krbd, kclient tasks

Signed-off-by: Sage Weil <sage@redhat.com>

fs/basic/tasks/cfuse_workunit_misc: timeout at 6h

The multiple_rsync can take a while.

Signed-off-by: Sage Weil <sage@redhat.com>

fs/multiclient: force mpi tests onto trusty

This will ensure we have a matching version of MPI.

Signed-off-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #234 from ceph/wip-dzaddscrub

Add scrub_test and repair_test to rados basic suite

Reviewed-by: Loic Dachary <ldachary@redhat.com>

smoke/basic/tasks/libcephfs_interface_tests.yaml: debug client

Hopefully we can catch #9864

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

tasks/radosbench: no log to stderr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #221 from ceph/wip-forward-scrub

Wip forward scrub

Reviewed-by: John Spray <john.spray@redhat.com>

mds_scrub_checks: Run scrub and flush commands against the MDS.

We mostly do a variety of successful ones, but we also corrupt the store
using the rados tool and make sure we get the expected error codes. Includes
a yaml fragment so the task gets run as part of the fs/basic suite.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #228 from ceph/wip-9995

tasks/cephfs/mount: use seperate for testing flock and posix lock

Reviewed-by: John Spray <john.spray@redhat.com>

repair_test: Wait for OSDs to come up before proceeding with test

Signed-off-by: David Zafman <dzafman@redhat.com>

Add scrub_test and repair_test to rados basic suite

Signed-off-by: David Zafman <david.zafman@inktank.com>

tasks/cephfs/mount: use seperate for testing flock and posix lock

Old version libfuse treats both flock and posix lock requests as posix
lock request. This is a workaround for the bug.

Fixes: #9995
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'gh/giant' into m

Conflicts:
tasks/ceph_manager.py

upgrade/dumpling-firefly-x: sync iwth giant

upgrade/firefly-x: sync with giant

Merge pull request #223 from ceph/wip-9672

Added timeout: 6h to one more workunit

Added timeout: 6h to one more workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #211 from ceph/wip-apache-worker

apache: switch to use the apache worker mpm

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-apache-worker'

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>

apache: change template to load mpm worker module

in apache 2.4

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

apache: switch to use the apache worker mpm

Fixes: #9169
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>

Smarter s3tests branch selection

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #165 from dachary/wip-erasure-code-isa

erasure-code: isa plugin thrashods suite

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #189 from ceph/wip-apache-max-requests

apache: set MaxRequestsPerChild to 0

Merge pull request #199 from ceph/wip-rgw-ec-slow-backend

rgw: mark ec related rgw suites as slow backend

Merge pull request #201 from dachary/wip-9841-no-lrc

erasure-code: enclose no-lrc in a task

erasure-code: enclose no-lrc in a task

http://tracker.ceph.com/issues/9841 Fixes: #9841

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Reviwed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

s/thrashosds/workload for now in dumpling-firefly-x/stress-split suite

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

replace thrashosd with a workload.
avoid running 2 thrashosds in the same job.

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

s/branch:dumpling/

no need for a dumpling workload here.

do not upgrade client when workload is in progress
do not run 2 rgw tasks in a row from the same client

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>

Merge pull request #216 from ceph/wip-cleanup-giant

Cleaned up confusing upgrade-sequence numbering