ceph_objectstore_tool: skip unused osds

When the number of objects is low, some OSDs may not be used at all.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 9a089ef369174a241098f2c5c28bcb29fe09eb58)

ceph_objectstore_tool: run tests on erasure coded pools

Create an erasure coded pool and run tests on it. The list of PGs is
adapted to contain the shard id.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit be39bd5d84613d30eecede78172619bb5339dbe0)

ceph_objectstore_tool: only run get/set bytes for replicated pools

It needs to be adapted to run with erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fbb33f3577d430ffcd62a4f70b1e0de3ae28af74)

ceph_objectstore_tool: use CephManager::get_pool_dump

Instead of custom made get_pool_id function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 2b57f4c59c045b042c779ffe384fdcccf9a6ddb4)

ceph_objectstore_tool: add pgnum option

Instead of hardcoding 12 use a configuration option that defaults to
12. It is handy during development to lower the number to 4 and speed up
the test cycle.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d0caf5cdffe26f27b0a25ebdb96d9b0fd37ffffc)

ceph_objectstore_tool: encapsulate into a function

Move code out of the task into function. Also remove the "REP" specifics
from helper functions that could also be used for erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit c9e6edebd95377f7dec2a34f390f250c0e1a026d)

ceph_objectstore_tool: add hinfo_key tests for erasure coded objects

When an hinfo_key attribute is found, assume an erasure coded object and
verify set-attr/get-attr works as expected by removing its content and
restoring it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d5912263731e79877a0ea459d3b7453a925402e3)

ceph_objectstore_tool: keep all json object representation

For erasure coded pools to be tested, the json object representation
must be preserved for all PG because they are all different. The
internal representation is changed from

   db[name]["pgid"] = pg
   db[name]["json"] = objjson

to a per pg map:

   db[name].setdefault("pg2json", {})[pg] = objjson

and the rest of the code is modified to adapt accordingly.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 9ef8887a3b219021296f18eb5b317e3134f1281e)

ceph_manager: add pool type constants

The osd dump command displays pool types using numerics instead of
symbolic names. Create constants in the CephManager class to use instead
of numbers.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b925be13bc1183b2a8f68fd59d7fcb59330d1404)

ceph_manager: fix typo in get_pgid docstring

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fd1ec39d3835e95cbe0a6853a63689e217d6567e)

ceph_manager: add get_pool_dump to return the json pool dump

Get the part of ceph osd dump specific to the pool given in argument.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 5fc42b98ab22a9b2edad4336677270b11a42d776)

ceph_manager: add get_osd_dump_json to get the full output

Converts ceph osd dump_json into a python object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f353c150a8234644a25d7eeb073f4d3dd4af7c07)

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Don't need to explicitly turn off the test during some upgrades
Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a295c18a808033dfbdacf8801c26fed137ccab35)

Conflicts:
suites/upgrade/dumpling-firefly-x/stress-split/13-thrash/default.yaml (not in firefly)
suites/upgrade/dumpling/fs/4-final/osdthrash.yaml
suites/upgrade/dumpling/rbd/4-final/osdthrash.yaml
suites/upgrade/dumpling/rgw/4-final/osdthrash.yaml

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bdbcf760d9060b3b8b940a52ce2fcd8406c8b95a)

document 'command' requirements on admin_socket method

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 4db95170e605b3c602a8e4749278943860623c9d)

tasks/ceph_manager: enable log for ceph_objectstore_tool

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7ba50e0c89eead17e7fc37232f8c9a91855ebd8c)

Conflicts:
tasks/ceph_manager.py

included an option to ceph_objectstore_tool, whenever we have keyvaluestore_backend as a configurable parameter

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>
(cherry picked from commit a5a1cce3c768df18436fec795b19f5b4e7899cac)

ceph_manager: Add test code to use export/import to move a pg

Check for more than 1 osd down and randomize on chance_move_pg (100%)
For now only export from older down osd to newly down osd to avoid missing map

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 05eee9fa79ef4cb432cb28e3a2eb0e66fd686a7b)

Conflicts:
tasks/ceph_manager.py

ceph_manager: Implement export/import when thrasher kills an osd

Use list-pgs to avoid races by seeing actual pgs present

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 0cdf6e813db6bddd4dd2b97b8722ed32a4cf56a8)

ceph_objectstore_tool: Add task for testing of tool of the same name

Based on ceph/src/test/ceph_objectstore_tool.py but only does
replicated pool testing and doesn't test argument validation.

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 9ade22dd3458f15eec03ad42b599beef10b0bf20)

move functions from ceph to ceph_manager

mount_osd_data and make_admin_daemon_dir are only used by
ceph_manager.py although they are defined in ceph.py

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f53ea258a4c86603f0982448b191488abb4e73f3)

Conflicts:
tasks/ceph.py

workunit: include /usr/sbin in the PATH for all commands

Fixes: #10600
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0e53f5f38b0cb06a6f4989e622a279c8ba9104eb)

Merge pull request #355 from dachary/wip-cram-firefly

all tests must end with .t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pull fsstress.sh tests from the firelfy branch

Signed-off-by: Loic Dachary <loic@dachary.org>

all tests must end with .t

The hb=firelfy cannot end the URL because of this constraint.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: fix typo in url

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #351 from dachary/wip-cram-firefly

pull cram tests from the firelfy branch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #352 from dachary/wip-qemu-firefly

Use an NFS mount instead of virtio-9p in the qemu task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Use an NFS mount instead of virtio-9p in the qemu task

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 93f2bea3530a523c7b2ac0575033697cea893f2d)

pull cram tests from the firelfy branch

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #348 from ceph/wip-10954-firefly

linter fixes for firefly

fix path for tox.ini to work on ceph-qa-suite only

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove unused import from ceph task

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove unused variable assignment

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9df27f7a1040dae69ef65122d13d1640cb1b6a66)

initial tox.ini to run flake8

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7108b699fda247ce12cbeaea9dd0a32843775c27)

ignore tox hidden dir

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4068f73ade68e36c1e7653733eacf2b1288cd753)

fs: move into the ceph-fuse mount before doing a test

Fixes: #10465
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8a5f76d10bc8bf0358f5215ac2aadfb210691507)

Add labels to a couple common task failures

The labels will help us quickly determine what the command was trying to
accomplish in log output.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a965d6ca142c12c74124f73c9ef874df85e07e3d)

Conflicts:
tasks/workunit.py

extra_packages: stop trying to install libcephfs stuff

These package names all differ across distros, and they're included in
teuthology's install task anyway.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 977a221f9226cf479c9d2c1fbdd1d8df13bbce44)

tiobench: remove more of it which was somehow skipped

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

do not try and install libcephfs-java via extra_packages

It's been added to the default install task, because the
package name varies across distros.

Fixes: #10510
Signed-off-by: Greg Farnum <greg@gregs42.com>
(cherry picked from commit 30e5f3d93192a6bf2428ea4f8d793e2a087a5353)

samba: run only on ubuntu until #10627 is resolved

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 1dcff968a4fc359086723e8a3cb941051297f14a)

Use qemu-kvm in the qemu task for rpm based systems.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #299 from ceph/wip-10538

Add v0.80.8 to firefly stable suites (issue 10538)

Added "force-branch: firefly-original" to make sure we run on good s3tests branch

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Fixed typos

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added new v0.80.8 to
/newer
singleton/version
singleton/version-x
Added more logging

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #297 from ceph/wip-10525

Changed to "osd map max advance: 1000"

Added warning "osd_map_max_advance" to the whitelist to aviod fail on it

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Changed to "osd map max advance: 1000"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #295 from ceph/wip-10525

Changed to "osd map max advance: 1000"

Changed to "osd map max advance: 1000"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #294 from ceph/wip-10525

Replaced step for v0.80.6 with v0.80.7

Replaced step for v0.80.6 with v0.80.7

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #292 from ceph/wip-10525

Fixes http://tracker.ceph.com/issues/10525

Fixes http://tracker.ceph.com/issues/10525

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #289 from ceph/wip-10525

Fixes http://tracker.ceph.com/issues/10525

Fixes http://tracker.ceph.com/issues/10525

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge remote-tracking branch 'gh/wip-slow-backend-firefly' into firefly

drop tiobench tests

The tiobench software has been abandoned upstream for years. Fedora and
Debian are no longer shipping the tiobench package, so we've had to
carry the package ourselves in the Ceph project, and we're trying to
slim down our dependencies where it makes sense to do so.

Nuke the tiobench tests.

http://tracker.ceph.com/issues/10152 Refs: #10152

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 8a317125fc542667e57c66017db6e80a3edc24f2)

Conflicts:
suites/kcephfs/cephfs/tasks/kclient_workunit_suites_tiobench.yaml

tasks/s3tests: add slow backend configurable

Adding this so that we can modify the clients' conf file as needed with slow backend.
This can be achieved by:

overrides:
  s3tests:
    slow_backend: true

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 61409179dfdd6969c624dd741813291332644003)

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Reviwed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Inherit teuthology's log level

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 31ff241f5a74d455d6e61f02065f2df43d3fa978)

Merge pull request #267 from ceph/wip-firelfy-yuriw

Removed import_export tests to fix #10176

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed import_export tests to fix #10176

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

radosgw-admin: wait before trimming usage

Fixes: #9892
Need to wait through the usage interval before trimming usage, otherwise we might not
remove all pending usage info.

Backport: dumpling, firefly, giant

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit dd09ecbfab8a659f3faaf879a52849caab5e8e8e)
(cherry picked from commit 47d56d779eb145f5bca1e8e1efdeec8be92b6579)

Merge pull request #255 from ceph/revert-251-wip-10178-firefly

Revert "fix mon restart in firefly upgrade tests"

Revert "fix mon restart in firefly upgrade tests"

Merge pull request #253 from ceph/wip-firelfy-yuriw

Removed import_export tests to fix #10176

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed import_export tests to fix #10176

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #251 from ceph/wip-10178-firefly

fix mon restart in firefly upgrade tests

upgrade/firefly: do not wait for clean when restarting mons

This may fail is the quorum features to not match.

Fixes: #10178
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #242 from ceph/wip-10057-firefly

ms die on skipped message = false for all krbd, kclient tasks

ms die on skipped message = false for all krbd, kclient tasks

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 219b72b6fdd5a020f266b3675fb149cce6cbac8e)

fs/basic/tasks/cfuse_workunit_misc: timeout at 6h

The multiple_rsync can take a while.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c58af9f26b63bd1df0c43706e5eda58953454eab)

fs/multiclient: force mpi tests onto trusty

This will ensure we have a matching version of MPI.

Signed-off-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0a88b34ba350b9eb057a735fe8082ebf03762327)

Merge pull request #231 from ceph/wip-firelfy-yuriw

Reverted to have upgrade-osd-mon-mds sequence for newer tests of firefly...

Added [client.0] for ceph-fuse, fixes looping issue for firefly upgrade suite

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Reverted to have upgrade-osd-mon-mds sequence for newer tests of firefly stable suite

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #229 from ceph/wip-firelfy-yuriw

Removed sequence upgrading osd before mon as invalid, fixes http://tracker.ceph.com/issues/7679

Removed sequence upgrading osd before mon as invalid, fixes http://tracker.ceph.com/issues/7679

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #224 from ceph/wip-9710

Wip 9710

Removed second rgw task as we need only one when combibed (+) workloads s3tests abd swift

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added "+"to all workloads to reduce amount of jobs

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #220 from ceph/wip-9419

Changed load to rados/load-gen-big.sh

Changed load to rados/load-gen-big.sh

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #219 from ceph/wip-9419

Wip 9419

Removed msd from upgrade sequence, as it's not correct for dumpling-to-firefly upgrade steps and removed cls line

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Restored restart order

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Changed rados workload to test-upgrade-firefly.sh on dumplimg branch

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Testing

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #218 from ceph/wip-9419

Restart osd only added

Fixed restart

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Fixed typo

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Modified restart to have mon.a, mon.b, mon.c, osd.0, osd.1

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Restart osd only added

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #217 from ceph/wip-9419

Cleaned up tests structure

Put mds after ods in restart

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #209 from ceph/wip-9866

upgrade client node before finish

Cleaned up tests structure

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>