squash: remove utility, moved to teuthology.git

(cherry picked from commit 73577f1077dcfb059d7a96d7502a147a943cd261)

squash: ceph_manager: add utility_task doc string

(cherry picked from commit 2a60852a1d0b0b7980004cf262d4af8ea4011667)

suites/rados: add test for 11429

This patch also adds some convenience facilities for making
some of the ceph_manager methods into tasks usable from a
yaml file.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 015ed70f8a46a44e35433d1f701e37f68af31cf9)

Conflicts:
tasks/radosbench.py
the pool creation call changed and the conflict is because
they were indented to implement the option to not create
a pool

kcephfs: remove 'kclient' workunit tests -- there are none left now

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #437 from ceph/wip-11180-firefly

ensure summary is looked for the user we need (part 2)

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 97e6d808f086834f2350084d5de36be285ad2bde)

Conflicts:
tasks/radosgw_admin.py
        the context around def successful_ops(out): changed but is
unrelated to the modification

knfs: remove the MDS debugging

It doesn't fit on the typica boxes with debugging, and we probably don't need
it for any NFS issues which do arise...

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 47fff708f936b7380446a4452a9f69f4d1c93c05)

Merge pull request #419 from ceph/wip-11139-firefly

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Merged again by David Zafman <dzafman@redhat.com> because firefly branch lost original Apr 3 merge on Apr 16.

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Fixes: #11139
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 6c5300552d00232d6ecb2c1aa641d515c9d8cd34)

Merge pull request #412 from ceph/wip-firefly-git

git.ceph.com (firefly)

Reviewed-by: Loic Dachary <ldachary@rehdat.com>

git.ceph.com (firefly)

Signed-off-by: Sage Weil <sage@redhat.com>

fs: fix up dd testing again

We need to change into the directory during the same shell session as we actually
run the dd...

Fixes: #10861
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8aeacaa59f32b15aa95a529d1040cb73cbd7efcc)

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7ea0a424a53bf68cad4cd4c7c78a07401dd6851)

Merge pull request #369 from dachary/wip-rgw-acl

create a new get_acl helper to ensure proper xml is compared

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #368 from dachary/wip-shard-missing

whitelist shard missing errors

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #361 from ceph/wip-cot-firefly

Backport ceph-objectstore-tool changes to firefly

Reviewed-by: Loic Dachary <ldachary@redhat.com>

create a new get_acl helper to ensure proper xml is compared

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b6ff638735d03c1f3304198fad51ca701a2c8bd)

Conflicts:
tasks/radosgw_admin.py
        the context changed but the patch seems otherwise good

whitelist shard missing errors

If a shard is missing during a radosbench it can safely be
ignored. It will either be recovered and radosbench will complete
successfully. Or there are too many shard missing to recover the object
and radosbench will fail. Either way we can rely on radosbench to
accurately report success or failure.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #365 from ceph/wip-11072

Added v0.80.9 to upgrade/firefly suite, fixed #11072

Added v0.80.9 to upgrade/firefly suite, fixed #11072

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2e4d884fdaabbeca67b519537a80166d3cc99fa3)

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 811c02e3922c4ae29c09bb5f7b48840d0eced951)

Formatting changes

(cherry picked from commit e5876126a9a8a474ddffaf60bdfde960e33717da)

Fix wrapped line

(cherry picked from commit 14e27650a62016da8caecbb160fbc5bf77d5515e)

ceph_manager: Fix Thrasher kill_osd() to ignore pool going away

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b239dadc02fc8e09c5bf633a06c479494929e85c)

Conflicts:
tasks/ceph_manager.py

ceph_manager: Thrasher fixes to run() calls to set check_status = False

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4eda2967a456d1280a8f081fb80bb274ac60ac0e)

Conflicts:
tasks/ceph_manager.py

ceph_manager: Re-enable Thrasher ceph-objectstore-tool testing

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit e0b0d41e4cc87c058386bbc00bba444671493838)

ceph_manager: In Thrasher randomly remove past intervals

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4a195c78c1d932ecf03055d3cfc2e5584947b657)

ceph_manager: In Thrasher disable ceph_objectstore_tool tests if old release missing command

Require ceph-objectstore-tool to be available on all OSD nodes
Log a message when tool is not available

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f9e088fcbcf06adc62c32f370e90591312491906)

ceph_objectstore_tool: fix pep8 errors

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 75287f0868c12c462f6758fcb11c60a74c24ced5)

ceph_objectstore_tool: --op list now prints [pg,object]

The format of the output of --op list was changed to include the PG to
which the object belong. It simplifies the loop in
ceph_objectstore_tool.py.

http://tracker.ceph.com/issues/10376 Fixes: #10376

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 3c2aaa9c359ff7f9273d0e0bcc139ee046e9a128)

ceph_objectstore_tool: Changes for rename of binary ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eec179c687664664eda6106d6705305fb16a0e82)

Conflicts:
tasks/ceph_manager.py

ceph_objectstore_tool: minor fixes

Remove development comment
nullfd isn't used anymore
s/log.errors/log.error/

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 6923e68dacd1d0230897b69a40b2005f5867723a)

ceph_objectstore_tool: fix flake8 issues

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a09d3a389dbb3dea8bb337c5b2af7b172ccc66d5)

ceph_objectstore_tool: add assert so that teuthology knows when this test fails

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f10edda4bc60faf8278e6c1a453c07c0ff317877)

ceph_objectstore_tool: skip unused osds

When the number of objects is low, some OSDs may not be used at all.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 9a089ef369174a241098f2c5c28bcb29fe09eb58)

ceph_objectstore_tool: run tests on erasure coded pools

Create an erasure coded pool and run tests on it. The list of PGs is
adapted to contain the shard id.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit be39bd5d84613d30eecede78172619bb5339dbe0)

ceph_objectstore_tool: only run get/set bytes for replicated pools

It needs to be adapted to run with erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fbb33f3577d430ffcd62a4f70b1e0de3ae28af74)

ceph_objectstore_tool: use CephManager::get_pool_dump

Instead of custom made get_pool_id function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 2b57f4c59c045b042c779ffe384fdcccf9a6ddb4)

ceph_objectstore_tool: add pgnum option

Instead of hardcoding 12 use a configuration option that defaults to
12. It is handy during development to lower the number to 4 and speed up
the test cycle.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d0caf5cdffe26f27b0a25ebdb96d9b0fd37ffffc)

ceph_objectstore_tool: encapsulate into a function

Move code out of the task into function. Also remove the "REP" specifics
from helper functions that could also be used for erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit c9e6edebd95377f7dec2a34f390f250c0e1a026d)

ceph_objectstore_tool: add hinfo_key tests for erasure coded objects

When an hinfo_key attribute is found, assume an erasure coded object and
verify set-attr/get-attr works as expected by removing its content and
restoring it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d5912263731e79877a0ea459d3b7453a925402e3)

ceph_objectstore_tool: keep all json object representation

For erasure coded pools to be tested, the json object representation
must be preserved for all PG because they are all different. The
internal representation is changed from

   db[name]["pgid"] = pg
   db[name]["json"] = objjson

to a per pg map:

   db[name].setdefault("pg2json", {})[pg] = objjson

and the rest of the code is modified to adapt accordingly.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 9ef8887a3b219021296f18eb5b317e3134f1281e)

ceph_manager: add pool type constants

The osd dump command displays pool types using numerics instead of
symbolic names. Create constants in the CephManager class to use instead
of numbers.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b925be13bc1183b2a8f68fd59d7fcb59330d1404)

ceph_manager: fix typo in get_pgid docstring

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fd1ec39d3835e95cbe0a6853a63689e217d6567e)

ceph_manager: add get_pool_dump to return the json pool dump

Get the part of ceph osd dump specific to the pool given in argument.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 5fc42b98ab22a9b2edad4336677270b11a42d776)

ceph_manager: add get_osd_dump_json to get the full output

Converts ceph osd dump_json into a python object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f353c150a8234644a25d7eeb073f4d3dd4af7c07)

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Don't need to explicitly turn off the test during some upgrades
Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a295c18a808033dfbdacf8801c26fed137ccab35)

Conflicts:
suites/upgrade/dumpling-firefly-x/stress-split/13-thrash/default.yaml (not in firefly)
suites/upgrade/dumpling/fs/4-final/osdthrash.yaml
suites/upgrade/dumpling/rbd/4-final/osdthrash.yaml
suites/upgrade/dumpling/rgw/4-final/osdthrash.yaml

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bdbcf760d9060b3b8b940a52ce2fcd8406c8b95a)

document 'command' requirements on admin_socket method

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 4db95170e605b3c602a8e4749278943860623c9d)

tasks/ceph_manager: enable log for ceph_objectstore_tool

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7ba50e0c89eead17e7fc37232f8c9a91855ebd8c)

Conflicts:
tasks/ceph_manager.py

included an option to ceph_objectstore_tool, whenever we have keyvaluestore_backend as a configurable parameter

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>
(cherry picked from commit a5a1cce3c768df18436fec795b19f5b4e7899cac)

ceph_manager: Add test code to use export/import to move a pg

Check for more than 1 osd down and randomize on chance_move_pg (100%)
For now only export from older down osd to newly down osd to avoid missing map

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 05eee9fa79ef4cb432cb28e3a2eb0e66fd686a7b)

Conflicts:
tasks/ceph_manager.py

ceph_manager: Implement export/import when thrasher kills an osd

Use list-pgs to avoid races by seeing actual pgs present

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 0cdf6e813db6bddd4dd2b97b8722ed32a4cf56a8)

ceph_objectstore_tool: Add task for testing of tool of the same name

Based on ceph/src/test/ceph_objectstore_tool.py but only does
replicated pool testing and doesn't test argument validation.

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 9ade22dd3458f15eec03ad42b599beef10b0bf20)

move functions from ceph to ceph_manager

mount_osd_data and make_admin_daemon_dir are only used by
ceph_manager.py although they are defined in ceph.py

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f53ea258a4c86603f0982448b191488abb4e73f3)

Conflicts:
tasks/ceph.py

workunit: include /usr/sbin in the PATH for all commands

Fixes: #10600
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0e53f5f38b0cb06a6f4989e622a279c8ba9104eb)

Merge pull request #355 from dachary/wip-cram-firefly

all tests must end with .t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pull fsstress.sh tests from the firelfy branch

Signed-off-by: Loic Dachary <loic@dachary.org>

all tests must end with .t

The hb=firelfy cannot end the URL because of this constraint.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: fix typo in url

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #351 from dachary/wip-cram-firefly

pull cram tests from the firelfy branch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #352 from dachary/wip-qemu-firefly

Use an NFS mount instead of virtio-9p in the qemu task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Use an NFS mount instead of virtio-9p in the qemu task

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 93f2bea3530a523c7b2ac0575033697cea893f2d)

pull cram tests from the firelfy branch

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #348 from ceph/wip-10954-firefly

linter fixes for firefly

fix path for tox.ini to work on ceph-qa-suite only

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove unused import from ceph task

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove unused variable assignment

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9df27f7a1040dae69ef65122d13d1640cb1b6a66)

initial tox.ini to run flake8

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7108b699fda247ce12cbeaea9dd0a32843775c27)

ignore tox hidden dir

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4068f73ade68e36c1e7653733eacf2b1288cd753)

fs: move into the ceph-fuse mount before doing a test

Fixes: #10465
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8a5f76d10bc8bf0358f5215ac2aadfb210691507)

Add labels to a couple common task failures

The labels will help us quickly determine what the command was trying to
accomplish in log output.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a965d6ca142c12c74124f73c9ef874df85e07e3d)

Conflicts:
tasks/workunit.py

extra_packages: stop trying to install libcephfs stuff

These package names all differ across distros, and they're included in
teuthology's install task anyway.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 977a221f9226cf479c9d2c1fbdd1d8df13bbce44)

tiobench: remove more of it which was somehow skipped

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

do not try and install libcephfs-java via extra_packages

It's been added to the default install task, because the
package name varies across distros.

Fixes: #10510
Signed-off-by: Greg Farnum <greg@gregs42.com>
(cherry picked from commit 30e5f3d93192a6bf2428ea4f8d793e2a087a5353)

samba: run only on ubuntu until #10627 is resolved

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 1dcff968a4fc359086723e8a3cb941051297f14a)

Use qemu-kvm in the qemu task for rpm based systems.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #299 from ceph/wip-10538

Add v0.80.8 to firefly stable suites (issue 10538)

Added "force-branch: firefly-original" to make sure we run on good s3tests branch

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Fixed typos

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added new v0.80.8 to
/newer
singleton/version
singleton/version-x
Added more logging

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #297 from ceph/wip-10525

Changed to "osd map max advance: 1000"

Added warning "osd_map_max_advance" to the whitelist to aviod fail on it

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Changed to "osd map max advance: 1000"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #295 from ceph/wip-10525

Changed to "osd map max advance: 1000"

Changed to "osd map max advance: 1000"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #294 from ceph/wip-10525

Replaced step for v0.80.6 with v0.80.7

Replaced step for v0.80.6 with v0.80.7

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #292 from ceph/wip-10525

Fixes http://tracker.ceph.com/issues/10525

Fixes http://tracker.ceph.com/issues/10525

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #289 from ceph/wip-10525

Fixes http://tracker.ceph.com/issues/10525

Fixes http://tracker.ceph.com/issues/10525

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge remote-tracking branch 'gh/wip-slow-backend-firefly' into firefly

drop tiobench tests

The tiobench software has been abandoned upstream for years. Fedora and
Debian are no longer shipping the tiobench package, so we've had to
carry the package ourselves in the Ceph project, and we're trying to
slim down our dependencies where it makes sense to do so.

Nuke the tiobench tests.

http://tracker.ceph.com/issues/10152 Refs: #10152

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 8a317125fc542667e57c66017db6e80a3edc24f2)

Conflicts:
suites/kcephfs/cephfs/tasks/kclient_workunit_suites_tiobench.yaml

tasks/s3tests: add slow backend configurable

Adding this so that we can modify the clients' conf file as needed with slow backend.
This can be achieved by:

overrides:
  s3tests:
    slow_backend: true

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 61409179dfdd6969c624dd741813291332644003)

rgw: mark ec related rgw suites as slow backend

We want to avoid tests that time out due to a too slow backend.

Reviwed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Inherit teuthology's log level

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 31ff241f5a74d455d6e61f02065f2df43d3fa978)

Merge pull request #267 from ceph/wip-firelfy-yuriw

Removed import_export tests to fix #10176

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed import_export tests to fix #10176

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>