Merge pull request #607 from dachary/wip-git-ceph-com-firefly

git ceph com and newdream hostname changes

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a14a951640f79fd7ed217c423dcc9f453c830556)

Conflicts:
suites/rbd/qemu/workloads/qemu_bonnie.yaml
suites/rbd/qemu/workloads/qemu_fsstress.yaml
suites/rbd/qemu/workloads/qemu_iozone.yaml.disabled
suites/rbd/qemu/workloads/qemu_xfstests.yaml
suites/rbd/singleton/all/formatted-output.yaml
the conflicts are because https is used in master instead of http,
the rest is otherwise identical.

tasks/chef.py
obsolete, replaced by ansible.

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f062b171f4ab191986cea525c831573109aba2f9)

teuthology: some suites still use http://ceph.newdream.net

This probably redirects to http://ceph.com but ceph.newdream.net still appears in some places

http://tracker.ceph.com/issues/9922 Fixes: #9922

Signed-off-by: Armando Segnini <armando.segnini@telecom-bretagne.eu>
(cherry picked from commit 9ace238e874b45afbdcef97fe656d9a707e6105a)

Merge pull request #588 from ceph/wip-12573

Removed cuttlefish from configurations as unsupported

Removed cuttlefish from configurations as unsupported

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #504 from ceph/wip-firefly2

Partly reverted ref to firefly-orginal

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Partly reverted ref to firefly-orginal
See https://github.com/ceph/ceph-qa-suite/pull/495/

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #501 from ceph/wip-12425-firefly

Fixes #12425, removed versions-steps-x suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #12425, removed versions-steps-x suite
from upgrade/firefly

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #495 from ceph/wip-firefly

Changed 'firefly-original' ref to 'firefly'

Changed 'firefly-original' ref to 'firefly'
same as in https://github.com/ceph/ceph-qa-suite/pull/494

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Drop chef from tgt suite

Chef has been replaced by ansible, which will be added to jobs at
schedule time.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 5b57b81e90393d42494c7b02fb263f1601421f16)

Removed rhel* configurations with a goal that they will be covered in the Octo lab
and won't be used by vps runs in Sepia lab

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit adfa1ee393c0ea0a36c0dfb8d7fda1a82f4b29ea)

Merge pull request #467 from ceph/wip-12022

Added firefly v0.80.10 to the mix

Added firefly v0.80.10 to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #459 from dachary/wip-ceph-manager-log

add the log object to ceph_manager

add the log object to ceph_manager

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7c1ca4a1e23b03ecf86d5e67d563f95b3a82f99)

Conflicts:
tasks/ceph_manager.py
lines shifted but nothing significant changed

Merge pull request #456 from ceph/wip-11937-firefly

thrasher: Can't test ceph-objectstore-tool if nodes turned off (power…

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

thrasher: Can't test ceph-objectstore-tool if nodes turned off (powercycle)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 8a9bcd1eb7e8e7acb132b09574bb5bf6d6d52b10)

Conflicts:
tasks/ceph_manager.py
        Logging message

Merge pull request #435 from dachary/wip-11429-firefly

suites/rados: add test for 11429 (firefly)

initialize ctx.manager in ceph.py

instead of rados.py because ceph.py is only run once where rados.py
could be run multiple time, leading to race conditions

http://tracker.ceph.com/issues/9027 Fixes: #9027

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 9782465c872b25710af278c701e15078a3748d03)

Conflicts:
tasks/rados.py
        because  if not hasattr(ctx, 'manager'): was not at the same place

squash: use sleep instead of utility.sleep

(cherry picked from commit cc107525bca02ca44fabe8513203680e0d1e422b)

squash: use != not is not in radosbench

(cherry picked from commit 91bc0e91864903f6635879432e6dc7a0b36c76c2)

Conflicts:
tasks/radosbench.py
the prototype of the functions changed but not the logic

squash: remove utility, moved to teuthology.git

(cherry picked from commit 73577f1077dcfb059d7a96d7502a147a943cd261)

squash: ceph_manager: add utility_task doc string

(cherry picked from commit 2a60852a1d0b0b7980004cf262d4af8ea4011667)

suites/rados: add test for 11429

This patch also adds some convenience facilities for making
some of the ceph_manager methods into tasks usable from a
yaml file.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 015ed70f8a46a44e35433d1f701e37f68af31cf9)

Conflicts:
tasks/radosbench.py
the pool creation call changed and the conflict is because
they were indented to implement the option to not create
a pool

kcephfs: remove 'kclient' workunit tests -- there are none left now

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #437 from ceph/wip-11180-firefly

ensure summary is looked for the user we need (part 2)

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 97e6d808f086834f2350084d5de36be285ad2bde)

Conflicts:
tasks/radosgw_admin.py
        the context around def successful_ops(out): changed but is
unrelated to the modification

knfs: remove the MDS debugging

It doesn't fit on the typica boxes with debugging, and we probably don't need
it for any NFS issues which do arise...

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 47fff708f936b7380446a4452a9f69f4d1c93c05)

Merge pull request #419 from ceph/wip-11139-firefly

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Merged again by David Zafman <dzafman@redhat.com> because firefly branch lost original Apr 3 merge on Apr 16.

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Fixes: #11139
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 6c5300552d00232d6ecb2c1aa641d515c9d8cd34)

Merge pull request #412 from ceph/wip-firefly-git

git.ceph.com (firefly)

Reviewed-by: Loic Dachary <ldachary@rehdat.com>

git.ceph.com (firefly)

Signed-off-by: Sage Weil <sage@redhat.com>

fs: fix up dd testing again

We need to change into the directory during the same shell session as we actually
run the dd...

Fixes: #10861
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8aeacaa59f32b15aa95a529d1040cb73cbd7efcc)

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7ea0a424a53bf68cad4cd4c7c78a07401dd6851)

Merge pull request #369 from dachary/wip-rgw-acl

create a new get_acl helper to ensure proper xml is compared

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #368 from dachary/wip-shard-missing

whitelist shard missing errors

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #361 from ceph/wip-cot-firefly

Backport ceph-objectstore-tool changes to firefly

Reviewed-by: Loic Dachary <ldachary@redhat.com>

create a new get_acl helper to ensure proper xml is compared

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b6ff638735d03c1f3304198fad51ca701a2c8bd)

Conflicts:
tasks/radosgw_admin.py
        the context changed but the patch seems otherwise good

whitelist shard missing errors

If a shard is missing during a radosbench it can safely be
ignored. It will either be recovered and radosbench will complete
successfully. Or there are too many shard missing to recover the object
and radosbench will fail. Either way we can rely on radosbench to
accurately report success or failure.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #365 from ceph/wip-11072

Added v0.80.9 to upgrade/firefly suite, fixed #11072

Added v0.80.9 to upgrade/firefly suite, fixed #11072

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2e4d884fdaabbeca67b519537a80166d3cc99fa3)

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 811c02e3922c4ae29c09bb5f7b48840d0eced951)

Formatting changes

(cherry picked from commit e5876126a9a8a474ddffaf60bdfde960e33717da)

Fix wrapped line

(cherry picked from commit 14e27650a62016da8caecbb160fbc5bf77d5515e)

ceph_manager: Fix Thrasher kill_osd() to ignore pool going away

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b239dadc02fc8e09c5bf633a06c479494929e85c)

Conflicts:
tasks/ceph_manager.py

ceph_manager: Thrasher fixes to run() calls to set check_status = False

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4eda2967a456d1280a8f081fb80bb274ac60ac0e)

Conflicts:
tasks/ceph_manager.py

ceph_manager: Re-enable Thrasher ceph-objectstore-tool testing

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit e0b0d41e4cc87c058386bbc00bba444671493838)

ceph_manager: In Thrasher randomly remove past intervals

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4a195c78c1d932ecf03055d3cfc2e5584947b657)

ceph_manager: In Thrasher disable ceph_objectstore_tool tests if old release missing command

Require ceph-objectstore-tool to be available on all OSD nodes
Log a message when tool is not available

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f9e088fcbcf06adc62c32f370e90591312491906)

ceph_objectstore_tool: fix pep8 errors

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 75287f0868c12c462f6758fcb11c60a74c24ced5)

ceph_objectstore_tool: --op list now prints [pg,object]

The format of the output of --op list was changed to include the PG to
which the object belong. It simplifies the loop in
ceph_objectstore_tool.py.

http://tracker.ceph.com/issues/10376 Fixes: #10376

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 3c2aaa9c359ff7f9273d0e0bcc139ee046e9a128)

ceph_objectstore_tool: Changes for rename of binary ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eec179c687664664eda6106d6705305fb16a0e82)

Conflicts:
tasks/ceph_manager.py

ceph_objectstore_tool: minor fixes

Remove development comment
nullfd isn't used anymore
s/log.errors/log.error/

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 6923e68dacd1d0230897b69a40b2005f5867723a)

ceph_objectstore_tool: fix flake8 issues

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a09d3a389dbb3dea8bb337c5b2af7b172ccc66d5)

ceph_objectstore_tool: add assert so that teuthology knows when this test fails

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f10edda4bc60faf8278e6c1a453c07c0ff317877)

ceph_objectstore_tool: skip unused osds

When the number of objects is low, some OSDs may not be used at all.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 9a089ef369174a241098f2c5c28bcb29fe09eb58)

ceph_objectstore_tool: run tests on erasure coded pools

Create an erasure coded pool and run tests on it. The list of PGs is
adapted to contain the shard id.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit be39bd5d84613d30eecede78172619bb5339dbe0)

ceph_objectstore_tool: only run get/set bytes for replicated pools

It needs to be adapted to run with erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fbb33f3577d430ffcd62a4f70b1e0de3ae28af74)

ceph_objectstore_tool: use CephManager::get_pool_dump

Instead of custom made get_pool_id function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 2b57f4c59c045b042c779ffe384fdcccf9a6ddb4)

ceph_objectstore_tool: add pgnum option

Instead of hardcoding 12 use a configuration option that defaults to
12. It is handy during development to lower the number to 4 and speed up
the test cycle.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d0caf5cdffe26f27b0a25ebdb96d9b0fd37ffffc)

ceph_objectstore_tool: encapsulate into a function

Move code out of the task into function. Also remove the "REP" specifics
from helper functions that could also be used for erasure coded pools.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit c9e6edebd95377f7dec2a34f390f250c0e1a026d)

ceph_objectstore_tool: add hinfo_key tests for erasure coded objects

When an hinfo_key attribute is found, assume an erasure coded object and
verify set-attr/get-attr works as expected by removing its content and
restoring it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit d5912263731e79877a0ea459d3b7453a925402e3)

ceph_objectstore_tool: keep all json object representation

For erasure coded pools to be tested, the json object representation
must be preserved for all PG because they are all different. The
internal representation is changed from

   db[name]["pgid"] = pg
   db[name]["json"] = objjson

to a per pg map:

   db[name].setdefault("pg2json", {})[pg] = objjson

and the rest of the code is modified to adapt accordingly.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 9ef8887a3b219021296f18eb5b317e3134f1281e)

ceph_manager: add pool type constants

The osd dump command displays pool types using numerics instead of
symbolic names. Create constants in the CephManager class to use instead
of numbers.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b925be13bc1183b2a8f68fd59d7fcb59330d1404)

ceph_manager: fix typo in get_pgid docstring

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit fd1ec39d3835e95cbe0a6853a63689e217d6567e)

ceph_manager: add get_pool_dump to return the json pool dump

Get the part of ceph osd dump specific to the pool given in argument.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 5fc42b98ab22a9b2edad4336677270b11a42d776)

ceph_manager: add get_osd_dump_json to get the full output

Converts ceph osd dump_json into a python object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f353c150a8234644a25d7eeb073f4d3dd4af7c07)

Thrasher: Disable ceph_objectstore_tool tests if old release missing command

Don't need to explicitly turn off the test during some upgrades
Leaving disabled until merge of import/export fixes

Fixes: #9805
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a295c18a808033dfbdacf8801c26fed137ccab35)

Conflicts:
suites/upgrade/dumpling-firefly-x/stress-split/13-thrash/default.yaml (not in firefly)
suites/upgrade/dumpling/fs/4-final/osdthrash.yaml
suites/upgrade/dumpling/rbd/4-final/osdthrash.yaml
suites/upgrade/dumpling/rgw/4-final/osdthrash.yaml

ceph_manager: ceph_objectstore_tool testing off by default

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bdbcf760d9060b3b8b940a52ce2fcd8406c8b95a)

document 'command' requirements on admin_socket method

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 4db95170e605b3c602a8e4749278943860623c9d)

tasks/ceph_manager: enable log for ceph_objectstore_tool

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7ba50e0c89eead17e7fc37232f8c9a91855ebd8c)

Conflicts:
tasks/ceph_manager.py

included an option to ceph_objectstore_tool, whenever we have keyvaluestore_backend as a configurable parameter

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>
(cherry picked from commit a5a1cce3c768df18436fec795b19f5b4e7899cac)

ceph_manager: Add test code to use export/import to move a pg

Check for more than 1 osd down and randomize on chance_move_pg (100%)
For now only export from older down osd to newly down osd to avoid missing map

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 05eee9fa79ef4cb432cb28e3a2eb0e66fd686a7b)

Conflicts:
tasks/ceph_manager.py

ceph_manager: Implement export/import when thrasher kills an osd

Use list-pgs to avoid races by seeing actual pgs present

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 0cdf6e813db6bddd4dd2b97b8722ed32a4cf56a8)

ceph_objectstore_tool: Add task for testing of tool of the same name

Based on ceph/src/test/ceph_objectstore_tool.py but only does
replicated pool testing and doesn't test argument validation.

Signed-off-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 9ade22dd3458f15eec03ad42b599beef10b0bf20)

move functions from ceph to ceph_manager

mount_osd_data and make_admin_daemon_dir are only used by
ceph_manager.py although they are defined in ceph.py

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f53ea258a4c86603f0982448b191488abb4e73f3)

Conflicts:
tasks/ceph.py

workunit: include /usr/sbin in the PATH for all commands

Fixes: #10600
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0e53f5f38b0cb06a6f4989e622a279c8ba9104eb)

Merge pull request #355 from dachary/wip-cram-firefly

all tests must end with .t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pull fsstress.sh tests from the firelfy branch

Signed-off-by: Loic Dachary <loic@dachary.org>

all tests must end with .t

The hb=firelfy cannot end the URL because of this constraint.

Signed-off-by: Loic Dachary <loic@dachary.org>

rbd: fix typo in url

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #351 from dachary/wip-cram-firefly

pull cram tests from the firelfy branch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #352 from dachary/wip-qemu-firefly

Use an NFS mount instead of virtio-9p in the qemu task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Use an NFS mount instead of virtio-9p in the qemu task

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 93f2bea3530a523c7b2ac0575033697cea893f2d)

pull cram tests from the firelfy branch

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #348 from ceph/wip-10954-firefly

linter fixes for firefly

fix path for tox.ini to work on ceph-qa-suite only

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove unused import from ceph task

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove unused variable assignment

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9df27f7a1040dae69ef65122d13d1640cb1b6a66)

initial tox.ini to run flake8

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7108b699fda247ce12cbeaea9dd0a32843775c27)

ignore tox hidden dir

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4068f73ade68e36c1e7653733eacf2b1288cd753)

fs: move into the ceph-fuse mount before doing a test

Fixes: #10465
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8a5f76d10bc8bf0358f5215ac2aadfb210691507)

Add labels to a couple common task failures

The labels will help us quickly determine what the command was trying to
accomplish in log output.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a965d6ca142c12c74124f73c9ef874df85e07e3d)

Conflicts:
tasks/workunit.py

extra_packages: stop trying to install libcephfs stuff

These package names all differ across distros, and they're included in
teuthology's install task anyway.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 977a221f9226cf479c9d2c1fbdd1d8df13bbce44)

tiobench: remove more of it which was somehow skipped

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

do not try and install libcephfs-java via extra_packages

It's been added to the default install task, because the
package name varies across distros.

Fixes: #10510
Signed-off-by: Greg Farnum <greg@gregs42.com>
(cherry picked from commit 30e5f3d93192a6bf2428ea4f8d793e2a087a5353)