suites/rados: add test for 11429

This patch also adds some convenience facilities for making
some of the ceph_manager methods into tasks usable from a
yaml file.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 015ed70f8a46a44e35433d1f701e37f68af31cf9)

Added hammer-client-x to the mix
Renamed dirs to better reflect what is being tested
Removed old dirs
Added logging
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 3c8730d035a90f221c05526983c40f0a11da1ba6)

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #430 from ceph/wip-11495-hammer

#11495/hammer: backport ceph_deploy task fix

Merge pull request #429 from dachary/wip-10712-hammer

tasks: fix intermittent failure in TestFlush

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/ceph_deploy: remove some dead code

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 99951f8150f1487c3bad47c2b199a137226055cb)

tasks/ceph_deploy: fix for multiple mons

Now that service IDs are modified during run, we have
to avoid repeatedly evaluating first_mon for where
to run ceph_deploy, as the answer will change.

Fixes: #11495
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit dbe2ad24212b9ff9a183d931402a49b544307e00)

tasks: fix intermittent failure in TestFlush

This was an overly strict success condition: the
flush operation doesn't promise to leave you an empty
journal, it promises that anything in the journal
before the flush will be flushed.

Fixes: #10712
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d20d9238cd59bca002d682cbcdc444e15e856de1)

knfs: remove the MDS debugging

It doesn't fit on the typica boxes with debugging, and we probably don't need
it for any NFS issues which do arise...

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 47fff708f936b7380446a4452a9f69f4d1c93c05)

Removed redundant tests issue #11520
Reduced nodes to be locked to two

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 8ab46b42c5ae387141917bb8eff35960e9c45699)

Merge pull request #424 from ceph/wip_ceph_deploy

Removed unneeded workloads

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed unneeded workloads

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #400 from ceph/wip_ceph_deploy

 Wip ceph deploy

Merge pull request #418 from ceph/hammer-10624

[hammer backport] task/samba: ignore return code of fuser/losf

task/samba: ignore return code of fuser/losf

if mount point is not used by anyone, both fuser and losf return 1

Fixes: #10624
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit c049387b7b0832e78a4a1370691798f726985799)

Merge pull request #397 from ceph/wip-11316-hammer

hammer: fix ceph-deploy task with CephFS (#11316)

tasks/ceph_deploy: configure CephFS

This test apparently had not been touched since
"fs new" was added.  In addition to calling
Filesystem.create:
 * modify the get_nodes_using_role
   function to modify ctx.cluster.remotes so that the
   service IDs match what ceph-deploy will set
 * log exceptions during ceph_deploy setup, as otherwise
   they can get lost if another exception occurs during
   teardown (so that it's all easier to debug).
 * default to passing --dev=master during install, so
   that we don't error out horribly when run without
   an explicit branch set (e.g. when run outside
   scheduled suite)

Fixes: #11316
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 7c2e6cdc6cd3d02969d3494d120b12daaa13efa8)

tasks/cephfs: cope with missing ctx.daemons attr

...so that we may be used with the ceph_deploy task
that doesn't set that up.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f36011051cabb8e7b06194c778e640e98d328e8a)

tasks/cephfs: tweak use of mon for admin commands

... s/mon_remote/admin_remote/ and allow caller to pass
in which remote they want to use for that.  Enables use
with ceph_deploy task which does not give admin keys
to mons.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8b61310ce3eedeb8ac60446933cd5db548de6434)

tasks/ceph: refactor legacy FS configuration check

Move up into Filesystem so that this can be used from
the ceph_deploy task as well.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit ea7c39222ae084c90b5a8510ce5af35a2f75a6e3)

suites/fs: update log whitelist

These ones had been passing because of #11177,
now that that is fixed they require an update.

Fixes: #11300
Signed-off-by: John Spray <john.spray@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit fa2a86196c3ed39c1726be689e42ed5d4aae197a)

Merge pull request #410 from ceph/wip-hammer-git

git.ceph.com (hammer)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

git.ceph.com (hammer)

Signed-off-by: Sage Weil <sage@redhat.com>

Reduced ammount of nodes from 4 to 3
Added logging
Changed replica "osd pool default size: 2"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

smoke: white marked down on python test

Backport: hammer
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 8cc4a5b0ed21b8ddef7e304d02e903d960b8bcfb)

Removed per #11070 and resolves #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 8a0de9a8ba18b8761f1612e4cdc5a306d320e30d)

Reduced ammount of nodes from 4 to 3
Changed default 'replica' to 2
Reduced osd to 2 per node
Added logging

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

msgr: move async msgr tests to singleton-nomsgr

Introduced: e92d2e4ca0013fa379fdf5830d4fe28abf65c33b
Fixes: 10918
Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e9d713b75182292c7b14c862bdeb92279ebd6b32)

Merge pull request #391 from ceph/wip-11306

Fixed #11306 Whitelist WRN "failed to encode map"

Reviewed-by: Samuel Just <sjust@redhat.com>

Fixed #11306 Whitelist WRN "failed to encode map"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

rados/thrash*: make scrubs happen a lot

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 4d8bc2127f753fb5b935af6c3607e39b077e83d2)

(scrub|rados)_test: tolerate best guess digest errors as well

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 9175f88943d2ea8382e999b7b0bee0b0352f8aef)

rados_python: whitelist wrongly marked me down

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 90393886fcf7f0a72829605ad4e52bfb7dcdf14b)

fs: fix up dd testing again

We need to change into the directory during the same shell session as we actually
run the dd...

Fixes: #10861
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8aeacaa59f32b15aa95a529d1040cb73cbd7efcc)

tasks/watch_notify_same_primary: wait for watch before notify

Make sure watch is done registering and ready before sending the
notifies.

Fixes: #10634
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bafe87a8e51cdedad84ac6ea8ecd6504a7d9dd2e)

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 97e6d808f086834f2350084d5de36be285ad2bde)

Merge pull request #377 from ceph/wip-11166

Fixes #11166, whitelisted 'Missing health data for MDS'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #11166, whitelisted 'Missing health data for MDS'

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #374 from ceph/wip-11139

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Fixes: #11139
Signed-off-by: David Zafman <dzafman@redhat.com>

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7ea0a424a53bf68cad4cd4c7c78a07401dd6851)

Implemented "exclude_packages" feature #11043

Fixed per Josh's comment

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 4250e22dde45c97ad24793f13fa41e95f16d03a4)

Merge pull request #363 from ceph/hammer-backports-hadoop

hadoop: fixup for new replication test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

hadoop: fixup for new replication test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit 6a348ae1870eb4e22a35f4ea86d897fb6d960b88)

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2e4d884fdaabbeca67b519537a80166d3cc99fa3)

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 811c02e3922c4ae29c09bb5f7b48840d0eced951)

Use an NFS mount instead of virtio-9p in the qemu task

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 93f2bea3530a523c7b2ac0575033697cea893f2d)

workunit: include /usr/sbin in the PATH for all commands

Fixes: #10600
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0e53f5f38b0cb06a6f4989e622a279c8ba9104eb)

rename test to tests, follows best practices for tests dirs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b2062602f7d3de206f4660c3e2847f86ac41e24)

create a new get_acl helper to ensure proper xml is compared

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b6ff638735d03c1f3304198fad51ca701a2c8bd)

tests for the new acl helper in radosgw_admin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3801ac9357f78ac05fa850d8944651f26c556d68)

Changed suite name from multi-versions to client-upgrade

Moved it under upgrades
Replaced rbd workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit bcf7eabc5f0bcd0c6665bc047c076925fbc9bdf5)

Merge pull request #349 from ceph/wip-10954-hammer

lint fixes for hammer

remove unused import from repair_test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ignore tox hidden dir

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4068f73ade68e36c1e7653733eacf2b1288cd753)

remove unused variable assignment

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9df27f7a1040dae69ef65122d13d1640cb1b6a66)

remove unused import from fuse

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove uneeded variable assignment for call

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a65e90db2e40b04c7cbb2347f4c759e283a2c2e)

remove unused import in calamari_setup

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d22cd6917bf77a57c323684b8c6e3687238df324)

remove unused imports from populate_rbd

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 959c4914629bd237d10cf5789906eba42b56a984)

remove unused imports from peering_speed

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6d45352782deb5606c8dcad39c75a94e748e292c)

remove redefined StringIO import

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3a18cb29a855a4b507f79a349821cb246a12c8c6)

stdin is no longer a kwarg

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 49a61dc2d2f66bd8dde34ba67dbd98d4c5148c9e)

add the log object to ceph_manager

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7c1ca4a1e23b03ecf86d5e67d563f95b3a82f99)

add missing mon object to osd_failsafe

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1673be45c0e3aeb1eadad7ecd5b87570e9bac5ca)

fix undefined var in divergent_priors

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 26c5a82af9cf4efd3601b6a32aa5444f61fa9f81)

exclude the virtualenv from being linted in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

fix path for tox.ini to work on ceph-qa-suite only

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4eb1787280f9f128cb9352ecbe8957a4be94fff9)

initial tox.ini to run flake8

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7108b699fda247ce12cbeaea9dd0a32843775c27)
(cherry picked from commit e1c85f330b834394d099653d3c618d41cd438cff)

rbd: fix copy-on-read setting

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit aa6240c6f4cb8f258072fbfb94edb027bff6a3f4)

rbd: add merge_diff test

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit e7bf3d05a63f7cf99b3ff46fb9ac96e73ffb5c40)

suites/rbd: add object map tests

Enable the object map feature for CLI tests and librbd API tests.
Moved the CLI tests to a new group to exercise old format, basic
new format, and new format w/ object map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b277946848fc2b5f454a7c6ffd8fa2a359369102)

ceph: ugh fix syntax

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1922c61bbf1165e23650e74767ac4d5f63becdbc)

Fixes #10869, added {role} to the dir name.

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
Fixed typo

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 9942440ddf9f6847e76de3b49f1ad0d98c62a3d0)

ceph: fix ps axuf lsof line

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 18307be0ca97f7221d6202884f8a68a4b26c1dbf)

ceph: ps axf too before lsof

Specifically, I want to know *who* is running the ceph-osd that is
holding the files open.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a68281e1476e6af38237e1d1031dd7bd0980ef9f)

ceph: lsof if umount fails

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 077e917af1394e5999293031afcc476b2bdce1e4)

ceph: archive logs at very end

This ensures that we still gather the logs even if the other nested tasks
throw an exception in the finally block.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ca09683f5fc1a6067c524c4034c27ab4a26e11f3)

Fixed typoes

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit e2794173026c196a30ee042e6e77c4a278267dc2)

Fixed typo causing "too many values to unpack" error

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit fdd175514738fd4f6cf3dea6d56ffb2eda78d64b)

Fixes #10704

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit a9252436e81ab7d21110ffbe2f6b469f08e47d82)

Fixes #10652
Removed firefly-giant dir as it's not needed
Fixed branch assignment in dumpling-x

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit a91994f09201fdf497798d2c8051b27189b5e486)

Merge branch 'hammer' of https://github.com/ceph/ceph-qa-suite into hammer

Merge pull request #328 from ceph/wip-10736

Created a new suite to test teuthology

Created a new suite to test teuthology

This suite locks machines of varying distros and runs the 'tests' tasks
against them. This will test both vps locking / provisioning and
baremetal locking.

Also, we test the chef and install tasks with ubuntu and rhel.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #327 from ceph/wip-10702

tasks/cephfs: fix fuse force unmount

tasks/cephfs: fix fuse force unmount

This was broken in the case of multiple
mounts, and in the case of stuck mounts.

Fixes: #10702
Signed-off-by: John Spray <john.spray@redhat.com>

Corrected "branch" to be outside of the 'clients:' section

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Corrected "branch" to be outside of the 'clients:' section

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #324 from dmick/master

tasks/calamari_setup: ice_setup 0.2.0 has added another prompt

tasks/calamari_setup: ice_setup 0.2.0 has added another prompt

Add an extra '\n' to the interaction string to answer it

Signed-off-by: Dan Mick <dan.mick@redhat.com>

extra_packages: stop trying to install libcephfs stuff

These package names all differ across distros, and they're included in
teuthology's install task anyway.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 977a221f9226cf479c9d2c1fbdd1d8df13bbce44)

extra_packages: stop trying to install libcephfs stuff

These package names all differ across distros, and they're included in
teuthology's install task anyway.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

overrides/short_pg_log: 30/50 was too short, make 100/200

Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #322 from ceph/wip-10753

Add labels to a couple common task failures

Add labels to a couple common task failures

The labels will help us quickly determine what the command was trying to
accomplish in log output.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #319 from athanatos/wip-min-size

Wip min size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #304 from yuyuyu101/add-msgr-test

rados: include msgr tests

Fixes http://tracker.ceph.com/issues/10652

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

rados/thrash: add short_pg_log variant to encourage backfill

Signed-off-by: Samuel Just <sjust@redhat.com>

do not try and install libcephfs-java via extra_packages

It's been added to the default install task, because the
package name varies across distros.

Fixes: #10510
Signed-off-by: Greg Farnum <greg@gregs42.com>