Merge pull request #328 from ceph/wip-10736

Created a new suite to test teuthology

Created a new suite to test teuthology

This suite locks machines of varying distros and runs the 'tests' tasks
against them. This will test both vps locking / provisioning and
baremetal locking.

Also, we test the chef and install tasks with ubuntu and rhel.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #327 from ceph/wip-10702

tasks/cephfs: fix fuse force unmount

tasks/cephfs: fix fuse force unmount

This was broken in the case of multiple
mounts, and in the case of stuck mounts.

Fixes: #10702
Signed-off-by: John Spray <john.spray@redhat.com>

Corrected "branch" to be outside of the 'clients:' section

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #324 from dmick/master

tasks/calamari_setup: ice_setup 0.2.0 has added another prompt

tasks/calamari_setup: ice_setup 0.2.0 has added another prompt

Add an extra '\n' to the interaction string to answer it

Signed-off-by: Dan Mick <dan.mick@redhat.com>

extra_packages: stop trying to install libcephfs stuff

These package names all differ across distros, and they're included in
teuthology's install task anyway.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

overrides/short_pg_log: 30/50 was too short, make 100/200

Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #322 from ceph/wip-10753

Add labels to a couple common task failures

Add labels to a couple common task failures

The labels will help us quickly determine what the command was trying to
accomplish in log output.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #319 from athanatos/wip-min-size

Wip min size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #304 from yuyuyu101/add-msgr-test

rados: include msgr tests

Fixes http://tracker.ceph.com/issues/10652

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

rados/thrash: add short_pg_log variant to encourage backfill

Signed-off-by: Samuel Just <sjust@redhat.com>

do not try and install libcephfs-java via extra_packages

It's been added to the default install task, because the
package name varies across distros.

Fixes: #10510
Signed-off-by: Greg Farnum <greg@gregs42.com>

Merge pull request #316 from ceph/wip-active-asok-10361

mds_scrub_checks: only direct ops at the active MDS

Reviewed-by: John Spray <john.spray@redhat.com>

rados: add size/min_size combinations to rados/thrash

Signed-off-by: Samuel Just <sjust@redhat.com>

fs: move into the ceph-fuse mount before doing a test

Fixes: #10465
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #312 from ceph/wip-prio-hb-pkts

exercise osd heartbeat prioritization

Merge pull request #296 from majianpeng/master

rados.py: Add write_fadvise_dontneed field

Merge pull request #302 from ceph/wip-crush-tunables

ceph: set crush tunables

rados: exercise a few different crush tunable profiles

Signed-off-by: Sage Weil <sage@redhat.com>

mds_scrub_checks: only direct ops at the active MDS

Change the config option from mds_id to mds_rank to reflect the
fact that it's the rank we want to make use of (and will continue
to want when we're doing stuff like force exporting from one rank
to another).

Fixes: #10361
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #315 from ceph/wip-10577

Added "branch: dumpling" to fix #10577

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Added "branch: dumpling" to fix #10577

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

ceph: set crush tunables

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #314 from ceph/wip-wait-for-failure

ceph: test pg removal interruption; add ceph daemon failur einjection support

suites/rbd: add copy-on-read tests

Add as a facet of librbd tests, since they all use cloning, and to a
couple tests with thrashing.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

suites/rbd: split qemu and librbd tests

qemu tests take a long time. We'll want to test more combinations of
options with librbd directly.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

suites/rbd: Add rbd api tests with exclusive locking features

Keep testing the old, non-default features in a renamed yaml file.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #303 from ceph/wip-10388

tasks/mds_flush: use new stray/purge perfcounters

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rados/singleton: test interrupted pg removal

Inject osd failure during pg removal.  Verify the OSD successfully
recovers.

Signed-off-by: Sage Weil <sage@redhat.com>

mds_flush: fix the flush test output expectations

Fixes: #10620
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph: no need to stop() before restart()

restart() will stop if the daemon is running.  This will get rid of the
spurious error

2015-01-23 15:19:36,828.828 ERROR:tasks.ceph.osd.0:tried to stop a non-running daemon

when the daemon isn't already running.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: add wait_for_failure command

Expect and then ignore a daemon failure.

Signed-off-by: Sage Weil <sage@redhat.com>

samba: run only on ubuntu until #10627 is resolved

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 1dcff968a4fc359086723e8a3cb941051297f14a)

samba: run only on ubuntu until #10627 is resolved

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #277 from ceph/wip-9780-9781

ceph-objectstore-tool import/export and related fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

suites/rados: exercise heartbeat prioritization

Signed-off-by: Sage Weil <sage@redhat.com>

Formatting changes

Fix wrapped line

Fix objectstore name changed to keyvaluestore

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: Fix Thrasher kill_osd() to ignore pool going away

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: Thrasher fixes to run() calls to set check_status = False

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: Re-enable Thrasher ceph-objectstore-tool testing

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: In Thrasher randomly remove past intervals

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: In Thrasher disable ceph_objectstore_tool tests if old release missing command

Require ceph-objectstore-tool to be available on all OSD nodes
Log a message when tool is not available

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #308 from ceph/wip-10579

suites/fs: isolate quota task

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suites/fs: isolate quota task

Quota script is moved out of fs/misc so that kclient
doesn't try to run it.  Update cfuse suite to find
the new location, and only enable quota for this
one script rather than enabling across
all misc tests.

Fixes: #10579
Signed-off-by: John Spray <john.spray@redhat.com>

rados: include msgr tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

tasks/mds_flush: use new stray/purge perfcounters

Where previously we only tracked RADOS-level delete
ops during deletion, now also verify that they
correspond to the right number of MDS-level purge
operations.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #285 from ceph/wip-9883

tasks/mds_journal_repair: create new test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/mds_journal_repair: add test_table_tool

Explicit test of the tool's various modes, to
complement the places it's used in other tests.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mds_journal_repair: create new test

This tests the new #9883 repair functionality
where we selectively scrape dentries out of
the journal while the MDS is offline.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #298 from ceph/wip-bi-sharding-3

Wip bi sharding 3

Merge pull request #300 from ceph/restore_giant-x_on_master

Restored giant-x suite on master

Merge pull request #301 from ceph/restore_giant-x_on_next

Restore giant x on next

Restored giant-x on next

Signed-off-by: Yuri Weinstein <yuriw@magna002.ceph.redhat.com>

Restored giant-x suite on master

Signed-off-by: Yuri Weinstein <yuriw@magna002.ceph.redhat.com>

Merge pull request #288 from dmick/master

Tweaks to calamari-setup

radosgw_admin: don't use boto for testing 301 responses

boto follows up on the redirects. Switched to using httplib2 for sending
this request.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set default region and zone to all clients

Otherwise it's going to use zone 'default' for client.0, not as expected. Also
add region configuration to task.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #291 from dachary/wip-10018-scrub

add erasure code repair / scrub tests (take 2)

Reviewed-by: David Zafman <dzafman@redhat.com>

test: add write_fadvise_dontneed.yaml to test write_fadvise_dontneed.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rados.py: Add write_fadvise_dontneed field

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

repair_test: hinfo_key error injection

Add a function dedicated to erasure coded pools tests, similar to
repair_test_1. Add a corrupter that removes the hinfo_key from the object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add ceph-objectstore-tool wrapper

Add the CephManager.objectstore_tool method to encapsulate a call to
ceph-objectstore-tool. The wrapper can convert an object name into the
PG id and figure out the primary OSD. The designated OSD is stopped
before running the command and restarted afterwards.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add object helpers

Add CephManager.methods to get information regarding the pools and PGs
used to store a given object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

repair_test: rework indentation to make it pep8 clean

Signed-off-by: Loic Dachary <ldachary@redhat.com>

repair_test: rework to remove levels of indirection

The commit is large but does not introduce any semantic change and
consists primarily in code moving around, re-indented and removed.

Replace functions generating functions by functions and sequentially
iterating over a list of functions with a sequential call to the
functions.

Replace the setup/teardown with an equivalent using a with
statement and the ceph_manager.pool method.

Replace inline code with a call to ceph_manager.wait_for_all_up

It makes it easier to modify the tests, for instance to create erasure
coded pools and tests specific to them.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: rework indentation to make it pep8 clean

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: fix docstring typo

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: define pool context

To create a pool before running a code bloc and remove it after.

   with manager.pool("mypool"):
        mytest..

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #290 from ceph/revert-250-wip-10018-scrub

Revert "add erasure code repair / scrub tests"

Revert "add erasure code repair / scrub tests"

Merge pull request #286 from ceph/wip-10261-wusui

Calamari_setup now allows > 1 mon.

Merge pull request #250 from dachary/wip-10018-scrub

add erasure code repair / scrub tests

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

repair_test: hinfo_key error injection

Add a function dedicated to erasure coded pools tests, similar to
repair_test_1. Add a corrupter that removes the hinfo_key from the object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add ceph-objectstore-tool wrapper

Add the CephManager.objectstore_tool method to encapsulate a call to
ceph-objectstore-tool. The wrapper can convert an object name into the
PG id and figure out the primary OSD. The designated OSD is stopped
before running the command and restarted afterwards.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: add object helpers

Add CephManager.methods to get information regarding the pools and PGs
used to store a given object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

repair_test: rework indentation to make it pep8 clean

Signed-off-by: Loic Dachary <ldachary@redhat.com>

repair_test: rework to remove levels of indirection

The commit is large but does not introduce any semantic change and
consists primarily in code moving around, re-indented and removed.

Replace functions generating functions by functions and sequentially
iterating over a list of functions with a sequential call to the
functions.

Replace the setup/teardown with an equivalent using a with
statement and the ceph_manager.pool method.

Replace inline code with a call to ceph_manager.wait_for_all_up

It makes it easier to modify the tests, for instance to create erasure
coded pools and tests specific to them.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: rework indentation to make it pep8 clean

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: fix docstring typo

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph_manager: define pool context

To create a pool before running a code bloc and remove it after.

   with manager.pool("mypool"):
        mytest..

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Fix objectstore name changed to keyvaluestore

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: Fix Thrasher kill_osd() to ignore pool going away

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: Thrasher fixes to run() calls to set check_status = False

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: Re-enable Thrasher ceph-objectstore-tool testing

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: In Thrasher randomly remove past intervals

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_manager: In Thrasher disable ceph_objectstore_tool tests if old release missing command

Require ceph-objectstore-tool to be available on all OSD nodes
Log a message when tool is not available

Signed-off-by: David Zafman <dzafman@redhat.com>

tasks/calamari_setup: build ice-tools virtualenv if building iceball

ice-tools needs a virtualenv populated to properly run to build
an iceball; add the commands to do that.  Also remove the built
iceball when the task exits.

Fixes: #10523
Signed-off-by: Dan Mick <dan.mick@redhat.com>

tasks/calamari_setup: remove HTTP-downloaded iceball, if any

Fixes: #10522
Signed-off-by: Dan Mick <dan.mick@redhat.com>

tasks/calamari_setup: use ice_version to completely specify iceball

Previously, the task would search for the lexicographically-greatest
filename matching ICE-*.tar.gz; now it builds a specific name
ICE-{ice_version}-{ice_distro}.tar.gz

Fixes: #10521
Signed-off-by: Dan Mick <dan.mick@redhat.com>

fs: enable client quota on cfuse_workunit_misc

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Calamari_setup now allows > 1 mon.

Signed-off-by: Warren Usui <warren.usui@inktank.com>
Fixes: #10261

Merge pull request #279 from ceph/wip-cephfs-cleanup

Assorted cleanup in the CephFS tests

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/mds_auto_repair: remove unneeded log settings

The small segments and small segment limit
were used when doing a hacky flush by doing
IO and waiting: now that we have the explicit
'flush journal' asok in use, we can just use
a normal journal configuration.

Signed-off-by: John Spray <john.spray@redhat.com>