Merge remote-tracking branch 'origin/wip-vstart-runner' into infernalis

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

smoke/basic/tasks/libcephfs: fix client debug override

Also debug mds

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Conflicts:
suites/upgrade/hammer-x/point-to-point-x/point-to-point.yaml

upgrade/hammer-x/point-to-point: whitelist wrongly marked me down

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #585 from dachary/wip-buildpackages-test

buildpackages: implicit for OpenStack

Merge pull request #596 from ceph/wip-radosbench

tasks/radosbench: wait just a bit longer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

White listed WAR 'failed to encode' fixes #13349

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 37a881aee191038139b37153a25555fc6b7f8fed)

Merge pull request #602 from ceph/wip13349

White listed WAR 'failed to encode' fixes #13349

White listed WAR 'failed to encode' fixes #13349

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #549 from ceph/wip-krbd-fio

Add rbd fio test to exercise block IO on all format/features

upgrade/hammer-x/point-to-point: use different client for second rgw

Can't do 2 rgw's on the same client id.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a0f0194ab2edec81f19b111b9317a5149defd4e9)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

krbd fio tests using sync engine for format 2 and features 1 and 2.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

krbd test using fio for format 2 and features 1 and 2.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rbd fio tests using rbd engine, tests on clones as well using features 1 and 13.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rbd io tests using fio, various fio options can be specified in yaml

test rbd or krbd using fio, can also run io on rbd clones if option is specified in yaml
various options like image-size, rbd format/features, fio io size, readwrite options can be provided in yaml.
check the docstring for exact usage.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #599 from ceph/wip-hammer-x

upgrade/hammer-x/point-to-point: use different client for second rgw

upgrade/hammer-x/point-to-point: use different client for second rgw

Can't do 2 rgw's on the same client id.

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/cephfs: fix FuseMount bin path in vstart

FuseMount only uses the prefix for finding the 'ceph'
executable, which is in ./ for either cmake or
authtools, not ./src for cmake like other binaries.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix test_journal_migration

It was trying to get the output file from
a different remote than the one used to
run the journal tool.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: move mds_scrub_checks

...into a CephFSTestCase.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: move journal migration test

...into a CephFSTestCase.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: extend vstart_runner's ctx&run

Sufficiently to enable using workunits.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add --interactive for vstart runner

Just like interactive-on-error in teuthology.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix FuseMount._asok_path

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: remove a redundant sudo

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: raise error on non-string stdins

Shouldn't be any from the fs tests that get run

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: warn if vstart_runner can't import mods

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: stop if needed binaries are absent

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add instructions to vstart_runner

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: updates for cmake environ

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add vstart runner script

This is to allow running CephFSTestCase tests
against a vstart cluster, for much faster turnaround
during development than running teuthology against
built ceph packages.

Not everything will be runnable this way, but for
certain things like filesystem repair scenarios we
have everything we need within a vstart environment.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: mark some tests as @needs_trimming

So that we can drop these tests when not running
client as root.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add needs_trimming decorator

For tests to advertise that they need the client
to be able to trim its cache (i.e. currently that
means requiring run as root)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: make FuseMount.teardown safer

(don't assume fuse_daemon exists)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: mark some tests as @long_running

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add @long_running decorator

A means for test cases to mark particular methods
as long running, so that the vstart runner can skip
them when running for developers.

This is not a scientific thing, anything that takes
more than about 2 minutes due to lots of iteration
or sleeps.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: cluster_down before fs rm

In teuthology this isn't needed because we join the
mds child processes after killing them.  In vstart
we're killing them asynchronously, so be a bit more
careful to ensure they can't re-insert themselves
to the mdsmap between our calling fail and our calling
fs rm.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: split up TestClientRecovery

...into the part that requires a network-isolated
client and the part that doesn't.

This happens to also be the part that won't work with
vstart vs. the part that will.  teuthology yaml will
still pick up and run both parts.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: make memstore dependency declarative

...instead of checking for it procedurally during
TestClusterFull.setUp

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: refine TestClientLimits.test_client_oldest_tid

* Instead of creating files in background, create
  them in foreground (simpler).
* Instead of creating max_request*2 files, just create
  max_requests plus a litle bit.
* Set max_requests to 1000 instead of 5000 to run a bit
  faster.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix race in TestStrays

We weren't waiting for export dir to complete (the asok
just starts the process).  This wasn't noticeable when running
remotely due to latency between the test runner and the MDS,
but it shows up when running against a local vstart cluster.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: work around fuse weirdness

I am seeing a strange thing where it seems like sometimes
a ls of /sys/fs/fuse/connections is returning empty when
connections do exist.  It is pretty easy to make this
a non-issue by waiting for "more conns than we started with"
instead of "list of conns is different", so do that.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: prefix override in FuseMount

For asok and invokation of ceph CLI.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: use stdin/stdout in Filesystem._read_data_xattr

...so that it doesn't rely on a temp local file and using
run.Raw(">") in commands.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: handle endpoint not connected

Previously failure to stat mnt dir was interpreted
as being unmounted.  For "transport endpoint no connected"
error we do want to recognise that it is mounted, albeit
with no ceph-fuse process.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: use Filesystem.rados instead of direct CLI

Because it's slicker and because I can wrap it for vstart.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add Filesystem.exists

Use this during test setup to check whether
a filesystem is configured at all, before
trying to tear it down.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add Filesystem._prefix

So that my vstart subclass can put ./ before
all the commands.

One could set $PATH, but I like to unambiguously point
it at the local built binaries in case someone also
has some systemwide packages.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: use raw_cluster_cmd instead of manual "ceph"

It's less repetitive, and I can override it for use with
vstart.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add Filesystem.delete_all

So that we can clear down filesysteems created by
someone other than ourselves (like vstart.sh)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: make Filesystem rados command overridable

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: use raw_cluster_cmd in Filesystem

...instead of manually invoking 'ceph' -- so that
it's easier to override this for vstart.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #595 from ceph/wip-quota

log-whitelist 'reached quota' for librados test.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/radosbench: wait just a bit longer

A run failed due to thrashing.. missed by about 30s (the osd
eventually sent the last reply but we'd already timed out).

Signed-off-by: Sage Weil <sage@redhat.com>

log-whitelist 'reached quota' for librados test.sh

A new test verifies that we are stopped by the pool quota (and get
the right error messages or block).  See ceph.git
32962740ce9211626d310a002b23afeb0d05b500.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #461 from ceph/wip-client-oldest-tid

cephfs/test_client_limits.py: update test_client_oldest_tid

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rados/singleton-nomsgr/all/full-tiering: time out rados -p

We will hit a quota and block (not error out).

See #13283
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #592 from ceph/wip-13000-centos-activate

ceph-deploy: always zap disk before creating an osd

Merge pull request #580 from dachary/wip-13250-ceph-deploy-branch

ceph-deploy: fix ceph-deploy-branch config options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph_manager: %d -> %s

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-deploy: always zap disk before creating an osd

The existing logic is to ceph-deploy osd create --zap-disk which will
zap the data device before preparing it. However it will not zap the
journal device (see http://tracker.ceph.com/issues/13291).

If ceph-deploy osd create fails, a fall back will zap both the data
device and the journal and try prepare again. This could work if
the device preparation and activation was synchronous and catch all
errors that could be caused by an unclean journal device. However,
the activation is asynchronous and it is entirely possible for a device
to be prepared successfully and fail to activate in the background.

The data and journal device are always zapped before calling ceph-deploy
osd create. The logic is simpler and the overhead is low.

http://tracker.ceph.com/issues/13000 Fixes: #13000

Signed-off-by: Loic Dachary <loic@dachary.org>

rados/singleton-nomsgr/all/11429: blackhole osd.0 to close race

Blackhole filestore ops so that we ensure it doesn't complete
the pg deletions before the restart function does a clean shutdown
etc.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/tiering: fix upgrade sequence

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/ceph_manager: fix logging on failed pool property

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #591 from athanatos/wip-13234

rados: add test for 13234.yaml

Reviewed-by: Sage Weil <sage@redhat.com>

rados: add test for 13234.yaml

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #586 from ceph/wip-mdscap

tasks/cephfs: mds allow

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #575 from ceph/wip-32bits-ceph-fuse

suites/fs: test ceph-fuse with 32-bits ino_t

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

buildpackages: implicit for OpenStack

When using the teuthology-openstack command, buildpackages is implicit
and does not need to be included when running the test suite. Move the
stanza to the debug directory as a reminder when debuging.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados/singleton-nomsgr/all/11429: grep osd.0 only

Restart can be slow enough that osd.1 and 2 finish deleting the
pgs.  Verifying one osd sees the instance is sufficient.

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/cephfs: mds allow

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-infernalis'

Merge pull request #583 from dachary/wip-buildpackages-flock

buildpackages: flock must --close

Merge pull request #584 from dachary/wip-buildpackages-centos-6

buildpackages: user-data must be per os-type/os-version

buildpackages: user-data must be per os-type/os-version

CentOS 6.5 needs to install a package and reboot to grow the root file
system. Instead of assuming a common user-data.txt file can fit all
Operating Systems, make one user data per os-type/os-version combination.

Signed-off-by: Loic Dachary <loic@dachary.org>

add cache tiering test for hammer-x

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rados/singleton-nomsgr/all/11439: fix greps

Signed-off-by: Sage Weil <sage@redhat.com>

buildpackages: flock must --close

The process run by flock must not inherit the file descriptor because
this will cause the lock to be held forever should the command survive
the call to flock. This is for instance the case for the ssh-agent.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #581 from dachary/wip-11883-git-url

buildpackages: get Ceph git url via get_ceph_git_url()

buildpackages: get Ceph git url via get_ceph_git_url()

Instead of relying on git_base_url, use the get_ceph_git_url() to obtain
the URL of the Ceph repository to use with git clone. This allows the
user to override it via the git_ceph_url configuration option and the
--git-ceph-url command line option to teuthology-openstack.

http://tracker.ceph.com/issues/11883 Refs: #11883

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-deploy: fix ceph-deploy-branch config options

The config paramter of download_ceph_deploy does not have a ceph-deploy
item, therefore the ceph-deploy-branch parameter is always assumed to be
master.

Signed-off-by: Loic Dachary <loic@dachary.org>

upgrade/hammer-x/parallel: set infernalis settings after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

releases/infernalis.yaml: final steps after infernalis upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/divergent_priors2: wait for osd to start before tell

Otherwise we can get

2015-09-24T19:22:15.191 INFO:teuthology.orchestra.run.mira080.stderr:Error ENXIO: problem getting command descriptions from osd.1

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr/all/11429: more reliably create orphan pgs

Make sure osds have the latest map with a pg ls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #574 from ceph/wip-13098

rados/singleton-nomsgr/all/full-tiering: test full cache tier

rados/singleton-nomsgr/all/full-tiering: test full cache tier

Verify fix for #13098

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #577 from ceph/wip-get-prop

tolerate pg_num get failure

tasks/ceph_manager: ignore failure getting pg_num

Otherwise, we may fail while racing with a workload that deletes a pool:

2015-09-23T15:01:52.855 INFO:tasks.workunit.client.1.vpm128.stdout:[ RUN      ] LibRadosTwoPoolsPP.PromoteSnapTrimRace
2015-09-23T15:01:53.892 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw pg_num'
2015-09-23T15:01:54.206 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.gc pg_num'
2015-09-23T15:01:54.462 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users.uid pg_num'
2015-09-23T15:01:54.696 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users.email pg_num'
2015-09-23T15:01:55.006 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .users pg_num'
2015-09-23T15:01:55.296 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.buckets.index pg_num'
2015-09-23T15:01:55.523 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .log pg_num'
2015-09-23T15:01:55.752 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .usage pg_num'
2015-09-23T15:01:56.188 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get .rgw.buckets.extra pg_num'
2015-09-23T15:01:56.625 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get test-rados-api-vpm128-17360-6 pg_num'
2015-09-23T15:01:56.928 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get test-rados-api-vpm128-17360-13 pg_num'
2015-09-23T15:01:57.193 INFO:teuthology.orchestra.run.vpm176.stderr:Error ENOENT: unrecognized pool 'test-rados-api-vpm128-17360-13'
2015-09-23T15:01:57.206 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #576 from ceph/wip-13031-packages

buildpackages: make rpm packages

Reviewed-by: Joe Julian <jjulian@io.com>

buildpackages: make rpm packages

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #573 from ceph/wip-hammer-x

hammer-x fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

upgrade/hammer-x/stress-split-erasure-code-x86_64: specify arch

Signed-off-by: Sage Weil <sage@redhat.com>

suites/fs: test ceph-fuse with 32-bits ino_t

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #571 from dachary/wip-isa-arch

erasure-code: isolate isa plugin arch constraint

upgrade/hammer-x: use supported distros, not hard-coded rhel7+trusty

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: use rgw civetweb

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/parallel: clients are hammer to for final workload

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/point-to-point: fix x client

- install hammer for hammer->x upgrade
- use hammer branch for workunit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #453 from ceph/wip-11835

tasks/cephfs: time out on ceph-fuses that don't die

Reviewed-by: Greg Farnum <gfarnum@redhat.com>