tasks/cephfs: fix wait in test_volume_client

Signed-off-by: John Spray <john.spray@redhat.com>

suites: add an extra client to fs/recovery

The volumeclient test wants 3, and the rest of the tests
can just ignore it.

Signed-off-by: John Spray <john.spray@redhat.com>

suites: add yaml for TestVolumeClient

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_volume_client

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix fuse killing in vstart

The "ps au" output was truncating lines which could cause
processes to get missed in some cases, like where there
is a long --client-mountpoint argument.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: write_n_mb optionally background

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: reset IDs of mounts between tests

because volumeclient test twiddles the IDs to
match the auth identities it creates.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: clear extra auth ids on test start

Because new volumeclient tests will be creating IDs
instead of modifying the default ones.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test case auth_list helper

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #743 from ceph/wip-root-metadata

tasks/cephfs: add TestSessionMap.test_session_reject

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #611 from ceph/wip-libcephfs-acl

suites/fs: test ceph-fuse with libcephfs permission check

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #796 from ceph/wip-fix-sumblink-distros

Fixed symblink for os configs

Fixed symblink for os configs

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #754 from ceph/wip_add_centos71_master

Removed CentOS 7.1 and added CentOS 7.2 instead

Change supported ditros to use latest CentOS 7.2

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #793 from ceph/wip-centos-71-72

suites/teuthology: Add CentOS 7.1 and 7.2

suites/teuthology: Add CentOS 7.1 and 7.2

Signed-off-by: Zack Cerza <zack@redhat.com>

suites/fs: update log whitelist for test_session_reject

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: generate mount paths on the fly

...because new tests that modify the client ID will
get confused about whether they're mounted or not, otherwise.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: support mount_path in Mount subclasses

For e.g. volumeclient guests that want to mount
a subpath.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #785 from ceph/wip-14257

tasks/cephfs: fix wait_for_state timing

tasks/cephfs: fix wait_for_state timing

Return actual elapsed wallclock time instead of
the number of times we polled.

Fixes: #14257
Signed-off-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Merge pull request #780 from ceph/jewel-14196

tasks/cephfs: fix race in test_damage

Merge remote-tracking branch 'origin/jewel-14196'

Merge remote-tracking branch 'origin/wip-13583'

tasks/cephfs: fix wait in assert_cluster_log

I had the monc tick interval wrong.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fixes to FuseMount

This was getting stressed in new ways by
TestSessionMap.test_session_reject, which
has a mount that fails initially.

Two changes here:
 * Raise CommandFailedError instead of RuntimeError when
   a mount fails (i.e. catch process termination instead
   of timing out on /sys/ population)
 * Generalise error handling on umount, so that we only
   raise the exception on an umount failure if the mount
   appears to really not be unmounted.  There is some
   EINVAL corner case that was getting triggered by the test.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add TestSessionMap.test_session_reject

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add assert_cluster_log

For capturing the cluster log and asserting
that we see a particular expected message.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: add run_ceph_w to CephManager

Analogous to raw_cluster_command, but instead
of calling blocking CLI command we're invoking
the -w mode.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: new hooks for auth keys and config

Used when configuring clients with dynamically
generated auth keys, and pointing them at mount paths.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #775 from ceph/wip-stat-error

repair_test: Add whitelist item for new stat error message

Merge pull request #781 from ceph/wip_remove_dumpling

Removed tests for EOL dumpling clients on jewel clusters

Removed tests for EOL dumpling clients on jewel clusters

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/cephfs: fix race in test_damage

When stray directory inodes are corrupted, MDS may go to damaged state
after becoming active. (MDCache::open_root/populate_mydir is called by
MDSRank::starting_done).

Fixes: #14196
Signed-off-by: Yan, Zheng <zyan@redhat.com>

tasks/osd_recovery: fewer objects in test_incomplete_pgs

Otherwise we risk filling up a (fast) disk.

Signed-off-by: Sage Weil <sage@redhat.com>

rados: 5m radosbench (not 10m)

Signed-off-by: Sage Weil <sage@redhat.com>

rados: run radosbench for 10m (not 15m)

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton/all/pg-removal-interruption: ensure osd.0 is marked down

We were waiting for teh dameon to fail, not for the mons to notice,
which meant that after we restart the daemon the next tell could
try to talk to the old osd.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: increase length of non-valgrind fsx workloads

New lab machines make these run much faster. These should take ~1-1.5 hours now.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: add helgrind to valgrind subsuite

This checks locking problems. This can be a separate facet for
workunits, but for now it requires a manual 2nd configuration for
rbd_fsx.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: add a couple new workunits

for rbd-nbd and the journalling cli

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: add some workloads with journalling enabled

Separating features into its own facet would be good,
but requires changing some of the tests a bit.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rbd: only test on xfs

btrfs is not recommended for production

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #755 from guangyy/wip-13441-test

pg: add test cases for pg list_missing on EC pool

Merge pull request #777 from ceph/wip-radosbench

rados: run radosbench for 15m instead of 30m

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rados: run radosbench for 15m instead of 30m

The smithi disks fill after about 20m.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #774 from ceph/wip-radostool

suites/rados/singleton/all/radostool: whitelist quota error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

suites/rados/singleton/all/radostool: whitelist quota error

The test sets and hits a quota; whitelist the message

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #772 from ceph/wip-14113-jewel

suites/rbd: disable pool validation for pool snaps test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

suites/rbd: disable pool validation for pool snaps test

Fixes: #14113
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #769 from dachary/wip-buildpackages-centos-7.1

buildpackages: support CentOS 7.1 and CentOS 7.2

buildpackages: support CentOS 7.1 and CentOS 7.2

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'gh/jewel'

repair_test: Add whitelist item for new stat error message

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #764 from dachary/wip-buildpackages-log

buildpackages: log the output of make

buildpackages: no need to sudo in bootcmd

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: log the output of make

Signed-off-by: Loic Dachary <loic@dachary.org>

Suites/jewel in the smithi lab will be on centos 7.1

Signed-off-by: yuriw <yuri.weinstein@gmail.com>

Merge pull request #756 from dachary/wip-13624-workunit-git

workunits: fetch from --ceph-git-url

Merge pull request #758 from dreamhost/wip-openstack-dhc

Fix runs on OpenStack clusters like DreamHost Compute

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buildpackages: pkgrepo host also needs user-data setup.

The packages repo host fails in environments where networking setup is
needed in VMs. Use the same user-data as the buildhosts to ensure this
is the case.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

buildpackages: copy teuthology user-data.txt changes

7b27e1db7: openstack: support /etc/network/intefaces injection
2358562cf: ensure VMs always have /etc/hosts set up
4378a505d: always allow unsigned deb packages
50b2db521: openstack: encode instance name with the full IP
6e828a33b: openstack: add 8.8.8.8 as a last resort resolver

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

buildpackages: catch instance creation failure.

Split the sleep from the server creation, so we catch 'server create'
failures (eg due to quota):
> Quota exceeded for cores: Requested 16, but already used 10 of 20 cores
> (HTTP 403) (Request-ID: req-6467934e-db50-4479-995c-4d44dedf553a)

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

buildpackages: Be robust about addresses

OpenStack could tell us the VM has multiple networks, and offers no
guarantee about the order of addresses either (the old code failed if
the v4 IP was first).

For now, take the first listed network, and the first listed IPv4
address therein. Comments contain more detailed examples of possible
output from openstack tool.

Also remove the need for using jq to parse the output.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #757 from dachary/wip-dummy

Revert "dummy: only use the user playbook"

Revert "dummy: only use the user playbook"

This reverts commit a016f29e211e378a50fd833d940abbb49449c37b.

Merge pull request #744 from ceph/wip-inode-claim

tasks/cephfs: add TestJournalRepair.test_table_tool_take_inos

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #703 from ceph/wip-layout-restrict

tasks/cephfs: test for auth caps pool-setting restriction

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #728 from ceph/wip-snap-strays

tasks/cephfs: cover snaps in test_strays

Merge pull request #709 from ceph/wip-scrub-basic

tasks/cephfs: add TestForwardScrub

workunit: fetch from --ceph-git-url

The commit from which workunits are fetched must be retrieved
from --ceph-git-url via teuth_config.get_ceph_git_url() instead of
assuming it is available via git://git.ceph.com/ceph.git.

Using git://git.ceph.com/ceph.git is convenient because it supports git
archive. In the general case, some git servers such as github do not
support git archive and a full git clone must be done instead.

Although it would be possible to

    git clone --branch=master --depth=1 --single-branch

to reduce the amount of data being retrieved, it would require a

    git fetch origin SHA1

but git version >= 1.7 do not support fetching a commit.

http://tracker.ceph.com/issues/13624 Fixes: #13624

Signed-off-by: Loic Dachary <loic@dachary.org>

workunit: keep the path to the list of tests in a variable

Signed-off-by: Loic Dachary <loic@dachary.org>

workunit: allow tag to override sha1

The sha1 for the workunit task is always set by the suite.py task. The
tag must be checked before the sha1 othewise it cannot be used to
override the sha1.

Signed-off-by: Loic Dachary <loic@dachary.org>

pg: add test cases for pg list_missing on EC pool

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>

Removed CentOS 7.0 and added CentOS 7.1.1503 instead

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/cephfs: avoid using bare remote.run in scrub_checks

Use the Mount.* wrappers for filesystem operations,
so that changes like making run_shell use sudo just work.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: update for run_shell vs. sudo

(TestStrays.test_snapshot_remove blocked on the sudo change)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: always sudo in run_shell

This was causing permissions issues when
running inside teuthology, as run_python
was using sudo and run_shell wasn't.

Would be nice to get rid of all the rootishness,
but for the moment just make it more uniform.

tasks/cephfs: cover snaps in test_strays

Signed-off-by: John Spray <john.spray@redhat.com>

suites: add yaml for TestForwardScrub

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add TestForwardScrub

This tests the forward scrub's ability to traverse
some metadata and tag it, and the corresponding
functionality in cephfs-data-scan to filter based
on tag and inject orphaned items.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #737 from ceph/wip-enable-data-scan

suites: yaml for TestDataScan

Merge pull request #751 from dachary/wip-teuthology-integration

teuthology-integration: run teuthology integration tests

teuthology-integration: run teuthology integration tests

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #664 from dachary/wip-nop

nop: suite that does not even lock targets

Merge pull request #748 from dachary/wip-13910-buildpackages-images

buildpackages: create the image if it does not exist

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

buildpackages: create the image if it does not exist

Since buildpackages runs before target provisioning, it is possible that
the desired image does not yet exist on a newly provisionned tenant (or
region).

http://tracker.ceph.com/issues/13910 Fixes: #13910

Signed-off-by: Loic Dachary <loic@dachary.org>

suites: yaml for TestDataScan

This was an oversight.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #679 from ceph/wip-ceph-whitespace

tasks/ceph: fix up whitespace

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/cephfs: add TestJournalRepair.test_table_tool_take_inos

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #740 from dachary/wip-13899-rbd-fio

task/rbd_fio: allow for unsigned packages

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

Merge pull request #742 from dachary/wip-openstack-morepggrow

openstack: rados/.../morepggrow.yaml may need more disk

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

openstack: rados/.../morepggrow.yaml may need more disk

Signed-off-by: Loic Dachary <loic@dachary.org>

task/rbd_fio: allow for unsigned packages

Similar to what the teuthology install.py task does, add --force-yes to
the apt-get install so that unsigned packages are successfully
installed. It is needed when the buildpackages task is used to create
packages on the fly.

There is no need to do the same for rpm packages because the
verification is controlled from the ceph-release package instead of from
the command line.

http://tracker.ceph.com/issues/13899 Fixes: #13899

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #736 from dachary/wip-buildpackage-delete-wait

buildpackages: wait for the build machine deletion

Merge pull request #738 from dachary/wip-openstack-debug

openstack: add 15GB for debugging purposes

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>