tasks/cephfs: set norequire_active_mds on kclient

The kernel client's cluster availability test is
more primitive than the fuse client, so we need
to switch it off to avoid client mounts failing
while MDSs are still coming up.

Fixes: http://tracker.ceph.com/issues/18161
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: use kill in test_reconnect_eviction

Previously relied on client being able to unmount
while the MDS was offline, which is not necessarily
so.  Use kill instead.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: disable broken python tests

Disabling strays because of:
http://tracker.ceph.com/issues/17193

Disabling test_failover because of:
http://tracker.ceph.com/issues/18161

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: make test_journal_repair work with more daemons

Previously this assumed it was running with exactly two MDS
daemons.  When there were more, it would fail to execute
"fs reset" because the extra daemons were active in
the map.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: run python tests with 4 MDSs

For the benefit of test_failover

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: skip volumeclient client test on kcephfs

This relies on quota-ish stuff that doesn't exist
in kclient.  We can still run the outer part
of the test though.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix unmount call in volumeclient test

Should have been to umount_wait, not umount (i.e.
the blocking foreground version).  This happened
to matter because umount_wait is more tolerant
of being called if the client is not already mounted.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: special case for kcephfs behaviour

It does not generate EIOs in all the cases that
the fuse client does.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: skip TestVolumeClient.test_evict on kcephfs

This test was probably buggy and only happened to work
with ceph-fuse, because it expects the MDS to immediately
respond to updates to the client's auth caps, but that
doesn't happen.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: disable test_full for kcephfs

Until http://tracker.ceph.com/issues/17204 is done.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix race in TestStrays.test_dirfrag_limit

With the kernel client, this was proceeding to attempt
a split before the directory had persisted, because
there was no fsync after creating it.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: remove task for cap flush test

This test skips all if not running against
fuse.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: skip test_client_abort on kclient

This tests a fuse-only feature, ticket for adding
it in kclient is:
http://tracker.ceph.com/issues/17805

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: support multi-fs in kernel client

Change the Mount interface to take it as an
argument to mount() instead of setting it
out of band in a config file as we used to
for the fuse client.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix eviction test for volumeclient

It was not correct to expect a client to block after
eviction unless it was also deauthorised.  I guess
this was working with fuse because fuse does a less
good job at re-establishing a session than the kclient?

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: update test_journal_repair for kclient

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph: handle down nodes in log rotation

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: skip ENOSPC testing on kclient

Ticket #17204 exists to revive the old patchset, until then
we skip the tests.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: put clients on dedicated nodes

Some tests want to use power cycling to reset stuck
mounts, and that needs to not kill OSDs as collateral
damage.

Need to revisit this to avoid unnecessarily using a whole
node for the client for those tests that don't require it
(i.e. those that don't use CephFSTestCase.REQUIRE_KCLIENT_REMOTE)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add a skip for non-fuse

Signed-off-by: John Spray <john.spray@redhat.com>

suites/fs: s/yam/yaml/

Oops!

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: s/yam/yaml/

Oops!

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: tolerate skipping tests

With fuse client we expect everything to be
runnable so treat SkipTest as a failure because
it indicates something bad in the config.  With kclient
we known we have to skip some tests, so tolerate it.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: add TestFailover

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/kclient: unmount conditionally

Don't assume the workload left the mount up and
running.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: skip test_session_reject cleanly

Instead of asserting in configure_auth (which in fact
works fine with KernelMount.write_secret_file), raise
a SkipTest in test_session_reject (because the kernel
client cannot handle the client_metadata setting to
inject bogus data)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: make KernelMount.umount_wait idempotent

Some places (notably CephFSTestCase.setUp) don't check
for mountedness before calling this.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/kcephfs: add python tests

Let's see which of these is happy running with a kernel client!

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1282 from batrick/fs-reorg

fs: unify common parts of sub-suites

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1275 from jcsp/wip-17853

Directory fragmentation tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tasks/cephfs: add test_standby_for_invalid_fscid

Signed-off-by: Ramakrishnan Periyasamy <rperiyas@redhat.com>

Merge pull request #532 from ceph/wip-12378

Adds task to wrap the aver utility

Merge pull request #734 from dachary/wip-rados-thrash

openstack: rados/thrash...cache-pool-snaps.yaml: need 15GB

Merge pull request #1019 from ceph/wip-rgw-fix-longpath

rgw: split osd's in 2 nodes

Merge pull request #980 from ceph/wip-fix-infinit

Fix infinite wait during monitor quorum check

Fix infinite wait during monitor quorum check

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Split osd's into 2 nodes

http://tracker.ceph.com/issues/15612
Not all test machines(mira, vps) have 4 osd's and without this
change it will fail on nodes with < 4 osds, since one of the
osd will be on ext4 and it will hit long filename issue

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #535 from ceph/wip-12376

tasks/radosbenchsweep task

Merge pull request #1024 from ceph/wip-default-fs

tasks/ceph: default to xfs instead of root dir

Merge pull request #1087 from sponce/master

rados: Added tests for the striper layer of rados

Merge pull request #956 from ceph/wip-rm-ext4

Remove ext4 option from rados:thrash tests

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #1301 from ceph/wip-valgrind-centos

do all valgrind runs on centos

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

do all valgrind runs on centos

The xenial valgrind has a bug with generating random
numbers; see http://tracker.ceph.com/issues/18126

Revert this when it is fixed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1299 from athanatos/wip-enable-hacky-flag

ec-data-pool: enable the hacky_overwrites flag

ec-data-pool: enable the hacky_overwrites flag

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1298 from ceph/wip-pre-single-major-branch

krbd/unmap: pre-single-major is now 3.12.z

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rados/thrash/workloads: shorten rados bench a bit

I hit ENOSPC during thrashing on one run.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1295 from dachary/wip-16506-ceph-coverage

upgrade: ceph-test is needed for ceph-coverage

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

upgrade: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1294 from ceph/wip-17960-master

Removed xenial + Excluded librgw2

upgrade/jewel-x/point-to-point: avoid xenial

Xenial has newer packages in the distro and we want to
avoid fragile install instructions to avoid them.

Signed-off-by: Sage Weil <sage@redhat.com>

Excluded librgw2
Fixes http://tracker.ceph.com/issues/17960

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

krbd/unmap: pre-single-major is now 3.12.z

3.13 won't build on newer distros, due to lack of gcc5 support.
nightly/pre-single-major (with a slash) is now gone.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1293 from ceph/wip-17977-fix

qemu: Set permissions on /dev/kvm

Reviewed-by: Dan Mick <dmick@redhat.com>

qemu: Set permissions on /dev/kvm

Hack until fix in https://bugzilla.redhat.com/show_bug.cgi?id=1333159
makes it into CentOS repos.

Fixes: http://tracker.ceph.com/issues/17977
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1291 from ceph/wip-18085

rados/thrash*: do no set osd_max_backfills=1

Reviewed-by: Samuel Just <sjust@redhat.com>

rados/thrash*: vary osd_max_backfills, always >1

This can lead to a copy-from vs backfill deadlock; see
http://tracker.ceph.com/issues/18085

This effectively reverts 5e880228fd7f59063d22a51eb5488b369b0c8360
which aimed to catch bugs in recovery reservations.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #894 from ceph/wip-bluestore

rados: test bluestore too

rados/objectstore/objectstore.yaml: do not test kstore

It's flaky.

Signed-off-by: Sage Weil <sage@redhat.com>

rados/objectstore/objectstore.yaml: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: test on filestore and bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

fs: test bluestore and filestore

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: test bluestore and filestore

Signed-off-by: Sage Weil <sage@redhat.com>

rados: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1289 from ceph/wip-16397-workaround

suites/knfs: pin to ubuntu

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #1283 from SUSE/wip-18014

thrashosds: try ceph-objectstore-tool for one minute

Reviewed-by: Loic Dachary <ldachary@redhat.com>

suites/knfs: pin to ubuntu

Workaround for http://tracker.ceph.com/issues/16397, which
manifests when we run bleeding edge kernels against the version
of nfs-utils in centos7.

Signed-off-by: John Spray <john.spray@redhat.com>

thrashosds: try ceph-objectstore-tool for 10 minutes

If ceph-objectstore-tool binary is not present, it's likely because we're in
the middle of an upgrade. Do not try to run the binary until we verify that
it's really present. If it is absent, spend up to 10 minutes waiting for it to
appear.

Before this patch there was quite a large window for a race to occur. This
patch doesn't entirely eliminate it, but drastically reduces it.

Fixes: http://tracker.ceph.com/issues/18014
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1235 from vshankar/wip-rbd-data-pool

suites/rbd: add data pool test cases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1277 from ceph/wip-exclude-upgrade-clients

Excluded 'libcephfs2','libcephfs-devel','libcephfs-dev'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Excluded 'libcephfs2','libcephfs-devel','libcephfs-dev'
Removed firefly and infernalis paths
Removed mds role from hammer (per chat with Jason)
Add "fs: xfs" override

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rbd: add datapool config YAML to rbd suite

- cli
- librbd
- qemu

Signed-off-by: Venky Shankar <vshankar@redhat.com>

suites/fs: link to common tasks

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common fs

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common overrides

Goal is to reduce arbitrary differences between fs suites.

Note:
  o fs/{multiclient,multifs,recovery,snaps,standbyreplay} now use basic/overrides/whitelist_wrongly_marked_down.yaml.
  o fs/{snaps,standbyreplay} now use basic/overrides/debug.yaml.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common (fuse) mount

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common cluster layout

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify install/start task creation

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

verify: remove old debugging addition

Added in 7ae1aefa to help resolve http://tracker.ceph.com/issues/1737.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: add ceph-fuse disable override

Some tests require fuse mounts to not be unmounted. This avoids the need
to manually specify clients as unmounted.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1262 from batrick/cephfs-delete-bugfix

cephfs: get new fsmap after marking clusters down

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_fragment

Fixes: http://tracker.ceph.com/issues/17853
Signed-off-by: John Spray <john.spray@redhat.com>

tasks: mark start and end of tests in clog

This makes it much easier to associate e.g. slow
requests messages with which test was running.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: reinstate ctx.filesystem

This got lost in the multi-filesystem
refactor.  It's just a debug convenience.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix wait_until_true logging

The success log message was in the wrong
place (and unreachable)

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1274 from ceph/wip-16064

Improve scrub analysis, list-inconsistent-obj output and osd-scrub-repair test

Merge pull request #1260 from ceph/wip-17902-master

Added excluded packages "'libcephfs-devel','libcephfs-dev'"

Merge pull request #1227 from athanatos/wip-ec-overwrites

ec overwrites tests (and bonus fix to fast reads)

Merge pull request #1271 from SUSE/wip_flake8_F401_time_imported_but_unused

tasks.rgw: 'time' imported but unused

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks.rgw: 'time' imported but unused

flake8 was failing.

Signed-off-by: Owen Synge <osynge@suse.com>

Added excluded packadges "'libcephfs-devel','libcephfs-dev'"

Fixes http://tracker.ceph.com/issues/17902
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/scrub_test.py: Make test deterministic by updating digests

ceph_test_rados: Get coverage of digest updates in deep-scrub/repair

Signed-off-by: David Zafman <dzafman@redhat.com>

repair_test, scrub_test: Fix whitelists for scrub changes

Signed-off-by: David Zafman <dzafman@redhat.com>

scrub_test: Fix for list-inconsistent-obj output changes

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #1269 from ceph/wip-rgw-sync-agent-retry

rgw: fix some races with radosgw and radosgw-agent startup

(verified by Orit)

rgw: remove unnecessary sleeps

remove the sleeps that were added to address radosgw startup races

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start_rgw() polls gateway until it accepts connections

resolves various races between radosgw startup and further operations -
both within the rgw task itself (such as the 'radosgw-admin realm pull'),
and in later tasks

Fixes: http://tracker.ceph.com/issues/17794
Fixes: http://tracker.ceph.com/issues/17872
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add retry/backoff to sync agent requests

resolves an issue where startup of the radosgw-agent races with the
requests we send to it to run sync. uses the requests package with
urllib3 to add retry with backoff to these requests

Fixes: http://tracker.ceph.com/issues/16129
Signed-off-by: Casey Bodley <cbodley@redhat.com>

objectstore.yaml: raise fd limit

Signed-off-by: Samuel Just <sjust@redhat.com>

fast.yaml: overrides not override

Signed-off-by: Samuel Just <sjust@redhat.com>