cephfs: test readahead is working

Test case for: https://github.com/ceph/ceph/pull/9374
Issue: http://tracker.ceph.com/issues/16024

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Merge remote-tracking branch 'upstream/pull/1045/head'

* upstream/pull/1045/head:
  tasks/cephfs: fix vstart runner _get_pid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tasks/cephfs: fix vstart runner _get_pid

Dropping the 'x' caused the ps lines not to include
the "-i <myid>" part of the command line that we use
to find named daemons' pids.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1043 from batrick/ps-uid

vstart_runner: only kill/look at user's processes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #1044 from batrick/info-placement

vstart_runner: move info message for loaded mods

Reviewed-by: John Spray <john.spray@redhat.com>

vstart_runner: move info message for loaded mods

The message was printing loaded modules before loading each module.

vstart_runner: only kill/look at user's processes

This prevents vstart_runner.py from trying to kill other user's
ceph-mds/ceph-fuse instances.

Merge pull request #1038 from ceph/wip-libcephfs-posixacl

suites/fs: fix config for enabling libcephfs posix ACL

Merge pull request #1039 from ceph/wip-16113

suites/rados: add test for 16113

suites/rados: add test for 16113

Signed-off-by: Samuel Just <sjust@redhat.com>

suites/fs: fix config for enabling libcephfs posix ACL

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #1031 from ceph/wip-sleep

tasks: fix non-existent sleep function

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks: fix non-existent sleep function

CephManager has no sleep function. Use time.sleep() instead.

Ran into this while testing a branch. Apparently it doesn't happen
much since this hasn't changed in years, but the error was copied
into several tasks.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1028 from dachary/wip-rbd

rbd: provision volumes to format as XFS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1014 from ceph/wip-ceph-restart-wusui

tasks/ceph.restart osd fix

Reviewed-by: Samuel Just <sjust@redhat.com>

rbd: provision volumes to format as XFS

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1026 from ceph/wip-16064

scrub_test: Shards are not marked in error when osd guesses

Reviewed-by: Samuel Just <sjust@redhat.com>

scrub_test: Shards are not marked in error when osd guesses

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #1017 from dachary/wip-buildpackages

openstack: ovh renamed flavors : s/eg/hg/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

openstack: ovh renamed flavors : s/eg/hg/

http://travaux.ovh.net/?do=details&id=17532

Signed-off-by: Loic Dachary <loic@dachary.org>

tasks/ceph.restart osd fix

ceph.restart should mark restarted osds down in order to avoid a
race condition with ceph_manager.wait_for_clean

Fixes: http://tracker.ceph.com/issues/15778
Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1013 from ceph/wip-15900

tasks/cephfs: fix failure in test_mount_conn_close

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: find processes correctly

Don't mistake the daemon-helper line for the actual
daemon process.

Fixes: http://tracker.ceph.com/issues/15900
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix mount wait in test_mount_conn_close

Fixes: http://tracker.ceph.com/issues/15900
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #983 from ajarr/wip-15614

tasks/cephfs: test read-only authorization for volumes

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: test read-only authorization for volumes

Test whether mount access to volumes can be restricted to read-only.

Fixes: http://tracker.ceph.com/issues/15614
Signed-off-by: Ramana Raja <rraja@redhat.com>

tasks/cephfs: add function to configure guest credentials

In the test_volume_client module, refactor code to add a function
that configures credentials for a guest to mount a volume. The
function could be used by tests to setup a guest client to mount
a volume with fewer lines of code.

Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #1012 from ceph/wip-teuthology-multi-cluster-suite

suites/teuthology/multi-cluster: make them pass again

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

suites/teuthology/multi-cluster: make them pass again

Use xfs to avoid object length limits and be explicit about which
clients to run on, so the workunit git clone doesn't race with another
client on the same machine.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

suites/rbd: added replication stress test workunitReviewed-by: Josh Durgin <jdurgin@redhat.com>

suites/rbd: added replication stress test workunit

suites/rbd: added replication stress test workunit

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1007 from ceph/wip-li-rados

scrub_test: Handle list-inconsistent-obj changes in pull #8983

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1006 from ceph/wip-15754-master

Initial check in for a complete jewel-x suite

Merge pull request #934 from ceph/wip-15266

tasks/cephfs: add TestVolumeClient.test_purge

Reviewed-by: Ramana Raja <rraja@redhat.com>

scrub_test: Handle list-inconsistent-obj changes in pull #8983

Add checking of error in the shards

Signed-off-by: David Zafman <dzafman@redhat.com>

Initial check in for a complete jewel-x suite

Fixes http://tracker.ceph.com/issues/15754
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #974 from ceph/wip-15599

tasks/cephfs: test 'df' output in volumeclient

Reviewed-by: Ramana Raja <rraja@redhat.com>

cron_wrapper: Added echo to log a command name to the log fileReviewed-by: Josh Durgin <jdurgin@redhat.com>

Added echo to log a command name to the log file

Added echo to log a command name to the log file

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1001 from ceph/wip-15563-master2

Added execute permission to the script

Added execute permission to the script

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #979 from ceph/wip-mon-rocksdb

rados: vary mon kv backend

rados: vary mon kv backend

Use just rocksdb for thrash to avoid blowing up the matrix.  Leave
leveldb for the ec thrash suites.

Signed-off-by: Sage Weil <sage@redhat.com>

make-rpm.sh: support openSUSE

make-rpm.sh: support openSUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

make-rpm.sh: support openSUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #993 from smithfarm/wip-opensuse-user-data

add user-data for openSUSE 42.1

Reviewed-by: Loic Dachary <ldachary@redhat.com>

add user-data for openSUSE 42.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #992 from ceph/wip-14339-master

Added v10.2.0 => jewel upgrade case

Merge pull request #991 from ceph/wip-15857

tasks/cephfs: fix ps usage to not truncate cmds

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: update failover test for standby changesReviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: update failover test for standby changes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #985 from ZhenyuLeng/noneed_op_is_discardable

suites/rados: test osd_discard_disconnected_ops

Test osd_discard_disconnected_ops
Signed-off-by: ZhenyuLeng <lengzhenyu@ict.ac.cn>

Merge pull request #987 from ceph/wip-15563-master

A script to be used in crontab sepia/ovh labs, so emails sent only on errors, logs stored

Reviewed-by: Dan MIck <dmick@redhat.com>

A script to be used in crontab sepia nd ovh lab, so emails sent only on errors

Fixes: http://tracker.ceph.com/issues/15563
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Added v10.2.0 => jewel upgrade case

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/cephfs: test 'df' output in volumeclient

Fixes: http://tracker.ceph.com/issues/15599
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix ps usage to not truncate cmds

Now that daemon commandlines include "--cluster" they
easily overflow ps's default behaviour.

Fixes: http://tracker.ceph.com/issues/15857
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #990 from ceph/wip-multi-cluster-fixes

fixes for some issues caused by the multi-cluster branch

tasks: fix {testdir}/data paths

The multi-cluster branch updated these to have a '$cluster.' prefix

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph.healthy: allow None as config

This is used by the pg-removal-interruption test.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/resolve_stuck_peering: fix whitespace and indentation

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/resolve_stuck_peering: adjust to new ctx.manager location

This task was re-added after the multi-cluster branch merged.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #986 from ceph/wip-crush-drop-legacy

rados: test jewel instead of legacy tunables

Reviewed-by: Samuel Just <sjust@redhat.com>

tasks/cephfs: add TestVolumeClient.test_purge

For reproducing error handling paths during purge,
such as #15266 (ASCII exception)

Fixes: http://tracker.ceph.com/issues/15266
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #975 from xiaoxichen/prefix

fs/volume_client: exercise the configurable prefix and ns_prefix.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #968 from ajarr/wip-15045

tasks/cephfs: test volume client eviction

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #989 from ceph/revert-981-revert-955-tracker13531

Revert "Revert "Rados: Tracker-13531-resolve stuck peering""

Reviewed-by: Samuel Just <sjust@redhat.com>

Revert "Revert "Rados: Tracker-13531-resolve stuck peering""

Merge pull request #937 from ceph/wip-multi-cluster

multi-cluster support and basic rbd mirroring test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tasks/create_verify_lfn_objects: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

suites: add multi-cluster tests to the teuthology suite

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks: update ctx.ceph.conf readers to use per-cluster conf

Just use the default cluster name for now, since these tasks aren't used
for multi-cluster tests yet.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

suites/rbd: add basic mirroring suite

Just one workunit to run so far.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks: add an rbd-mirror task

This simply runs the rbd-mirror daemon. Nothing special is needed
since the runtime files (logs, asok, etc) go to the standard ceph
locations.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: ignore EEXIST for the archive data dir creation

With multiple clusters this will be called multiple times.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: pull each mon dir only once

No need to pull all mon dirs for the host for each mon on the host.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: only run ceph_log and valgrind_post once

These setup and parse logs on all hosts, so they should be run only
for the first cluster setup. This cluster will be torn down last, so
the cleanup happens after all clusters are shutdown as well.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: make scrubbing cluster-aware

Simplify implementation by using manager and teuthology.misc methods
instead of reinventing them here.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/watch_notify_same_primary: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/thrash_pool_snaps: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/repair_test: clean up manager usage, adjust to new location

Only repair_test_2 actually uses ctx, the rest of the functions only
need the manager.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/reg11184: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/radosbench: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/rados: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/populate_rbd_pool: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/peering_speed_test: adust for new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/osd_failsafe_enospc: adjust for new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/divergent_priors*: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph_objectstore_tool: use existing manager

This is only used after the ceph task, which already createds a
manager for the cluster. Use the default 'ceph' cluster manager for
now.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph_manager: make utility_task cluster-aware

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: make restart subtask cluster-aware

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: make wait_for_mon_quorum cluster-aware

Accept a 'daemons' list like other ceph subtasks, so it can get an
optional 'cluster' setting too.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: make wait_for_osds_up cluster-aware

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph: update ctx.manager usage to ctx.managers

Not sure this function is ever used (no users in ceph-qa-suite yamls
or tasks.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/workunit: work with roles that include cluster

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks: move find_remote to util, rename and add helper

This is a useful for any cluster-aware task.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/blktrace: make cluster-aware

Need to adjust to lookup devices by role rather than id, so might as
well add a cluster config option while we're at it. Remove the
confusing default config that is not actually used.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph_manager: make Thrasher cluster-aware

Just a few spots need to know to lookup only osds in this cluster, or
prefix a filename with the cluster. Use CephManager.find_remote() to
avoid a bunch of repetition and look only in the intended cluster.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph_manager: make mount_osd_data() cluster-aware

Use a cluster-specific mount point, and address osds by full role,
rather than just id, in the ctx.disk_config structures.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tasks/ceph_manager: add cluster param to write_conf()

Only used by cephfs right now, so don't bother changing callers.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>