tasks/cephfs: tweak use of mon for admin commands

... s/mon_remote/admin_remote/ and allow caller to pass
in which remote they want to use for that.  Enables use
with ceph_deploy task which does not give admin keys
to mons.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph: refactor legacy FS configuration check

Move up into Filesystem so that this can be used from
the ceph_deploy task as well.

Signed-off-by: John Spray <john.spray@redhat.com>

suites: whitelist log messages for auto-repair test

Log messages being checked now since #11177
was fixed.

Signed-off-by: John Spray <john.spray@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

tasks: implement TestStrays.test_mv_hardlink_cleanup

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: update test_journal_repair

This broke with recent Client changes that
do better caching of readdir results, such
that doing an ls twice is no longer sufficient
to see a fresh result after repair - we need
to remount instead.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/fs: clean up whitespace in YAML

Signed-off-by: John Spray <john.spray@redhat.com>

suites: run fs/recovery with two MDSs

Some tests need both, and others should
ignore it.

Signed-off-by: John Spray <john.spray@redhat.com>

suites: add .yamls for the recent cephfs tests

Signed-off-by: John Spray <john.spray@redhat.com>

suites: put fuse clients on separate hosts

To mitigate the weirdness when we start
trying to abort /sys/fs/fuse/connections/*

Signed-off-by: John Spray <john.spray@redhat.com>

suites: update tasks to use cephfs_test_runner

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: update journal_repair test for 'damaged' state

To track recent change in master where instead of
crashing on missing MDSTable object we'll go
into damaged state.

Instead of catching a crash, handle the rank's
transition to the damanged state.  Leave the crash
handling code (unused for the moment) in the
Filesystem class in case it's needed elsewhere
soon.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_strays

This tests the new purge file/ops throttling
in the MDS, via the new perf counters for
strays/purging.

Fixes: #10390
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_sessionmap

Tests for the persistence behaviour of SessionMap.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: generalise cephfs test runner

...to avoid having boilerplate in each test module,
and gain the ability to run them all in one go
with a nice test-by-test pass/fail report.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph_fuse: populate ctx.mounts earlier

...so that if an error happens during mount, I can
use the interactive console to access ctx.mounts.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mds_flush: be more careful monitoring stats

Were previously taking the baseline from just after the
client did a delete, which was racy: should have taken
it from before, to get a steady state.

Also update the perf dump calls to take advantage of
the new filtering syntax.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: lots of s/mds_restart/mds_fail_restart/

Wherever we are subsequently waiting for daemons
to be healthy, we should be doing a fail during the restart.

Also catch some places that were doing this longhand and use
the handy fail_restart version instead.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: better multiple-mds handling

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph_manager: DRY in mds_status

Signed-off-by: John Spray <john.spray@redhat.com>

ceph_manager: fix bad type assertions

In python, isinstance(foo, str) will fail if
a unicode string is passed in.  The correct check
is basestring.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: be tolerant of multiple MDSs

...as long as only one is active, all the ops
that default to talking to a single MDS should
be happy to talk to the active MDS, even if there
happens to be a standby lying around too.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #395 from fullerdj/wip-rbd-xfstests-201504

RBD: add YAML variables to override locations for ceph-qa-chef and xfstests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #399 from ceph/wip-11043

Removed per #11070 and resolves #11043

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed per #11070 and resolves #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

RBD: added optional YAML parameters to test xfstests from different repos

These variables are needed because ceph-qa-suite bootstraps ceph-qa-chef via
http download of solo-from/scratch/run. This adds a variable to override the
default script. It also adds variables to the rbd task to override the versions
of run_xfstests_krbd.sh and run_xfstests.sh downloaded by the default task.

variables added
======
tasks:
-chef
  script_url: # override default location for solo-from-scratch for Chef
  chef_repo: # override default Chef repo used by solo-from-scratch
  chef_branch: # to choose a different git upstream branch for ceph-qa-chef
-rbd.xfstests:
  client.0:
   xfstests_branch: # to choose a different git upstream branch for xfstests
   xfstests_url: # override git base URL for run_xfstests{_krbd}.sh

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Increased default test RBD size to 10G to help tests pass

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #367 from dachary/wip-ec-cache-agent

add erasure code and cache agent upgrade tests

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>

erasure-code: ec-cache-agent in firefly-x/stress-split-erasure-code

Immediately after the Firefly installation, create an erasure code pool
behind a replicated cache pool. Run deep-scrub on all OSD while a rados
task runs. After upgrading half of the cluster (MON and OSD), run a
rados task again also deep-scrub in parallel.

http://tracker.ceph.com/issues/11054 Fixes: #11054

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: rename firefly-x/stress-split-erasure-code

Rename sub-directories of firefly-x/stress-split-erasure-code make room
for a workload immediately after the installation, i.e. number 2.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados: explain that the task is asynchronous by default

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #393 from ceph/revert-385-wip-mkfs

Revert "ceph: be less weird about passing -f to mkfs"

Revert "ceph: be less weird about passing -f to mkfs"

Fixed #11306 Whitelist WRN "failed to encode map"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 9b9d27b25ee3f75fedc5db788bbe87b93d517f0e)

Merge pull request #388 from athanatos/wip-wrongly

rados_python: whitelist wrongly marked me down

Merge pull request #387 from athanatos/wip-scrub-repair-test

(scrub|rados)_test: tolerate best guess digest errors as well

Merge pull request #386 from athanatos/wip-11156

rados/thrash*: make scrubs happen a lot

Merge pull request #385 from ceph/wip-mkfs

ceph: be less weird about passing -f to mkfs

ceph: fix mkfs -f bug

Pass -f by default to btrfs instead of first trying without and *then*
trying with.

Among other things, this avoids a confusing failure where we try mkfs.ext4
device (no -f), fail for some reason, and then try again with -f and get
a usage error (-f does not mean force for mke2fs).

Signed-off-by: Sage Weil <sage@redhat.com>

rados_python: whitelist wrongly marked me down

Signed-off-by: Samuel Just <sjust@redhat.com>

(scrub|rados)_test: tolerate best guess digest errors as well

Signed-off-by: Samuel Just <sjust@redhat.com>

rados/thrash*: make scrubs happen a lot

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #380 from ceph/wip-11204

Make sure that ulimits are adjusted for ceph-objectstore-tool

Merge pull request #382 from dmick/master

calamari_setup: der.  Use dict.update() correctly

calamari_setup: der.  Use dict.update() correctly

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #381 from dmick/master

calamari_setup fixes

Merge remote-tracking branch 'gh/hammer'

Conflicts:
.gitignore

calamari_setup: Require test_image to be set

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: centralize config defaults

Make a DEFAULTS dict that is updated by any user parms, so that
defaults are documented centrally and so config.get(key, defval) is
no longer necessary everywhere.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: remove "build test image" code; add 'test_image' cfgvar

Stop trying to build test images inside this test; presume the test
image is available built externally (in a file path or an http URL).
Config vars ice_tool_dir, ice_version, iceball_location, and
ice_git_location go away in favor of 'test_image', the path to the
testable image (which can still be a tar.gz or an .iso).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: mounting iso on older distros requires -o loop

Ubuntu's mount/kernel support "mount <file> <mntpnt>" directly;
apparently Centos 6 (and presumably RHEL6) require specifying at
least '-o loop' (a /dev/loopN will be dynamically allocated and removed
on unmount).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #379 from ceph/wip-wn

fix watch-notify test

tasks/watch_notify_same_primary: wait for watch before notify

Make sure watch is done registering and ready before sending the
notifies.

Fixes: #10634
Signed-off-by: Sage Weil <sage@redhat.com>

Make sure that ulimits are adjusted for ceph-objectstore-tool

Fixes: #11204
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #378 from dachary/wip-11221-erasure-code-not-parallel

erasure-code: enable ec-rados-default.yaml

erasure-code: enable ec-rados-default.yaml

The ec-rados-default.yaml started with:

  workload:
    sequential:

which is only suitable for suites/upgrade/giant-x/parallel/2-workload/sequential_run/ec-rados-default.yaml

because suites/upgrade/giant-x/parallel/1-giant-install/giant.yaml has

   - parallel:
      - workload
      - upgrade-sequence

The same file was included in contexts where the parallel task was not
used and the workload did not run:

  ./suites/upgrade/firefly-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code-x86_64/5-workload/ec-rados-default.yaml

The ec-rados-default.yaml is modified to be a task instead of a
sequential task in a parallel tasks. The ec-rados-sequential.yaml is
added and is linked in
suites/upgrade/giant-x/parallel/2-workload/sequential_run instead of ec-rados-default.yaml.

http://tracker.ceph.com/issues/11221 Fixes: #11221

Signed-off-by: Loic Dachary <loic@dachary.org>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 97e6d808f086834f2350084d5de36be285ad2bde)

Merge pull request #377 from ceph/wip-11166

Fixes #11166, whitelisted 'Missing health data for MDS'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #11166, whitelisted 'Missing health data for MDS'

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #375 from dachary/wip-rgw-regional-summary

ensure summary is looked for the user we need (part 2)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #374 from ceph/wip-11139

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Fixes: #11139
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #373 from ceph/wip-10861

fs: fix up dd testing again

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #371 from ceph/wip-iso-style

Add support for ISOs to tasks/calamari_setup

fs: fix up dd testing again

We need to change into the directory during the same shell session as we actually
run the dd...

Fixes: #10861
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

calamari_setup: oops: ice-tools is still where it was on github.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7ea0a424a53bf68cad4cd4c7c78a07401dd6851)

Merge pull request #372 from ceph/wip-11180

ensure summary is looked for the user we need

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>

calamari_setup: add -y to yum localinstall icesetup

also fix a few flake8 things

Signed-off-by: Ubuntu <ubuntu@vpm047.front.sepia.ceph.com>

calamari_setup: handle iso, preserving support for tar.gz

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: small comment/logmsg fixes, correct git user path

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: make iceball fetch/creators return path to iceball

Also, widen the concept of "iceball" to include both iso and tar.gz

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: Refactor/simplify in preparation for iso support

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Implemented "exclude_packages" feature #11043

Fixed per Josh's comment

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 4250e22dde45c97ad24793f13fa41e95f16d03a4)

Merge pull request #364 from ceph/wip-11043

Implemented "exclude_packages" feature #11043
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #363 from ceph/hammer-backports-hadoop

hadoop: fixup for new replication test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Implemented "exclude_packages" feature #11043

Fixed per Josh's comment

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Implemented "exclude_packages" feature #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #357 from ceph/wip-10949

tasks/cephfs: clean up core on deliberate crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: clean up core on deliberate crash

To avoid internal.coredump task synthesizing a failure
during teardown from the core we left behind.

Fixes: #10949
Signed-off-by: John Spray <john.spray@redhat.com>

hadoop: fixup for new replication test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit 6a348ae1870eb4e22a35f4ea86d897fb6d960b88)

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2e4d884fdaabbeca67b519537a80166d3cc99fa3)

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 811c02e3922c4ae29c09bb5f7b48840d0eced951)

Merge pull request #362 from ceph/wip-11020

Control the nfs service correctly on rpm-based systems

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #360 from ceph/wip-11013

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #356 from ceph/wip-10948

tasks/cephfs: don't run iptables in parallel

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #358 from ceph/wip-10712

tasks: fix intermittent failure in TestFlush

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #334 from ceph/wip-fuse-umount

tasks/cephfs: fix fuse force umount (again)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Use an NFS mount instead of virtio-9p in the qemu task

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 93f2bea3530a523c7b2ac0575033697cea893f2d)

workunit: include /usr/sbin in the PATH for all commands

Fixes: #10600
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0e53f5f38b0cb06a6f4989e622a279c8ba9104eb)

rename test to tests, follows best practices for tests dirs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b2062602f7d3de206f4660c3e2847f86ac41e24)

create a new get_acl helper to ensure proper xml is compared

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b6ff638735d03c1f3304198fad51ca701a2c8bd)

tests for the new acl helper in radosgw_admin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3801ac9357f78ac05fa850d8944651f26c556d68)

tasks: fix intermittent failure in TestFlush

This was an overly strict success condition: the
flush operation doesn't promise to leave you an empty
journal, it promises that anything in the journal
before the flush will be flushed.

Fixes: #10712
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: don't run iptables in parallel

Where multiple MDSs were on the same node, trying
to concurrently update their firewall state was
causing an exception because the iptables command
errors out if another instance is already running.

Fixes: #10948
Signed-off-by: John Spray <john.spray@redhat.com>

Changed suite name from multi-versions to client-upgrade

Moved it under upgrades
Replaced rbd workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit bcf7eabc5f0bcd0c6665bc047c076925fbc9bdf5)

Merge pull request #354 from ceph/wip-multi-versions

Changed suite name from multi-versions to client-upgrade

Reviewed-by: Josh Durgin <jdurgin@redhat.com>