Merge pull request #422 from ceph/wip-log-whitelist

suites: log whitelist for journal repair test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suites: log whitelist for journal repair test

"bad backtrace on dir ino" is emitted because
journaltool's recover-dentries doesn't write
backtraces.

Signed-off-by: John Spray <john.spray@redhat.com>

rbd.xfstests: remove bad assert

Misguided input validation. A better error comes
from the run_xfstests() method later anyway.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #405 from ceph/wip-10781

Initial checkin for hammer-x suite

Merge pull request #414 from ceph/wip_11413

Added hammer-client-x to the mix

xfstests: fix typo

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

xfstests: define defaults in root task

The early non-defaults caused failures due to xfstests_url: None not
being overridden by run_xfstests(). Move the defaults to xfstests() and
don pass xfstests_branch past that point.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #417 from ceph/wip-10657

cephfs/test_client_limits.py: test for client oldest tid warning

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #415 from ceph/wip-10624

task/samba: ignore return code of fuser/losf

Reviewed-by: Sage Weil <sage@redhat.com>

cephfs/test_client_limits.py: test for client oldest tid warning

Signed-off-by: Yan, Zheng <zyan@redhat.com>

task/samba: ignore return code of fuser/losf

if mount point is not used by anyone, both fuser and losf return 1

Fixes: #10624
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge branch 'greg-testing-snapshot-rename'

Merge pull request #404 from ceph/wip-3645-fixup

DNM: tasks: s/pending_create/pending_update/ in MDS

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'origin/wip-3645-fixup' into greg-testing-snapshot-rename

Merge remote-tracking branch 'origin/wip-11316' into greg-testing-quotas

Merge remote-tracking branch 'origin/wip-10390' into greg-testing-quotas

Conflicts:
suites/fs/recovery/tasks/mds-full.yaml

suites/fs: update log whitelist

These ones had been passing because of #11177,
now that that is fixed they require an update.

Fixes: #11300
Signed-off-by: John Spray <john.spray@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph_deploy: configure CephFS

This test apparently had not been touched since
"fs new" was added.  In addition to calling
Filesystem.create:
 * modify the get_nodes_using_role
   function to modify ctx.cluster.remotes so that the
   service IDs match what ceph-deploy will set
 * log exceptions during ceph_deploy setup, as otherwise
   they can get lost if another exception occurs during
   teardown (so that it's all easier to debug).
 * default to passing --dev=master during install, so
   that we don't error out horribly when run without
   an explicit branch set (e.g. when run outside
   scheduled suite)

Fixes: #11316
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: cope with missing ctx.daemons attr

...so that we may be used with the ceph_deploy task
that doesn't set that up.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: tweak use of mon for admin commands

... s/mon_remote/admin_remote/ and allow caller to pass
in which remote they want to use for that.  Enables use
with ceph_deploy task which does not give admin keys
to mons.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph: refactor legacy FS configuration check

Move up into Filesystem so that this can be used from
the ceph_deploy task as well.

Signed-off-by: John Spray <john.spray@redhat.com>

suites: whitelist log messages for auto-repair test

Log messages being checked now since #11177
was fixed.

Signed-off-by: John Spray <john.spray@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Added hammer-client-x to the mix
Renamed dirs to better reflect what is being tested
Removed old dirs
Added logging

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #403 from ceph/wip-6613

task/samba: use SIGTERM to stop samba server

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #409 from ceph/wip-git.ceph.com

git.ceph.com

http not https for git.ceph.com

Until we have https at least!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #408 from ceph/wip-git.ceph.com

git.ceph.com

Reviewed-by: Alfredo Deza <adeza@redhat.com>

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #407 from ceph/wip-git.ceph.com

git.ceph.com

Reviewed-by: Alfredo Deza <adeza@redhat.com>

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

Initial checkin for hammer-x suite
Used "ceph osd crush tunables hammer"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Update latest_firefly_release.yaml

Changed tag to "branch: firefly"

tasks: s/pending_create/pending_update/ in MDS

Tests inspecting snapserver dump need updating
to reflect the rename.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: implement TestStrays.test_mv_hardlink_cleanup

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: update test_journal_repair

This broke with recent Client changes that
do better caching of readdir results, such
that doing an ls twice is no longer sufficient
to see a fresh result after repair - we need
to remount instead.

Signed-off-by: John Spray <john.spray@redhat.com>

suites/fs: clean up whitespace in YAML

Signed-off-by: John Spray <john.spray@redhat.com>

suites: run fs/recovery with two MDSs

Some tests need both, and others should
ignore it.

Signed-off-by: John Spray <john.spray@redhat.com>

suites: add .yamls for the recent cephfs tests

Signed-off-by: John Spray <john.spray@redhat.com>

suites: put fuse clients on separate hosts

To mitigate the weirdness when we start
trying to abort /sys/fs/fuse/connections/*

Signed-off-by: John Spray <john.spray@redhat.com>

suites: update tasks to use cephfs_test_runner

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: update journal_repair test for 'damaged' state

To track recent change in master where instead of
crashing on missing MDSTable object we'll go
into damaged state.

Instead of catching a crash, handle the rank's
transition to the damanged state.  Leave the crash
handling code (unused for the moment) in the
Filesystem class in case it's needed elsewhere
soon.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_strays

This tests the new purge file/ops throttling
in the MDS, via the new perf counters for
strays/purging.

Fixes: #10390
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_sessionmap

Tests for the persistence behaviour of SessionMap.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: generalise cephfs test runner

...to avoid having boilerplate in each test module,
and gain the ability to run them all in one go
with a nice test-by-test pass/fail report.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph_fuse: populate ctx.mounts earlier

...so that if an error happens during mount, I can
use the interactive console to access ctx.mounts.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mds_flush: be more careful monitoring stats

Were previously taking the baseline from just after the
client did a delete, which was racy: should have taken
it from before, to get a steady state.

Also update the perf dump calls to take advantage of
the new filtering syntax.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: lots of s/mds_restart/mds_fail_restart/

Wherever we are subsequently waiting for daemons
to be healthy, we should be doing a fail during the restart.

Also catch some places that were doing this longhand and use
the handy fail_restart version instead.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: better multiple-mds handling

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph_manager: DRY in mds_status

Signed-off-by: John Spray <john.spray@redhat.com>

ceph_manager: fix bad type assertions

In python, isinstance(foo, str) will fail if
a unicode string is passed in.  The correct check
is basestring.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: be tolerant of multiple MDSs

...as long as only one is active, all the ops
that default to talking to a single MDS should
be happy to talk to the active MDS, even if there
happens to be a standby lying around too.

Signed-off-by: John Spray <john.spray@redhat.com>

task/samba: use SIGTERM to stop samba server

man samba(8) contains sentences:

To shut down a user's smbd process it is recommended that SIGKILL (-9)
NOT be used, except as a last resort, as this may leave the shared
memory area in an inconsistent state. The safe way to terminate an smbd
is to send it a SIGTERM (-15) signal and wait for it to die on its own.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #395 from fullerdj/wip-rbd-xfstests-201504

RBD: add YAML variables to override locations for ceph-qa-chef and xfstests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #399 from ceph/wip-11043

Removed per #11070 and resolves #11043

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed per #11070 and resolves #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

RBD: added optional YAML parameters to test xfstests from different repos

These variables are needed because ceph-qa-suite bootstraps ceph-qa-chef via
http download of solo-from/scratch/run. This adds a variable to override the
default script. It also adds variables to the rbd task to override the versions
of run_xfstests_krbd.sh and run_xfstests.sh downloaded by the default task.

variables added
======
tasks:
-chef
  script_url: # override default location for solo-from-scratch for Chef
  chef_repo: # override default Chef repo used by solo-from-scratch
  chef_branch: # to choose a different git upstream branch for ceph-qa-chef
-rbd.xfstests:
  client.0:
   xfstests_branch: # to choose a different git upstream branch for xfstests
   xfstests_url: # override git base URL for run_xfstests{_krbd}.sh

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Increased default test RBD size to 10G to help tests pass

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #367 from dachary/wip-ec-cache-agent

add erasure code and cache agent upgrade tests

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>

erasure-code: ec-cache-agent in firefly-x/stress-split-erasure-code

Immediately after the Firefly installation, create an erasure code pool
behind a replicated cache pool. Run deep-scrub on all OSD while a rados
task runs. After upgrading half of the cluster (MON and OSD), run a
rados task again also deep-scrub in parallel.

http://tracker.ceph.com/issues/11054 Fixes: #11054

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: rename firefly-x/stress-split-erasure-code

Rename sub-directories of firefly-x/stress-split-erasure-code make room
for a workload immediately after the installation, i.e. number 2.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados: explain that the task is asynchronous by default

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #393 from ceph/revert-385-wip-mkfs

Revert "ceph: be less weird about passing -f to mkfs"

Revert "ceph: be less weird about passing -f to mkfs"

Fixed #11306 Whitelist WRN "failed to encode map"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 9b9d27b25ee3f75fedc5db788bbe87b93d517f0e)

Merge pull request #388 from athanatos/wip-wrongly

rados_python: whitelist wrongly marked me down

Merge pull request #387 from athanatos/wip-scrub-repair-test

(scrub|rados)_test: tolerate best guess digest errors as well

Merge pull request #386 from athanatos/wip-11156

rados/thrash*: make scrubs happen a lot

Merge pull request #385 from ceph/wip-mkfs

ceph: be less weird about passing -f to mkfs

ceph: fix mkfs -f bug

Pass -f by default to btrfs instead of first trying without and *then*
trying with.

Among other things, this avoids a confusing failure where we try mkfs.ext4
device (no -f), fail for some reason, and then try again with -f and get
a usage error (-f does not mean force for mke2fs).

Signed-off-by: Sage Weil <sage@redhat.com>

rados_python: whitelist wrongly marked me down

Signed-off-by: Samuel Just <sjust@redhat.com>

(scrub|rados)_test: tolerate best guess digest errors as well

Signed-off-by: Samuel Just <sjust@redhat.com>

rados/thrash*: make scrubs happen a lot

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #380 from ceph/wip-11204

Make sure that ulimits are adjusted for ceph-objectstore-tool

Merge pull request #382 from dmick/master

calamari_setup: der.  Use dict.update() correctly

calamari_setup: der.  Use dict.update() correctly

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #381 from dmick/master

calamari_setup fixes

Merge remote-tracking branch 'gh/hammer'

Conflicts:
.gitignore

calamari_setup: Require test_image to be set

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: centralize config defaults

Make a DEFAULTS dict that is updated by any user parms, so that
defaults are documented centrally and so config.get(key, defval) is
no longer necessary everywhere.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: remove "build test image" code; add 'test_image' cfgvar

Stop trying to build test images inside this test; presume the test
image is available built externally (in a file path or an http URL).
Config vars ice_tool_dir, ice_version, iceball_location, and
ice_git_location go away in favor of 'test_image', the path to the
testable image (which can still be a tar.gz or an .iso).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: mounting iso on older distros requires -o loop

Ubuntu's mount/kernel support "mount <file> <mntpnt>" directly;
apparently Centos 6 (and presumably RHEL6) require specifying at
least '-o loop' (a /dev/loopN will be dynamically allocated and removed
on unmount).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #379 from ceph/wip-wn

fix watch-notify test

tasks/watch_notify_same_primary: wait for watch before notify

Make sure watch is done registering and ready before sending the
notifies.

Fixes: #10634
Signed-off-by: Sage Weil <sage@redhat.com>

Make sure that ulimits are adjusted for ceph-objectstore-tool

Fixes: #11204
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #378 from dachary/wip-11221-erasure-code-not-parallel

erasure-code: enable ec-rados-default.yaml

erasure-code: enable ec-rados-default.yaml

The ec-rados-default.yaml started with:

  workload:
    sequential:

which is only suitable for suites/upgrade/giant-x/parallel/2-workload/sequential_run/ec-rados-default.yaml

because suites/upgrade/giant-x/parallel/1-giant-install/giant.yaml has

   - parallel:
      - workload
      - upgrade-sequence

The same file was included in contexts where the parallel task was not
used and the workload did not run:

  ./suites/upgrade/firefly-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code-x86_64/5-workload/ec-rados-default.yaml

The ec-rados-default.yaml is modified to be a task instead of a
sequential task in a parallel tasks. The ec-rados-sequential.yaml is
added and is linked in
suites/upgrade/giant-x/parallel/2-workload/sequential_run instead of ec-rados-default.yaml.

http://tracker.ceph.com/issues/11221 Fixes: #11221

Signed-off-by: Loic Dachary <loic@dachary.org>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 97e6d808f086834f2350084d5de36be285ad2bde)

Merge pull request #377 from ceph/wip-11166

Fixes #11166, whitelisted 'Missing health data for MDS'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #11166, whitelisted 'Missing health data for MDS'

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #375 from dachary/wip-rgw-regional-summary

ensure summary is looked for the user we need (part 2)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #374 from ceph/wip-11139

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Fixes: #11139
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #373 from ceph/wip-10861

fs: fix up dd testing again

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #371 from ceph/wip-iso-style

Add support for ISOs to tasks/calamari_setup

fs: fix up dd testing again

We need to change into the directory during the same shell session as we actually
run the dd...

Fixes: #10861
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

calamari_setup: oops: ice-tools is still where it was on github.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7ea0a424a53bf68cad4cd4c7c78a07401dd6851)

Merge pull request #372 from ceph/wip-11180

ensure summary is looked for the user we need

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>