Initial checkin for hammer-x suite
Used "ceph osd crush tunables hammer"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit f825d32d9deab22f6113b96a6b36734fe1bdb289)

task/samba: ignore return code of fuser/losf

if mount point is not used by anyone, both fuser and losf return 1

Fixes: #10624
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit c049387b7b0832e78a4a1370691798f726985799)

task/samba: use SIGTERM to stop samba server

man samba(8) contains sentences:

To shut down a user's smbd process it is recommended that SIGKILL (-9)
NOT be used, except as a last resort, as this may leave the shared
memory area in an inconsistent state. The safe way to terminate an smbd
is to send it a SIGTERM (-15) signal and wait for it to die on its own.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 86bd6bc3771694cd9ee5cd071255f3cf4c40c812)

Merge pull request #416 from ceph/wip-11316-next

Backport to next: fix ceph-deploy task with CephFS (#11316)

tasks/ceph_deploy: configure CephFS

This test apparently had not been touched since
"fs new" was added.  In addition to calling
Filesystem.create:
 * modify the get_nodes_using_role
   function to modify ctx.cluster.remotes so that the
   service IDs match what ceph-deploy will set
 * log exceptions during ceph_deploy setup, as otherwise
   they can get lost if another exception occurs during
   teardown (so that it's all easier to debug).
 * default to passing --dev=master during install, so
   that we don't error out horribly when run without
   an explicit branch set (e.g. when run outside
   scheduled suite)

Fixes: #11316
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 7c2e6cdc6cd3d02969d3494d120b12daaa13efa8)

tasks/cephfs: cope with missing ctx.daemons attr

...so that we may be used with the ceph_deploy task
that doesn't set that up.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f36011051cabb8e7b06194c778e640e98d328e8a)

tasks/cephfs: tweak use of mon for admin commands

... s/mon_remote/admin_remote/ and allow caller to pass
in which remote they want to use for that.  Enables use
with ceph_deploy task which does not give admin keys
to mons.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8b61310ce3eedeb8ac60446933cd5db548de6434)

tasks/ceph: refactor legacy FS configuration check

Move up into Filesystem so that this can be used from
the ceph_deploy task as well.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit ea7c39222ae084c90b5a8510ce5af35a2f75a6e3)

suites/fs: update log whitelist

These ones had been passing because of #11177,
now that that is fixed they require an update.

Fixes: #11300
Signed-off-by: John Spray <john.spray@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit fa2a86196c3ed39c1726be689e42ed5d4aae197a)

Merge pull request #395 from fullerdj/wip-rbd-xfstests-201504

RBD: add YAML variables to override locations for ceph-qa-chef and xfstests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #399 from ceph/wip-11043

Removed per #11070 and resolves #11043

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed per #11070 and resolves #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

RBD: added optional YAML parameters to test xfstests from different repos

These variables are needed because ceph-qa-suite bootstraps ceph-qa-chef via
http download of solo-from/scratch/run. This adds a variable to override the
default script. It also adds variables to the rbd task to override the versions
of run_xfstests_krbd.sh and run_xfstests.sh downloaded by the default task.

variables added
======
tasks:
-chef
  script_url: # override default location for solo-from-scratch for Chef
  chef_repo: # override default Chef repo used by solo-from-scratch
  chef_branch: # to choose a different git upstream branch for ceph-qa-chef
-rbd.xfstests:
  client.0:
   xfstests_branch: # to choose a different git upstream branch for xfstests
   xfstests_url: # override git base URL for run_xfstests{_krbd}.sh

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Increased default test RBD size to 10G to help tests pass

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #367 from dachary/wip-ec-cache-agent

add erasure code and cache agent upgrade tests

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>

erasure-code: ec-cache-agent in firefly-x/stress-split-erasure-code

Immediately after the Firefly installation, create an erasure code pool
behind a replicated cache pool. Run deep-scrub on all OSD while a rados
task runs. After upgrading half of the cluster (MON and OSD), run a
rados task again also deep-scrub in parallel.

http://tracker.ceph.com/issues/11054 Fixes: #11054

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: rename firefly-x/stress-split-erasure-code

Rename sub-directories of firefly-x/stress-split-erasure-code make room
for a workload immediately after the installation, i.e. number 2.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados: explain that the task is asynchronous by default

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #393 from ceph/revert-385-wip-mkfs

Revert "ceph: be less weird about passing -f to mkfs"

Revert "ceph: be less weird about passing -f to mkfs"

Fixed #11306 Whitelist WRN "failed to encode map"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 9b9d27b25ee3f75fedc5db788bbe87b93d517f0e)

Merge pull request #388 from athanatos/wip-wrongly

rados_python: whitelist wrongly marked me down

Merge pull request #387 from athanatos/wip-scrub-repair-test

(scrub|rados)_test: tolerate best guess digest errors as well

Merge pull request #386 from athanatos/wip-11156

rados/thrash*: make scrubs happen a lot

Merge pull request #385 from ceph/wip-mkfs

ceph: be less weird about passing -f to mkfs

ceph: fix mkfs -f bug

Pass -f by default to btrfs instead of first trying without and *then*
trying with.

Among other things, this avoids a confusing failure where we try mkfs.ext4
device (no -f), fail for some reason, and then try again with -f and get
a usage error (-f does not mean force for mke2fs).

Signed-off-by: Sage Weil <sage@redhat.com>

rados_python: whitelist wrongly marked me down

Signed-off-by: Samuel Just <sjust@redhat.com>

(scrub|rados)_test: tolerate best guess digest errors as well

Signed-off-by: Samuel Just <sjust@redhat.com>

rados/thrash*: make scrubs happen a lot

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #380 from ceph/wip-11204

Make sure that ulimits are adjusted for ceph-objectstore-tool

Merge pull request #382 from dmick/master

calamari_setup: der.  Use dict.update() correctly

calamari_setup: der.  Use dict.update() correctly

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #381 from dmick/master

calamari_setup fixes

Merge remote-tracking branch 'gh/hammer'

Conflicts:
.gitignore

calamari_setup: Require test_image to be set

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: centralize config defaults

Make a DEFAULTS dict that is updated by any user parms, so that
defaults are documented centrally and so config.get(key, defval) is
no longer necessary everywhere.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: remove "build test image" code; add 'test_image' cfgvar

Stop trying to build test images inside this test; presume the test
image is available built externally (in a file path or an http URL).
Config vars ice_tool_dir, ice_version, iceball_location, and
ice_git_location go away in favor of 'test_image', the path to the
testable image (which can still be a tar.gz or an .iso).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: mounting iso on older distros requires -o loop

Ubuntu's mount/kernel support "mount <file> <mntpnt>" directly;
apparently Centos 6 (and presumably RHEL6) require specifying at
least '-o loop' (a /dev/loopN will be dynamically allocated and removed
on unmount).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #379 from ceph/wip-wn

fix watch-notify test

tasks/watch_notify_same_primary: wait for watch before notify

Make sure watch is done registering and ready before sending the
notifies.

Fixes: #10634
Signed-off-by: Sage Weil <sage@redhat.com>

Make sure that ulimits are adjusted for ceph-objectstore-tool

Fixes: #11204
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #378 from dachary/wip-11221-erasure-code-not-parallel

erasure-code: enable ec-rados-default.yaml

erasure-code: enable ec-rados-default.yaml

The ec-rados-default.yaml started with:

  workload:
    sequential:

which is only suitable for suites/upgrade/giant-x/parallel/2-workload/sequential_run/ec-rados-default.yaml

because suites/upgrade/giant-x/parallel/1-giant-install/giant.yaml has

   - parallel:
      - workload
      - upgrade-sequence

The same file was included in contexts where the parallel task was not
used and the workload did not run:

  ./suites/upgrade/firefly-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code-x86_64/5-workload/ec-rados-default.yaml

The ec-rados-default.yaml is modified to be a task instead of a
sequential task in a parallel tasks. The ec-rados-sequential.yaml is
added and is linked in
suites/upgrade/giant-x/parallel/2-workload/sequential_run instead of ec-rados-default.yaml.

http://tracker.ceph.com/issues/11221 Fixes: #11221

Signed-off-by: Loic Dachary <loic@dachary.org>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 97e6d808f086834f2350084d5de36be285ad2bde)

Merge pull request #377 from ceph/wip-11166

Fixes #11166, whitelisted 'Missing health data for MDS'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #11166, whitelisted 'Missing health data for MDS'

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #375 from dachary/wip-rgw-regional-summary

ensure summary is looked for the user we need (part 2)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #374 from ceph/wip-11139

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Fixes: #11139
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #373 from ceph/wip-10861

fs: fix up dd testing again

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #371 from ceph/wip-iso-style

Add support for ISOs to tasks/calamari_setup

fs: fix up dd testing again

We need to change into the directory during the same shell session as we actually
run the dd...

Fixes: #10861
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

calamari_setup: oops: ice-tools is still where it was on github.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7ea0a424a53bf68cad4cd4c7c78a07401dd6851)

Merge pull request #372 from ceph/wip-11180

ensure summary is looked for the user we need

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>

calamari_setup: add -y to yum localinstall icesetup

also fix a few flake8 things

Signed-off-by: Ubuntu <ubuntu@vpm047.front.sepia.ceph.com>

calamari_setup: handle iso, preserving support for tar.gz

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: small comment/logmsg fixes, correct git user path

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: make iceball fetch/creators return path to iceball

Also, widen the concept of "iceball" to include both iso and tar.gz

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: Refactor/simplify in preparation for iso support

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Implemented "exclude_packages" feature #11043

Fixed per Josh's comment

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 4250e22dde45c97ad24793f13fa41e95f16d03a4)

Merge pull request #364 from ceph/wip-11043

Implemented "exclude_packages" feature #11043
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #363 from ceph/hammer-backports-hadoop

hadoop: fixup for new replication test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Implemented "exclude_packages" feature #11043

Fixed per Josh's comment

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Implemented "exclude_packages" feature #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #357 from ceph/wip-10949

tasks/cephfs: clean up core on deliberate crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: clean up core on deliberate crash

To avoid internal.coredump task synthesizing a failure
during teardown from the core we left behind.

Fixes: #10949
Signed-off-by: John Spray <john.spray@redhat.com>

hadoop: fixup for new replication test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit 6a348ae1870eb4e22a35f4ea86d897fb6d960b88)

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2e4d884fdaabbeca67b519537a80166d3cc99fa3)

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 811c02e3922c4ae29c09bb5f7b48840d0eced951)

Merge pull request #362 from ceph/wip-11020

Control the nfs service correctly on rpm-based systems

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #360 from ceph/wip-11013

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #356 from ceph/wip-10948

tasks/cephfs: don't run iptables in parallel

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #358 from ceph/wip-10712

tasks: fix intermittent failure in TestFlush

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #334 from ceph/wip-fuse-umount

tasks/cephfs: fix fuse force umount (again)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Use an NFS mount instead of virtio-9p in the qemu task

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 93f2bea3530a523c7b2ac0575033697cea893f2d)

workunit: include /usr/sbin in the PATH for all commands

Fixes: #10600
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0e53f5f38b0cb06a6f4989e622a279c8ba9104eb)

rename test to tests, follows best practices for tests dirs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b2062602f7d3de206f4660c3e2847f86ac41e24)

create a new get_acl helper to ensure proper xml is compared

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b6ff638735d03c1f3304198fad51ca701a2c8bd)

tests for the new acl helper in radosgw_admin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3801ac9357f78ac05fa850d8944651f26c556d68)

tasks: fix intermittent failure in TestFlush

This was an overly strict success condition: the
flush operation doesn't promise to leave you an empty
journal, it promises that anything in the journal
before the flush will be flushed.

Fixes: #10712
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: don't run iptables in parallel

Where multiple MDSs were on the same node, trying
to concurrently update their firewall state was
causing an exception because the iptables command
errors out if another instance is already running.

Fixes: #10948
Signed-off-by: John Spray <john.spray@redhat.com>

Changed suite name from multi-versions to client-upgrade

Moved it under upgrades
Replaced rbd workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit bcf7eabc5f0bcd0c6665bc047c076925fbc9bdf5)

Merge pull request #354 from ceph/wip-multi-versions

Changed suite name from multi-versions to client-upgrade

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Changed suite name from multi-versions to client-upgrade

Moved it under upgrades
Replaced rbd workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

tasks/cephfs: fix/improve fuse force umount (again)

teuthology helpfully escapes things for us so
the \; didn't need the backslash.  The logic
was still falling over in some cases too.

Additionally, make the FUSE /sys/ abort operation
more surgical by working out the connection name
of our own mount during mount().

Signed-off-by: John Spray <john.spray@redhat.com>

msgr: move async msgr tests to singleton-nomsgr

Introduced: e92d2e4ca0013fa379fdf5830d4fe28abf65c33b
Fixes: 10918
Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

exclude the virtualenv from being linted in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b944b06dc2479d77d6efac4f13932cb102ddfe1c)
(cherry picked from commit 6de0a684d5b4460390e610806201e3eaf05feb6e)

Merge pull request #349 from ceph/wip-10954-hammer

lint fixes for hammer

remove unused import from repair_test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ignore tox hidden dir

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4068f73ade68e36c1e7653733eacf2b1288cd753)

remove unused variable assignment

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9df27f7a1040dae69ef65122d13d1640cb1b6a66)

remove unused import from fuse

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove uneeded variable assignment for call

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a65e90db2e40b04c7cbb2347f4c759e283a2c2e)

remove unused import in calamari_setup

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d22cd6917bf77a57c323684b8c6e3687238df324)

remove unused imports from populate_rbd

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 959c4914629bd237d10cf5789906eba42b56a984)