Merge pull request #395 from fullerdj/wip-rbd-xfstests-201504

RBD: add YAML variables to override locations for ceph-qa-chef and xfstests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #399 from ceph/wip-11043

Removed per #11070 and resolves #11043

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed per #11070 and resolves #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

RBD: added optional YAML parameters to test xfstests from different repos

These variables are needed because ceph-qa-suite bootstraps ceph-qa-chef via
http download of solo-from/scratch/run. This adds a variable to override the
default script. It also adds variables to the rbd task to override the versions
of run_xfstests_krbd.sh and run_xfstests.sh downloaded by the default task.

variables added
======
tasks:
-chef
  script_url: # override default location for solo-from-scratch for Chef
  chef_repo: # override default Chef repo used by solo-from-scratch
  chef_branch: # to choose a different git upstream branch for ceph-qa-chef
-rbd.xfstests:
  client.0:
   xfstests_branch: # to choose a different git upstream branch for xfstests
   xfstests_url: # override git base URL for run_xfstests{_krbd}.sh

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Increased default test RBD size to 10G to help tests pass

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #367 from dachary/wip-ec-cache-agent

add erasure code and cache agent upgrade tests

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>

erasure-code: ec-cache-agent in firefly-x/stress-split-erasure-code

Immediately after the Firefly installation, create an erasure code pool
behind a replicated cache pool. Run deep-scrub on all OSD while a rados
task runs. After upgrading half of the cluster (MON and OSD), run a
rados task again also deep-scrub in parallel.

http://tracker.ceph.com/issues/11054 Fixes: #11054

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: rename firefly-x/stress-split-erasure-code

Rename sub-directories of firefly-x/stress-split-erasure-code make room
for a workload immediately after the installation, i.e. number 2.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados: explain that the task is asynchronous by default

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #393 from ceph/revert-385-wip-mkfs

Revert "ceph: be less weird about passing -f to mkfs"

Revert "ceph: be less weird about passing -f to mkfs"

Fixed #11306 Whitelist WRN "failed to encode map"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 9b9d27b25ee3f75fedc5db788bbe87b93d517f0e)

Merge pull request #388 from athanatos/wip-wrongly

rados_python: whitelist wrongly marked me down

Merge pull request #387 from athanatos/wip-scrub-repair-test

(scrub|rados)_test: tolerate best guess digest errors as well

Merge pull request #386 from athanatos/wip-11156

rados/thrash*: make scrubs happen a lot

Merge pull request #385 from ceph/wip-mkfs

ceph: be less weird about passing -f to mkfs

ceph: fix mkfs -f bug

Pass -f by default to btrfs instead of first trying without and *then*
trying with.

Among other things, this avoids a confusing failure where we try mkfs.ext4
device (no -f), fail for some reason, and then try again with -f and get
a usage error (-f does not mean force for mke2fs).

Signed-off-by: Sage Weil <sage@redhat.com>

rados_python: whitelist wrongly marked me down

Signed-off-by: Samuel Just <sjust@redhat.com>

(scrub|rados)_test: tolerate best guess digest errors as well

Signed-off-by: Samuel Just <sjust@redhat.com>

rados/thrash*: make scrubs happen a lot

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #380 from ceph/wip-11204

Make sure that ulimits are adjusted for ceph-objectstore-tool

Merge pull request #382 from dmick/master

calamari_setup: der.  Use dict.update() correctly

calamari_setup: der.  Use dict.update() correctly

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #381 from dmick/master

calamari_setup fixes

Merge remote-tracking branch 'gh/hammer'

Conflicts:
.gitignore

calamari_setup: Require test_image to be set

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: centralize config defaults

Make a DEFAULTS dict that is updated by any user parms, so that
defaults are documented centrally and so config.get(key, defval) is
no longer necessary everywhere.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: remove "build test image" code; add 'test_image' cfgvar

Stop trying to build test images inside this test; presume the test
image is available built externally (in a file path or an http URL).
Config vars ice_tool_dir, ice_version, iceball_location, and
ice_git_location go away in favor of 'test_image', the path to the
testable image (which can still be a tar.gz or an .iso).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: mounting iso on older distros requires -o loop

Ubuntu's mount/kernel support "mount <file> <mntpnt>" directly;
apparently Centos 6 (and presumably RHEL6) require specifying at
least '-o loop' (a /dev/loopN will be dynamically allocated and removed
on unmount).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #379 from ceph/wip-wn

fix watch-notify test

tasks/watch_notify_same_primary: wait for watch before notify

Make sure watch is done registering and ready before sending the
notifies.

Fixes: #10634
Signed-off-by: Sage Weil <sage@redhat.com>

Make sure that ulimits are adjusted for ceph-objectstore-tool

Fixes: #11204
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #378 from dachary/wip-11221-erasure-code-not-parallel

erasure-code: enable ec-rados-default.yaml

erasure-code: enable ec-rados-default.yaml

The ec-rados-default.yaml started with:

  workload:
    sequential:

which is only suitable for suites/upgrade/giant-x/parallel/2-workload/sequential_run/ec-rados-default.yaml

because suites/upgrade/giant-x/parallel/1-giant-install/giant.yaml has

   - parallel:
      - workload
      - upgrade-sequence

The same file was included in contexts where the parallel task was not
used and the workload did not run:

  ./suites/upgrade/firefly-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code/5-workload/ec-rados-default.yaml
  ./suites/upgrade/giant-x/stress-split-erasure-code-x86_64/5-workload/ec-rados-default.yaml

The ec-rados-default.yaml is modified to be a task instead of a
sequential task in a parallel tasks. The ec-rados-sequential.yaml is
added and is linked in
suites/upgrade/giant-x/parallel/2-workload/sequential_run instead of ec-rados-default.yaml.

http://tracker.ceph.com/issues/11221 Fixes: #11221

Signed-off-by: Loic Dachary <loic@dachary.org>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 97e6d808f086834f2350084d5de36be285ad2bde)

Merge pull request #377 from ceph/wip-11166

Fixes #11166, whitelisted 'Missing health data for MDS'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes #11166, whitelisted 'Missing health data for MDS'

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #375 from dachary/wip-rgw-regional-summary

ensure summary is looked for the user we need (part 2)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ensure summary is looked for the user we need (part 2)

Move the get_user_summary(out, user) logic to util.rgw so that it can be
shared between radosgw_admin_rest.py and radosgw_admin.py and modify
them accordingly.

http://tracker.ceph.com/issues/11180 Fixes: #11180

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #374 from ceph/wip-11139

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

ceph_manager: Check for exit status 11 from ceph-objectstore-tool import

Fixes: #11139
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #373 from ceph/wip-10861

fs: fix up dd testing again

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Travis Rhoden <trhoden@redhat.com>

Merge pull request #371 from ceph/wip-iso-style

Add support for ISOs to tasks/calamari_setup

fs: fix up dd testing again

We need to change into the directory during the same shell session as we actually
run the dd...

Fixes: #10861
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

calamari_setup: oops: ice-tools is still where it was on github.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7ea0a424a53bf68cad4cd4c7c78a07401dd6851)

Merge pull request #372 from ceph/wip-11180

ensure summary is looked for the user we need

ensure summary is looked for the user we need

Signed-off-by: Alfredo Deza <adeza@redhat.com>

calamari_setup: add -y to yum localinstall icesetup

also fix a few flake8 things

Signed-off-by: Ubuntu <ubuntu@vpm047.front.sepia.ceph.com>

calamari_setup: handle iso, preserving support for tar.gz

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: small comment/logmsg fixes, correct git user path

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: make iceball fetch/creators return path to iceball

Also, widen the concept of "iceball" to include both iso and tar.gz

Signed-off-by: Dan Mick <dan.mick@redhat.com>

calamari_setup: Refactor/simplify in preparation for iso support

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Implemented "exclude_packages" feature #11043

Fixed per Josh's comment

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 4250e22dde45c97ad24793f13fa41e95f16d03a4)

Merge pull request #364 from ceph/wip-11043

Implemented "exclude_packages" feature #11043
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #363 from ceph/hammer-backports-hadoop

hadoop: fixup for new replication test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Implemented "exclude_packages" feature #11043

Fixed per Josh's comment

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Implemented "exclude_packages" feature #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #357 from ceph/wip-10949

tasks/cephfs: clean up core on deliberate crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: clean up core on deliberate crash

To avoid internal.coredump task synthesizing a failure
during teardown from the core we left behind.

Fixes: #10949
Signed-off-by: John Spray <john.spray@redhat.com>

hadoop: fixup for new replication test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
(cherry picked from commit 6a348ae1870eb4e22a35f4ea86d897fb6d960b88)

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2e4d884fdaabbeca67b519537a80166d3cc99fa3)

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 811c02e3922c4ae29c09bb5f7b48840d0eced951)

Merge pull request #362 from ceph/wip-11020

Control the nfs service correctly on rpm-based systems

Control the nfs service correctly on rpm-based systems

This fixes issue #11020

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #360 from ceph/wip-11013

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Fixes #11013, use time.sleep instead of manager.sleep which isn't there.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #356 from ceph/wip-10948

tasks/cephfs: don't run iptables in parallel

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #358 from ceph/wip-10712

tasks: fix intermittent failure in TestFlush

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #334 from ceph/wip-fuse-umount

tasks/cephfs: fix fuse force umount (again)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Use an NFS mount instead of virtio-9p in the qemu task

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 93f2bea3530a523c7b2ac0575033697cea893f2d)

workunit: include /usr/sbin in the PATH for all commands

Fixes: #10600
Backport: hammer, giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 0e53f5f38b0cb06a6f4989e622a279c8ba9104eb)

rename test to tests, follows best practices for tests dirs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b2062602f7d3de206f4660c3e2847f86ac41e24)

create a new get_acl helper to ensure proper xml is compared

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b6ff638735d03c1f3304198fad51ca701a2c8bd)

tests for the new acl helper in radosgw_admin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3801ac9357f78ac05fa850d8944651f26c556d68)

tasks: fix intermittent failure in TestFlush

This was an overly strict success condition: the
flush operation doesn't promise to leave you an empty
journal, it promises that anything in the journal
before the flush will be flushed.

Fixes: #10712
Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: don't run iptables in parallel

Where multiple MDSs were on the same node, trying
to concurrently update their firewall state was
causing an exception because the iptables command
errors out if another instance is already running.

Fixes: #10948
Signed-off-by: John Spray <john.spray@redhat.com>

Changed suite name from multi-versions to client-upgrade

Moved it under upgrades
Replaced rbd workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit bcf7eabc5f0bcd0c6665bc047c076925fbc9bdf5)

Merge pull request #354 from ceph/wip-multi-versions

Changed suite name from multi-versions to client-upgrade

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Changed suite name from multi-versions to client-upgrade

Moved it under upgrades
Replaced rbd workunit

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

tasks/cephfs: fix/improve fuse force umount (again)

teuthology helpfully escapes things for us so
the \; didn't need the backslash.  The logic
was still falling over in some cases too.

Additionally, make the FUSE /sys/ abort operation
more surgical by working out the connection name
of our own mount during mount().

Signed-off-by: John Spray <john.spray@redhat.com>

msgr: move async msgr tests to singleton-nomsgr

Introduced: e92d2e4ca0013fa379fdf5830d4fe28abf65c33b
Fixes: 10918
Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

exclude the virtualenv from being linted in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b944b06dc2479d77d6efac4f13932cb102ddfe1c)
(cherry picked from commit 6de0a684d5b4460390e610806201e3eaf05feb6e)

Merge pull request #349 from ceph/wip-10954-hammer

lint fixes for hammer

remove unused import from repair_test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ignore tox hidden dir

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4068f73ade68e36c1e7653733eacf2b1288cd753)

remove unused variable assignment

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9df27f7a1040dae69ef65122d13d1640cb1b6a66)

remove unused import from fuse

Signed-off-by: Alfredo Deza <adeza@redhat.com>

remove uneeded variable assignment for call

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a65e90db2e40b04c7cbb2347f4c759e283a2c2e)

remove unused import in calamari_setup

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d22cd6917bf77a57c323684b8c6e3687238df324)

remove unused imports from populate_rbd

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 959c4914629bd237d10cf5789906eba42b56a984)

remove unused imports from peering_speed

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6d45352782deb5606c8dcad39c75a94e748e292c)

remove redefined StringIO import

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3a18cb29a855a4b507f79a349821cb246a12c8c6)

stdin is no longer a kwarg

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 49a61dc2d2f66bd8dde34ba67dbd98d4c5148c9e)

add the log object to ceph_manager

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f7c1ca4a1e23b03ecf86d5e67d563f95b3a82f99)

add missing mon object to osd_failsafe

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1673be45c0e3aeb1eadad7ecd5b87570e9bac5ca)

fix undefined var in divergent_priors

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 26c5a82af9cf4efd3601b6a32aa5444f61fa9f81)

exclude the virtualenv from being linted in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

fix path for tox.ini to work on ceph-qa-suite only

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4eb1787280f9f128cb9352ecbe8957a4be94fff9)

initial tox.ini to run flake8

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7108b699fda247ce12cbeaea9dd0a32843775c27)
(cherry picked from commit e1c85f330b834394d099653d3c618d41cd438cff)