cram: support fetching from sha1 branch, tag, commit hash

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge remote-tracking branch 'upstream/pull/1302/head'

* upstream/pull/1302/head:
  fsx-mpi: fsx-mpi expects a file name not a dir
  fsx-mpi: remove chef task
  Fix mpi calls

fsx-mpi: fsx-mpi expects a file name not a dir

The test still fails even after being enabled:

2016-12-07T18:00:44.337 INFO:teuthology.orchestra.run.mira105:Running: 'mpiexec -f /home/ubuntu/cephtest/mpi-hosts -wdir /home/ubuntu/cephtest/gmnt sudo /home/ubuntu/cephtest/fsx-mpi -o 1MB -N 50000 -p 10000 -l 1048576 /home/ubuntu/cephtest/gmnt/test'
2016-12-07T18:00:44.486 INFO:teuthology.orchestra.run.mira105.stderr:Warning: Permanently added '172.21.8.122' (ECDSA) to the list of known hosts.
2016-12-07T18:00:44.571 INFO:teuthology.orchestra.run.mira105.stdout:skipping zero size read
2016-12-07T18:00:44.591 INFO:teuthology.orchestra.run.mira105.stdout:truncating to largest ever: 0x7cccb
2016-12-07T18:00:44.606 INFO:teuthology.orchestra.run.mira083:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
2016-12-07T18:00:44.611 INFO:teuthology.orchestra.run.mira100:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
2016-12-07T18:00:44.614 INFO:teuthology.orchestra.run.mira105:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
2016-12-07T18:00:44.887 INFO:teuthology.orchestra.run.mira105.stdout:skipping zero size read
2016-12-07T18:00:44.954 INFO:teuthology.orchestra.run.mira105.stdout:Size error: expected 0xa6f7c stat 0xd4000 seek 0xd5000
2016-12-07T18:00:44.954 INFO:teuthology.orchestra.run.mira105.stdout:LOG DUMP (2 total operations):
2016-12-07T18:00:44.954 INFO:teuthology.orchestra.run.mira105.stdout:1(1 mod 256): SKIPPED (no operation)
2016-12-07T18:00:44.954 INFO:teuthology.orchestra.run.mira105.stdout:2(2 mod 256): WRITE        0x1c748 thru 0xa6f7b    (0x8a834 bytes) HOLE
2016-12-07T18:00:44.990 INFO:teuthology.orchestra.run.mira105.stdout:Correct content saved for comparison
2016-12-07T18:00:44.990 INFO:teuthology.orchestra.run.mira105.stdout:(maybe hexdump "/home/ubuntu/cephtest/gmnt/test" vs "/home/ubuntu/cephtest/gmnt/test.fsxgood")
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stdout:
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stdout:===================================================================================
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stdout:=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stdout:=   EXIT CODE: 120
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stdout:=   CLEANING UP REMAINING PROCESSES
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stdout:=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stdout:===================================================================================
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stderr:[proxy:0:0@mira105] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
2016-12-07T18:00:45.000 INFO:teuthology.orchestra.run.mira105.stderr:[proxy:0:0@mira105] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
2016-12-07T18:00:45.001 INFO:teuthology.orchestra.run.mira105.stderr:[proxy:0:0@mira105] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event
2016-12-07T18:00:45.002 INFO:teuthology.orchestra.run.mira105.stderr:[mpiexec@mira105] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting
2016-12-07T18:00:45.002 INFO:teuthology.orchestra.run.mira105.stderr:[mpiexec@mira105] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
2016-12-07T18:00:45.002 INFO:teuthology.orchestra.run.mira105.stderr:[mpiexec@mira105] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion
2016-12-07T18:00:45.002 INFO:teuthology.orchestra.run.mira105.stderr:[mpiexec@mira105] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion

I am not sure what the cause is. I'm leaving the test disabled for now and merging this PR.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

fsx-mpi: remove chef task

Per Sage's comment: https://github.com/ceph/ceph-qa-suite/pull/284#issuecomment-141463034

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #977 from ceph/wip-rgw-compression

rgw/verify: use rgw_compression_type = "random"

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

upgrade/hammer-jewel-x/stress-split: no xenial

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1211 from ceph/wip-15753-master

upgrade/hammer-jewel-x suite

upgrade/hammer-jewel-x: redo stress-split

Make this easy: write a singel yaml that does the hammer install,
some limited work, then upgardes to jewel.  Copy it from the
parallel suite.  Then, symlink all of the rest from the jewel-x
stress-split suite.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-jewel-x: misc fixes

upgrade/hammer-jewel-x: fix upgrade order

upgrade/hammer-jewel-x: no xenial

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-jewel-x: no mds, no fs tests

exclude ceph-mgr

upgrade/hammer-jewel-x/parallel: update

Merger hammer-x (jewel branch) and jewel-x (master branch).

Signed-off-by: Sage Weil <sage@redhat.com>

Added new hammer-jewel-x suite

Removed rbd/test_librbd_python.sh for now
Removed all `sudo` from exec lines

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1300 from batrick/multimds-merge

suites/multimds: sync with fs suites

Reviewed-by: John Spray <john.spray@redhat.com>

suites/multimds: add kclient disable override

The libcephfs tests are negatively affected by other mounts. This commit
adds a kclient disable in addition to the ceph-fuse one.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/multimds: sync with fs suites

This commit synchronizes the multimds suite with the fs suite. The
basic/verify sub-suites now do the same tests except with different
cluster layouts (i.e. multiple actives). This is mostly accomplished by
symlinking parts of each sub-suite to its counterpart in the fs suite.

This commit also does a few things of note to the prior multimds suite:

o Turn on directory fragmentation.
o Add several tests from fs/basic/tasks to multimds/basic.
o Remove libcephfs as fs/basic/tasks already contain
  multimds/basic/tasks.

Prior implementation and discussion are in PR#1114: https://github.com/ceph/ceph-qa-suite/pull/1114

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1282 from batrick/fs-reorg

fs: unify common parts of sub-suites

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1275 from jcsp/wip-17853

Directory fragmentation tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tasks/cephfs: add test_standby_for_invalid_fscid

Signed-off-by: Ramakrishnan Periyasamy <rperiyas@redhat.com>

Merge pull request #532 from ceph/wip-12378

Adds task to wrap the aver utility

Merge pull request #734 from dachary/wip-rados-thrash

openstack: rados/thrash...cache-pool-snaps.yaml: need 15GB

Merge pull request #1019 from ceph/wip-rgw-fix-longpath

rgw: split osd's in 2 nodes

Merge pull request #980 from ceph/wip-fix-infinit

Fix infinite wait during monitor quorum check

rgw: configure compression in default zone when not multisite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/verify: set compression type=random

instead of adding each compression type to the matrix for the rgw/verify
suite, use the new 'random' compression type to get test coverage of
the various compression plugins

this adds an 'rgw compression type' option to the client config, and later
adds that to the zone placement config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Fix infinite wait during monitor quorum check

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Split osd's into 2 nodes

http://tracker.ceph.com/issues/15612
Not all test machines(mira, vps) have 4 osd's and without this
change it will fail on nodes with < 4 osds, since one of the
osd will be on ext4 and it will hit long filename issue

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #535 from ceph/wip-12376

tasks/radosbenchsweep task

Merge pull request #1024 from ceph/wip-default-fs

tasks/ceph: default to xfs instead of root dir

Merge pull request #1087 from sponce/master

rados: Added tests for the striper layer of rados

Merge pull request #956 from ceph/wip-rm-ext4

Remove ext4 option from rados:thrash tests

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #1301 from ceph/wip-valgrind-centos

do all valgrind runs on centos

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

do all valgrind runs on centos

The xenial valgrind has a bug with generating random
numbers; see http://tracker.ceph.com/issues/18126

Revert this when it is fixed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1299 from athanatos/wip-enable-hacky-flag

ec-data-pool: enable the hacky_overwrites flag

ec-data-pool: enable the hacky_overwrites flag

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1298 from ceph/wip-pre-single-major-branch

krbd/unmap: pre-single-major is now 3.12.z

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rados/thrash/workloads: shorten rados bench a bit

I hit ENOSPC during thrashing on one run.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1295 from dachary/wip-16506-ceph-coverage

upgrade: ceph-test is needed for ceph-coverage

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

upgrade: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1294 from ceph/wip-17960-master

Removed xenial + Excluded librgw2

upgrade/jewel-x/point-to-point: avoid xenial

Xenial has newer packages in the distro and we want to
avoid fragile install instructions to avoid them.

Signed-off-by: Sage Weil <sage@redhat.com>

Excluded librgw2
Fixes http://tracker.ceph.com/issues/17960

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

krbd/unmap: pre-single-major is now 3.12.z

3.13 won't build on newer distros, due to lack of gcc5 support.
nightly/pre-single-major (with a slash) is now gone.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1293 from ceph/wip-17977-fix

qemu: Set permissions on /dev/kvm

Reviewed-by: Dan Mick <dmick@redhat.com>

qemu: Set permissions on /dev/kvm

Hack until fix in https://bugzilla.redhat.com/show_bug.cgi?id=1333159
makes it into CentOS repos.

Fixes: http://tracker.ceph.com/issues/17977
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1291 from ceph/wip-18085

rados/thrash*: do no set osd_max_backfills=1

Reviewed-by: Samuel Just <sjust@redhat.com>

rados/thrash*: vary osd_max_backfills, always >1

This can lead to a copy-from vs backfill deadlock; see
http://tracker.ceph.com/issues/18085

This effectively reverts 5e880228fd7f59063d22a51eb5488b369b0c8360
which aimed to catch bugs in recovery reservations.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #894 from ceph/wip-bluestore

rados: test bluestore too

rados/objectstore/objectstore.yaml: do not test kstore

It's flaky.

Signed-off-by: Sage Weil <sage@redhat.com>

rados/objectstore/objectstore.yaml: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: test on filestore and bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

fs: test bluestore and filestore

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: test bluestore and filestore

Signed-off-by: Sage Weil <sage@redhat.com>

rados: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1289 from ceph/wip-16397-workaround

suites/knfs: pin to ubuntu

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #1283 from SUSE/wip-18014

thrashosds: try ceph-objectstore-tool for one minute

Reviewed-by: Loic Dachary <ldachary@redhat.com>

suites/knfs: pin to ubuntu

Workaround for http://tracker.ceph.com/issues/16397, which
manifests when we run bleeding edge kernels against the version
of nfs-utils in centos7.

Signed-off-by: John Spray <john.spray@redhat.com>

thrashosds: try ceph-objectstore-tool for 10 minutes

If ceph-objectstore-tool binary is not present, it's likely because we're in
the middle of an upgrade. Do not try to run the binary until we verify that
it's really present. If it is absent, spend up to 10 minutes waiting for it to
appear.

Before this patch there was quite a large window for a race to occur. This
patch doesn't entirely eliminate it, but drastically reduces it.

Fixes: http://tracker.ceph.com/issues/18014
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1235 from vshankar/wip-rbd-data-pool

suites/rbd: add data pool test cases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1277 from ceph/wip-exclude-upgrade-clients

Excluded 'libcephfs2','libcephfs-devel','libcephfs-dev'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Excluded 'libcephfs2','libcephfs-devel','libcephfs-dev'
Removed firefly and infernalis paths
Removed mds role from hammer (per chat with Jason)
Add "fs: xfs" override

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rbd: add datapool config YAML to rbd suite

- cli
- librbd
- qemu

Signed-off-by: Venky Shankar <vshankar@redhat.com>

suites/fs: link to common tasks

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common fs

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common overrides

Goal is to reduce arbitrary differences between fs suites.

Note:
  o fs/{multiclient,multifs,recovery,snaps,standbyreplay} now use basic/overrides/whitelist_wrongly_marked_down.yaml.
  o fs/{snaps,standbyreplay} now use basic/overrides/debug.yaml.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common (fuse) mount

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify/reuse common cluster layout

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: unify install/start task creation

Goal is to reduce arbitrary differences between fs suites.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

verify: remove old debugging addition

Added in 7ae1aefa to help resolve http://tracker.ceph.com/issues/1737.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

suites/fs: add ceph-fuse disable override

Some tests require fuse mounts to not be unmounted. This avoids the need
to manually specify clients as unmounted.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1262 from batrick/cephfs-delete-bugfix

cephfs: get new fsmap after marking clusters down

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_fragment

Fixes: http://tracker.ceph.com/issues/17853
Signed-off-by: John Spray <john.spray@redhat.com>

tasks: mark start and end of tests in clog

This makes it much easier to associate e.g. slow
requests messages with which test was running.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: reinstate ctx.filesystem

This got lost in the multi-filesystem
refactor.  It's just a debug convenience.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix wait_until_true logging

The success log message was in the wrong
place (and unreachable)

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1274 from ceph/wip-16064

Improve scrub analysis, list-inconsistent-obj output and osd-scrub-repair test

Merge pull request #1260 from ceph/wip-17902-master

Added excluded packages "'libcephfs-devel','libcephfs-dev'"

Merge pull request #1227 from athanatos/wip-ec-overwrites

ec overwrites tests (and bonus fix to fast reads)

Merge pull request #1271 from SUSE/wip_flake8_F401_time_imported_but_unused

tasks.rgw: 'time' imported but unused

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks.rgw: 'time' imported but unused

flake8 was failing.

Signed-off-by: Owen Synge <osynge@suse.com>

Added excluded packadges "'libcephfs-devel','libcephfs-dev'"

Fixes http://tracker.ceph.com/issues/17902
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/scrub_test.py: Make test deterministic by updating digests

ceph_test_rados: Get coverage of digest updates in deep-scrub/repair

Signed-off-by: David Zafman <dzafman@redhat.com>

repair_test, scrub_test: Fix whitelists for scrub changes

Signed-off-by: David Zafman <dzafman@redhat.com>

scrub_test: Fix for list-inconsistent-obj output changes

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #1269 from ceph/wip-rgw-sync-agent-retry

rgw: fix some races with radosgw and radosgw-agent startup

(verified by Orit)

rgw: remove unnecessary sleeps

remove the sleeps that were added to address radosgw startup races

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start_rgw() polls gateway until it accepts connections

resolves various races between radosgw startup and further operations -
both within the rgw task itself (such as the 'radosgw-admin realm pull'),
and in later tasks

Fixes: http://tracker.ceph.com/issues/17794
Fixes: http://tracker.ceph.com/issues/17872
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add retry/backoff to sync agent requests

resolves an issue where startup of the radosgw-agent races with the
requests we send to it to run sync. uses the requests package with
urllib3 to add retry with backoff to these requests

Fixes: http://tracker.ceph.com/issues/16129
Signed-off-by: Casey Bodley <cbodley@redhat.com>

objectstore.yaml: raise fd limit

Signed-off-by: Samuel Just <sjust@redhat.com>

fast.yaml: overrides not override

Signed-off-by: Samuel Just <sjust@redhat.com>

thrash-erasure-code: disable ceph-objectstore-tool testing for now

Signed-off-by: Samuel Just <sjust@redhat.com>

rados,thrash-erasure-code: add overwrites workloads

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1263 from ceph/wip-cephfs-perm

rados/singleton-nomsgr/all/cache-fs-trunc: debug

cephfs: get new fsmap after marking clusters down

Credit to John Spray for identifying the problem/cause.

Fixes: http://tracker.ceph.com/issues/17894
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rados/singleton-nomsgr/all/cache-fs-trunc: debug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1258 from ceph/jewel-x-fixes

rados/upgrade: skip libcephfs-devel install

Reviewed-by: Dan Mick <dmick@redhat.com>

rados/upgrade: skip libcephfs-devel install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1253 from ceph/wip-vive-la-libcephfs-devel

packages: s/libcephfs2-devel/libcephfs-devel/