tasks/cephfs: fix race in test_sessionmap

Fixes: #13698
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #666 from ceph/wip-openstack-fixes

Added $4 for filter out "random.yaml\,async.yaml" tests for now

Added $4 for filter out "random.yaml\,async.yaml" tests for now

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #656 from ceph/wip-frag

fs: fix two frag_enable fragments

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #658 from dachary/wip-buildpackages-tags

buildpackages: fetch tags from the official Ceph repository

buildpackages: fetch tags from the official Ceph repository

A clone of Ceph is not automagically updated with the tags from the
official Ceph repository. For a pull request based on master, git
describe will use whatever tags existed at the time the clone was made,
unless the author pull them from the official Ceph repository and later
git push --tags them.

The output of git describe is used to name the packages and if the
official tags are not present, the packages will be incorrectly
named. For instance instead of 9.0.3-34 the packages could be named
0.87-8433 because the v0.87 tag is the most recent tag in the
repository. That confuses the install task that will fail with:

   'ceph version 0.87 was not installed, found 9.0.3.'

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #655 from dachary/wip-13450-openstack-rados

openstack: rados/thrash: allocate three disks, always

Reviewed-by: Kefu Chai <kchai@redhat.com>

fs: fix two frag_enable fragments

Badly formatted yaml

Signed-off-by: Sage Weil <sage@redhat.com>

openstack: rados/thrash: allocate three disks, always

The thrasher needs disk attached to run against xfs, ext4 or btrfs. And
some jobs use more disks and do not fit in 40GB.

http://tracker.ceph.com/issues/13450 Fixes: #13450

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #654 from dachary/wip-ceph-disk

ceph-disk: switch to using install / ceph

openstack: resource hint must contain an array

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: switch to using install / ceph

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph: log which ceph.conf file is written

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #622 from dachary/wip-13451-openstack-rados-objectstore

rados/objectstore/alloc-hint: need three attached disks

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #645 from ceph/wip-openstack-fixes

Added openstack config openstack.yaml

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Added openstack config openstack.yaml
Added +

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #648 from ceph/wip-f-h-x-offline

test offline upgrade from firefly -> x

Reviewed-by: David Zafman <dzafman@redhat.com>

upgrade/hammer-x/f-h-x-offline: test firefly -> x upgrade

This is an offline (osds all restart) upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #644 from ceph/wip-fix-f-h-x-branch

Added one more "write_append_excl: false"

Added one more "write_append_excl: false"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Added "write_append_excl: false"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 3a13ec04ce68bbc152fcde2a31c573bb6b454219)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #643 from ceph/wip-fix-f-h-x-branch

Added "write_append_excl: false"

Added "write_append_excl: false"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Removed osd.6 from roles to fix #13515

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 8ae31f15836d84dc886224ff8849b9bb3dd07e81)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #642 from ceph/wip-13515

Removed osd.6 from roles to fix #13515

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Removed osd.6 from roles to fix #13515

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #641 from ceph/wip-13485

Added write_append_excl: false to fix #13485

Added write_append_excl: false to fix #13485

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Removed /parallel/7-final-workload/rgw_swift.yaml as it was creating the same user as the s3tests run right before

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 64dcc28e14f43e68c9253621c4ae7479f9673381)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #640 from ceph/wip-fix-f-h-x-branch

Removed /parallel/7-final-workload/rgw_swift.yaml as it was creating …

Removed /parallel/7-final-workload/rgw_swift.yaml as it was creating the same user as the s3tests run right before

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #632 from ceph/wip-rbd-dir-fix

use cephtest as base dir during testing

Use cephtest as base dir for testing and let nuke cleanup in case of abrupt failures

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rgw: sleep before agent startup

For good measure.. it's racy.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/verify: allow rgw's to start up before starting agent

Startup is slow(er) under valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/multifs: run multipart (perl) test on ubuntu

no Net::Amazon::S3 on el7

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #635 from ceph/wip-cpan

rgw: do quota tests on ubuntu

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: do quota tests on ubuntu

The Amazon/S3.pm perl module from CPAN is not available as an RPM.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #630 from dachary/wip-13408-stress-split

upgrade/stress-split: needs only six OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #623 from dachary/wip-11260-openstack-singleton-nomsgr

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

Reviewed-by: Sage Weil <sage@redhat.com>

tasks/cephfs: fix TestClientLimits.test_client_oldest_tid

To get the health warning, first we need to make sure requests are
added to session's completed request list. Then we need to send an
extra request to MDS to trigger the code that generates the warning.

Fixes: #13437
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit b03ac0798058e543a80aaab88101314ab63dd22b)

Merge pull request #624 from ceph/wip-13437

Wip 13437 (tasks/cephfs: fix TestClientLimits.test_client_oldest_tid …)

Reviewed-by: John Spray <john.spray@redhat.com>

upgrade/stress-split: needs only six OSDs

The large number of OSDs is only needed for LRC or SHEC in some
cases. All other jobs will do fine with six OSDs.

http://tracker.ceph.com/issues/13408 Fixes: #13408

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #628 from dachary/wip-13454-openstack-flavor

buildpackages: implement notcmalloc flavor

buildpackages: implement notcmalloc flavor

http://tracker.ceph.com/issues/13454 Fixes: #13454

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #621 from dachary/wip-13449-admin-socket

tasks/admin_socket.py: wait 120 seconds instead of 60

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #619 from dachary/wip-buildpackages-submodules

buildpackages: get ceph submodules

tasks/cephfs: fix TestClientLimits.test_client_oldest_tid

To get the health warning, first we need to make sure requests are
added to session's completed request list. Then we need to send an
extra request to MDS to trigger the code that generates the warning.

Fixes: #13437
Signed-off-by: Yan, Zheng <zyan@redhat.com>

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

If given only 8GB RAM,  ceph_test_msgr may abort with buffer::bad_alloc.

http://tracker.ceph.com/issues/11260 Fixes: #11260

Signed-off-by: Loic Dachary <loic@dachary.org>

rados/objectstore/alloc-hint: need three attached disks

So they can be mkfs xfs.

http://tracker.ceph.com/issues/13451 Fixes: #13451

Signed-off-by: Loic Dachary <loic@dachary.org>

tasks/admin_socket.py: wait 120 seconds instead of 60

When running on virtual machines, it may take more than one minute for a
daemon to create the admin socket.

http://tracker.ceph.com/issues/13449 Fixes: #13449

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #618 from yehudasa/wip-infernalis-fix

radosgw_admin: skip log objects that start with obj_delete_at_hint

buildpackages: get ceph submodules

Prior to v0.80.9, autogen.sh did not get submodules. Copy/paste the
submodule initialization from newer autogen.sh in common.sh so that
v0.80.8 and below can be rebuilt from sources. It does not hurt to
update the submodules twice.

Signed-off-by: Loic Dachary <loic@dachary.org>

radosgw_admin: skip log objects that start with obj_delete_at_hint

when testing radosgw-admin log show --object, don't use these objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #616 from ceph/wip-buildpackages-centos

buildpackages: use _get_version instead of os_version

buildpackages: use _get_version instead of os_version

os_version is from the remote and will be 7.1.23 for CentOS 7
instead of the expected 7.0 for all 7.* CentOS.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #614 from ceph/wip-sam-qa-fixes

Wip sam qa fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

radosbench: use pool config even if create_pool is false

Signed-off-by: Samuel Just <sjust@redhat.com>

divergent_priors2: give divergent time to come up

Fixes: #13322
Signed-off-by: Samuel Just <sjust@redhat.com>

Specified 'hammer' branch

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 490dbb28ffa501d7041182332523eb5998d03777)

Merge pull request #613 from ceph/wip-fix-f-h-x-branch

Specified 'firefly' branch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Specified 'hammer' branch

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #609 from dachary/wip-13376-buildpackages-upgrade

buildpackages: honour install priorities tags, branch, sha1

buildpackages: enforce ceph-object-corpus canonical URL

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: walk the whole config tree to find sha1 to build

It is not enough to look for the first install task. In upgrade tests,
the install.upgrade task requires more packages to be built. In more
complicated tests using sequential and parallel tasks, the actual
install or install.upgrade task may be deeper in the config tree.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: honour install priorities tags, branch, sha1

The install config may have contradicting tag/branch and sha1.  When
suite.py prepares the jobs, it always overrides the sha1 with whatever
default is provided on the command line with --distro and what is found
in the gitbuilder. If it turns out that the tag or the branch in the
install config task is about another sha1, it will override anyway.

Instead of obtaining the tag, branch and sha1 directly from the
packaging.GitbuilderProject object, compute them from the returned
uri_reference data member. The uri_reference is used by the install task
to fetch packages in the gitbuilders and this is what buildpackages
needs to build.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: branches are found in refs/remotes/origin

Note in refs/heads which is populated for local branches only and not
after a fresh clone.

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: do not override the sha1

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: allow for concurrent build if != sha1

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: prefer GitbuilderProject.os_{type,version} over config.

The config['os_type'] and config['os_version'] are not always set for a given
job (for instance, in the rbd suite). When a suite runs, it relies on
default values, depending on the target Operating System and internal,
hard coded values associating ubuntu to 14.04 etc.

Instead of using config['os_{type,version}'] use the GitbuilderProject
equivalent which is set with the appropriate defaults.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #589 from ceph/wip-ecpool-s3test

s3tests using ec data pool for smoke

Merge remote-tracking branch 'origin/wip-vstart-runner' into infernalis

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

smoke/basic/tasks/libcephfs: fix client debug override

Also debug mds

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Conflicts:
suites/upgrade/hammer-x/point-to-point-x/point-to-point.yaml

upgrade/hammer-x/point-to-point: whitelist wrongly marked me down

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #585 from dachary/wip-buildpackages-test

buildpackages: implicit for OpenStack

Merge pull request #596 from ceph/wip-radosbench

tasks/radosbench: wait just a bit longer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

White listed WAR 'failed to encode' fixes #13349

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 37a881aee191038139b37153a25555fc6b7f8fed)

Merge pull request #602 from ceph/wip13349

White listed WAR 'failed to encode' fixes #13349

White listed WAR 'failed to encode' fixes #13349

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #549 from ceph/wip-krbd-fio

Add rbd fio test to exercise block IO on all format/features

upgrade/hammer-x/point-to-point: use different client for second rgw

Can't do 2 rgw's on the same client id.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a0f0194ab2edec81f19b111b9317a5149defd4e9)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

krbd fio tests using sync engine for format 2 and features 1 and 2.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

krbd test using fio for format 2 and features 1 and 2.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rbd fio tests using rbd engine, tests on clones as well using features 1 and 13.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rbd io tests using fio, various fio options can be specified in yaml

test rbd or krbd using fio, can also run io on rbd clones if option is specified in yaml
various options like image-size, rbd format/features, fio io size, readwrite options can be provided in yaml.
check the docstring for exact usage.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #599 from ceph/wip-hammer-x

upgrade/hammer-x/point-to-point: use different client for second rgw

upgrade/hammer-x/point-to-point: use different client for second rgw

Can't do 2 rgw's on the same client id.

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/cephfs: fix FuseMount bin path in vstart

FuseMount only uses the prefix for finding the 'ceph'
executable, which is in ./ for either cmake or
authtools, not ./src for cmake like other binaries.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix test_journal_migration

It was trying to get the output file from
a different remote than the one used to
run the journal tool.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: move mds_scrub_checks

...into a CephFSTestCase.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: move journal migration test

...into a CephFSTestCase.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: extend vstart_runner's ctx&run

Sufficiently to enable using workunits.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add --interactive for vstart runner

Just like interactive-on-error in teuthology.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix FuseMount._asok_path

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: remove a redundant sudo

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: raise error on non-string stdins

Shouldn't be any from the fs tests that get run

Signed-off-by: John Spray <john.spray@redhat.com>