Merge pull request #637 from dachary/wip-openstack-ceph-deploy-hammer

ceph-deploy: hints for OpenStack

ceph-deploy: hints for OpenStack

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 3da86724c3db305f395bce5ba2175258294dadb6)

Merge pull request #636 from dachary/wip-distro-supported

hammer now only supports CentOS 7 and Ubuntu 14.04

hammer now only supports CentOS 7 and Ubuntu 14.04

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #631 from dachary/wip-isa-distro-hammer

erasure-code: isolate isa plugin arch constraint

erasure-code: isolate isa plugin arch constraint

Link the distro directory to the directory containing all supported
distros. Add the x86_64 arch contraint required by the isa plugin to an
isolated file that is combined with all jobs.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 65565de50dbc841cc2a8254317140131348b65d9)

Merge pull request #620 from ceph/wip-13366

Removed centos 6.5 from config, fixes #13366

Merge pull request #629 from dachary/wip-13454-openstack-flavor-hammer

buildpackages: implement notcmalloc flavor

buildpackages: implement notcmalloc flavor

http://tracker.ceph.com/issues/13454 Fixes: #13454

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit baa8ea327e136197072faffedbe23087c9db2cdb)

Merge pull request #625 from dachary/wip-buildpackages-submodules-hammer

buildpackages: get ceph submodules

buildpackages: get ceph submodules

Prior to v0.80.9, autogen.sh did not get submodules. Copy/paste the
submodule initialization from newer autogen.sh in common.sh so that
v0.80.8 and below can be rebuilt from sources. It does not hurt to
update the submodules twice.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 81f5c3ad0374039280030fba239bf46934584829)

Removed centos 6.5 from config, fixes #13366

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #617 from dachary/wip-buildpackages-centos-hammer

buildpackages: use _get_version instead of os_version

buildpackages: use _get_version instead of os_version

os_version is from the remote and will be 7.1.23 for CentOS 7
instead of the expected 7.0 for all 7.* CentOS.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #612 from ceph/wip-13376-buildpackages-upgrade-hammer

buildpackages: walk the whole config tree to find sha1 to build

buildpackages: enforce ceph-object-corpus canonical URL

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0045dfc8a5364d20cc4ea5f1f0b3947f05543af9)

buildpackages: walk the whole config tree to find sha1 to build

It is not enough to look for the first install task. In upgrade tests,
the install.upgrade task requires more packages to be built. In more
complicated tests using sequential and parallel tasks, the actual
install or install.upgrade task may be deeper in the config tree.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4d89c9e4dee3ededc403509010093a55be54a95a)

buildpackages: honour install priorities tags, branch, sha1

The install config may have contradicting tag/branch and sha1.  When
suite.py prepares the jobs, it always overrides the sha1 with whatever
default is provided on the command line with --distro and what is found
in the gitbuilder. If it turns out that the tag or the branch in the
install config task is about another sha1, it will override anyway.

Instead of obtaining the tag, branch and sha1 directly from the
packaging.GitbuilderProject object, compute them from the returned
uri_reference data member. The uri_reference is used by the install task
to fetch packages in the gitbuilders and this is what buildpackages
needs to build.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c103579a55cff3bd90e2d8d7b8df03bf29560961)

buildpackages: branches are found in refs/remotes/origin

Note in refs/heads which is populated for local branches only and not
after a fresh clone.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 810caaf55f2439efbc0bec7ff21332d40d4dff8e)

buildpackages: do not override the sha1

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit e54c916891a3a8b61a856773a70650f387e290a5)

buildpackages: allow for concurrent build if != sha1

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit a630da722008278969bcdbbdf8a1e26169653fa7)

buildpackages: prefer GitbuilderProject.os_{type,version} over config.

The config['os_type'] and config['os_version'] are not always set for a given
job (for instance, in the rbd suite). When a suite runs, it relies on
default values, depending on the target Operating System and internal,
hard coded values associating ubuntu to 14.04 etc.

Instead of using config['os_{type,version}'] use the GitbuilderProject
equivalent which is set with the appropriate defaults.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 9db757b811a08e07c8c68866af1d41aaf7641de1)

Merge pull request #610 from ceph/wip-13393

Removed rhel distro, fixes #13393 as rhel is not supported in sepia lab

Removed rhel distro, fixes #13393 as rhel is not supported in sepia lab

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #605 from dachary/wip-buildpackages-test-hammer

buildpackages: implicit for OpenStack

buildpackages: implicit for OpenStack

When using the teuthology-openstack command, buildpackages is implicit
and does not need to be included when running the test suite. Move the
stanza to the debug directory as a reminder when debuging.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f133725ba37d1f48b5f80fcf733264b60f788051)

Merge pull request #601 from dachary/wip-git-ceph-com

more hammer git ceph com updates

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a14a951640f79fd7ed217c423dcc9f453c830556)

Conflicts:
suites/rbd/qemu/workloads/qemu_bonnie.yaml
suites/rbd/qemu/workloads/qemu_fsstress.yaml
suites/rbd/qemu/workloads/qemu_iozone.yaml.disabled
suites/rbd/qemu/workloads/qemu_xfstests.yaml
suites/rbd/singleton/all/formatted-output.yaml
the conflicts are because https is used in master instead of http,
the rest is otherwise identical.

tasks/chef.py
obsolete, replaced by ansible.

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f062b171f4ab191986cea525c831573109aba2f9)

teuthology: some suites still use http://ceph.newdream.net

This probably redirects to http://ceph.com but ceph.newdream.net still appears in some places

http://tracker.ceph.com/issues/9922 Fixes: #9922

Signed-off-by: Armando Segnini <armando.segnini@telecom-bretagne.eu>
(cherry picked from commit 9ace238e874b45afbdcef97fe656d9a707e6105a)

Merge pull request #594 from dachary/wip-13250-ceph-deploy-branch-hammer

ceph-deploy: fix ceph-deploy-branch config options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-deploy: fix ceph-deploy-branch config options

The config paramter of download_ceph_deploy does not have a ceph-deploy
item, therefore the ceph-deploy-branch parameter is always assumed to be
master.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit fdcfb79b3c57765441a6d15bea8ef2fcbcebf5f8)

Merge pull request #582 from ceph/wip-buildpackages-hammer

hammer backport: buildpackages task

buildpackages: user-data must be per os-type/os-version

CentOS 6.5 needs to install a package and reboot to grow the root file
system. Instead of assuming a common user-data.txt file can fit all
Operating Systems, make one user data per os-type/os-version combination.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 05228e1e5c95654ce6e1c81f557002cb0c966a2f)

buildpackages: flock must --close

The process run by flock must not inherit the file descriptor because
this will cause the lock to be held forever should the command survive
the call to flock. This is for instance the case for the ssh-agent.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 204e27fecda583c312c35e905b6c864a5c8a1925)

buildpackages: get Ceph git url via get_ceph_git_url()

Instead of relying on git_base_url, use the get_ceph_git_url() to obtain
the URL of the Ceph repository to use with git clone. This allows the
user to override it via the git_ceph_url configuration option and the
--git-ceph-url command line option to teuthology-openstack.

http://tracker.ceph.com/issues/11883 Refs: #11883

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 78b2a9aa740e6780890944f90f8a2615f08e3d33)

buildpackages: make rpm packages

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7e3a271a89d4e7794a1ee547a3105a871f76f9f1)

buildpackages: build the repository and the packages

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit e38a6d69dfb25fae11902f537b6f2b752689236f)

buildpackages: re-use config from install task

So that the packages built match what the install task expects.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8c618e564501112fc5751888991aab87dac22fb2)

buildpackages: pep8 compliance

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d40d8ff50186a552cbc6be8649c660b1534835db)

buildpackages: build ceph packages and upload them

Import install from task.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0eeb441bb092f0ba7b4af866d58243a917c0d3f1)

buildpackages: build ceph packages and upload them

Build Ceph packages from source for the required revision, os_type,
os_version and architecture and upload them to the gitbuilder
repository.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 2b25080d4faf0ae9ddb0000808830cd4d5c7bea1)

Merge pull request #539 from ceph/wip-12715

Whitelisted "bad backtrace on dir ino" to fis #12715

Whitelisted "bad backtrace on dir ino" to fis #12715

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #507 from ceph/wip-12068-hammer

Moved point-to-point test to upgrade/hammer-x

Reviewed-by: Samuel Just <sjust@redhat.com>

Moved point-to-point test to upgrade/hammer-x

RE: http://tracker.ceph.com/issues/12068

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #502 from ceph/wip-dumplig-firefly-x-hammer

Fixes #12021 for upgrade/dumpling-firefly-x suite

Fixes #12021 for upgrade/dumpling-firefly-x suite
added "mon debug unsafe allow tier with nonempty snaps: true"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Fixes #12425, moved versions-steps-x suite from upgrade/firefly to upgrade/firefly-x
as the old location was wrong and never ran correctely to upgrade tp -x

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit d413124dddc85876e1da2c8d7acc1b3bfefc08a4)

Fixed s3test branch ref to hammer (firefly seemed incorrect)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Added config settings to resolve and test #12021
"mon debug unsafe allow tier with nonempty snaps: true"
(only for firefly-x, giant-x and upgrade/hammer/point-to-point running againts hammer branch)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit e09f761ac49d56cb1069d629943f90010790fa9a)

Drop chef from the teuthology suite

Chef has been replaced by ansible, which will be added to jobs at
schedule time.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit afe148820a96e503fec65b902cdbec33fb407273)

Drop chef from tgt suite

Chef has been replaced by ansible, which will be added to jobs at
schedule time.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 5b57b81e90393d42494c7b02fb263f1601421f16)

Merge pull request #494 from ceph/wip-hammer-2

Changed s3tests ref to firefly

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Changed s3tests ref to firefly
Per IRC chat by Yehuda, Josh and Dan:

"the purpose of firefly-original in general was for upgrades from prior point releases of firefly, where newer tests on the firefly branch would fail
it shouldn't be used if it's just a firefly->hammer upgrade though, the plain old firefly branch should be fine for that"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Fixes #11570
Removed test_cls_rbd.sh per Josh's suggestion
Removed 'branch: hammer' from 3-upgrade-sequence/upgrade-*

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 52fa9a32c8dd5cf527838cd92ca8eec312ab3d41)

Merge pull request #488 from ceph/hammer-11756

tasks/cephfs: mount fusectl before listing fuse connections

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: mount fusectl before listing fuse connections

Fixes: #11756
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit f6bba29fe9c54f6d4909833eab855450879ee192)

Merge pull request #482 from ceph/wip-fuse-umount-hammer

tasks/cephfs: fix/improve fuse force umount (again)

Removed rhel* configurations with a goal that they will be covered in the Octo lab
and won't be used by vps runs in Sepia lab

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit adfa1ee393c0ea0a36c0dfb8d7fda1a82f4b29ea)

tasks/cephfs: fix/improve fuse force umount (again)

teuthology helpfully escapes things for us so
the \; didn't need the backslash.  The logic
was still falling over in some cases too.

Additionally, make the FUSE /sys/ abort operation
more surgical by working out the connection name
of our own mount during mount().

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit fc4e90930b2c6ba0d395ae7d6edb8f8daba3fd64)

Merge pull request #469 from ceph/wip_remove_test_cls_rbd_2

Wip remove test_cls_rbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed test_cls_rbd.sh as it conflicts with giant osds

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Initial checking for #12023
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 029fdfd741a280bf5f2712fcd706f8d80a199782)

rbd: use formatted output test from the hammer branch

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Added script to schedule 1/14th part of rados

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit a7064a7b3d8a865dead3800a4d97bd70da6dec33)

Merge pull request #439 from ceph/wip-11594

Make sure that ulimits are adjusted for ceph-objectstore-tool

Make sure that ulimits are adjusted for ceph-objectstore-tool

Fixes: #11204
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit e6ce90fdb16dcda6902ecf7857d96a86d6e3fbe5)

Replaced work units with ceph-deploy/ceph-deploy_hello_world.sh re: #11547
Removed per Josh's suggestion fs as redundant and renamed rados to bacis

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit dcfd6a3b099fd0e14fbfd47a8678a411387e01da)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #432 from dachary/wip-11429-hammer

suites/rados: add test for 11429 (hammer)

Reviewed-by: Samuel Just <sjust@redhat.com>

Initial check in for upgrades/hammer stable upgrade suite

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 8c823e7813d23f71bb8a7586c968c3e035f1d4b2)

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

squash: use sleep instead of utility.sleep

(cherry picked from commit cc107525bca02ca44fabe8513203680e0d1e422b)

squash: use != not is not in radosbench

(cherry picked from commit 91bc0e91864903f6635879432e6dc7a0b36c76c2)

squash: remove utility, moved to teuthology.git

(cherry picked from commit 73577f1077dcfb059d7a96d7502a147a943cd261)

squash: ceph_manager: add utility_task doc string

(cherry picked from commit 2a60852a1d0b0b7980004cf262d4af8ea4011667)

suites/rados: add test for 11429

This patch also adds some convenience facilities for making
some of the ceph_manager methods into tasks usable from a
yaml file.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 015ed70f8a46a44e35433d1f701e37f68af31cf9)

Added hammer-client-x to the mix
Renamed dirs to better reflect what is being tested
Removed old dirs
Added logging
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 3c8730d035a90f221c05526983c40f0a11da1ba6)

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #430 from ceph/wip-11495-hammer

#11495/hammer: backport ceph_deploy task fix

Merge pull request #429 from dachary/wip-10712-hammer

tasks: fix intermittent failure in TestFlush

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/ceph_deploy: remove some dead code

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 99951f8150f1487c3bad47c2b199a137226055cb)

tasks/ceph_deploy: fix for multiple mons

Now that service IDs are modified during run, we have
to avoid repeatedly evaluating first_mon for where
to run ceph_deploy, as the answer will change.

Fixes: #11495
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit dbe2ad24212b9ff9a183d931402a49b544307e00)

tasks: fix intermittent failure in TestFlush

This was an overly strict success condition: the
flush operation doesn't promise to leave you an empty
journal, it promises that anything in the journal
before the flush will be flushed.

Fixes: #10712
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d20d9238cd59bca002d682cbcdc444e15e856de1)

knfs: remove the MDS debugging

It doesn't fit on the typica boxes with debugging, and we probably don't need
it for any NFS issues which do arise...

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 47fff708f936b7380446a4452a9f69f4d1c93c05)

Removed redundant tests issue #11520
Reduced nodes to be locked to two

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 8ab46b42c5ae387141917bb8eff35960e9c45699)

Merge pull request #424 from ceph/wip_ceph_deploy

Removed unneeded workloads

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed unneeded workloads

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #400 from ceph/wip_ceph_deploy

 Wip ceph deploy

Merge pull request #418 from ceph/hammer-10624

[hammer backport] task/samba: ignore return code of fuser/losf

task/samba: ignore return code of fuser/losf

if mount point is not used by anyone, both fuser and losf return 1

Fixes: #10624
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit c049387b7b0832e78a4a1370691798f726985799)

Merge pull request #397 from ceph/wip-11316-hammer

hammer: fix ceph-deploy task with CephFS (#11316)

tasks/ceph_deploy: configure CephFS

This test apparently had not been touched since
"fs new" was added.  In addition to calling
Filesystem.create:
 * modify the get_nodes_using_role
   function to modify ctx.cluster.remotes so that the
   service IDs match what ceph-deploy will set
 * log exceptions during ceph_deploy setup, as otherwise
   they can get lost if another exception occurs during
   teardown (so that it's all easier to debug).
 * default to passing --dev=master during install, so
   that we don't error out horribly when run without
   an explicit branch set (e.g. when run outside
   scheduled suite)

Fixes: #11316
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 7c2e6cdc6cd3d02969d3494d120b12daaa13efa8)

tasks/cephfs: cope with missing ctx.daemons attr

...so that we may be used with the ceph_deploy task
that doesn't set that up.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f36011051cabb8e7b06194c778e640e98d328e8a)

tasks/cephfs: tweak use of mon for admin commands

... s/mon_remote/admin_remote/ and allow caller to pass
in which remote they want to use for that.  Enables use
with ceph_deploy task which does not give admin keys
to mons.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8b61310ce3eedeb8ac60446933cd5db548de6434)

tasks/ceph: refactor legacy FS configuration check

Move up into Filesystem so that this can be used from
the ceph_deploy task as well.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit ea7c39222ae084c90b5a8510ce5af35a2f75a6e3)

suites/fs: update log whitelist

These ones had been passing because of #11177,
now that that is fixed they require an update.

Fixes: #11300
Signed-off-by: John Spray <john.spray@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit fa2a86196c3ed39c1726be689e42ed5d4aae197a)

Merge pull request #410 from ceph/wip-hammer-git

git.ceph.com (hammer)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

git.ceph.com (hammer)

Signed-off-by: Sage Weil <sage@redhat.com>

Reduced ammount of nodes from 4 to 3
Added logging
Changed replica "osd pool default size: 2"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

smoke: white marked down on python test

Backport: hammer
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 8cc4a5b0ed21b8ddef7e304d02e903d960b8bcfb)

Removed per #11070 and resolves #11043

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 8a0de9a8ba18b8761f1612e4cdc5a306d320e30d)