buildpackages: silently ignore IPv6 addresses

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7b8f205952d5e49609efbd77a1becae3aff4e531)

Merge pull request #671 from ceph/wip-13234-hammer

rados: add test for 13234.yaml

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #653 from ceph/wip-13567-hammer

suites/rbd/qemu: excercise CoW support during QEMU testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #606 from dachary/wip-12320-ceph-fuse-hammer

tasks/ceph_fuse.py: virtual machines need flexible mount timeout

Merge pull request #659 from dachary/wip-buildpackages-tags-hammer

buildpackages: fetch tags from the official Ceph repository

buildpackages: fetch tags from the official Ceph repository

A clone of Ceph is not automagically updated with the tags from the
official Ceph repository. For a pull request based on master, git
describe will use whatever tags existed at the time the clone was made,
unless the author pull them from the official Ceph repository and later
git push --tags them.

The output of git describe is used to name the packages and if the
official tags are not present, the packages will be incorrectly
named. For instance instead of 9.0.3-34 the packages could be named
0.87-8433 because the v0.87 tag is the most recent tag in the
repository. That confuses the install task that will fail with:

   'ceph version 0.87 was not installed, found 9.0.3.'

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 970a37226f884c7f109e48c90373b7fd2697f6d0)

Merge pull request #593 from dachary/wip-13000-centos-activate-hammer

ceph-deploy: always zap disk before creating an osd

suites/rbd/qemu: excercise CoW support during QEMU testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 59e768bb29960b4ca411a71342ef982bb2499354)

Merge pull request #651 from ceph/wip-fix-hammer-stable

Added v0.94.4 to the mix

Added v0.94.4 to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #650 from ceph/wip-openstack-fixes-fs_hammer

Added openstack.yaml config

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #626 from ceph/wip-13450-openstack-rados-thrash

openstack: rados/thrash: allocate three disks, always

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #627 from dachary/wip-13449-admin-socket-hammer

tasks/admin_socket.py: wait 120 seconds instead of 60

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #634 from dachary/wip-11260-openstack-singleton-nomsgr-hammer

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

Reviewed-by: Kefu Chai <kchai@redhat.com>

Added openstack.yaml config
Moved it to tasks/cfuse_workunit_kernel_untar_build.yaml

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #637 from dachary/wip-openstack-ceph-deploy-hammer

ceph-deploy: hints for OpenStack

ceph-deploy: hints for OpenStack

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 3da86724c3db305f395bce5ba2175258294dadb6)

Merge pull request #636 from dachary/wip-distro-supported

hammer now only supports CentOS 7 and Ubuntu 14.04

hammer now only supports CentOS 7 and Ubuntu 14.04

Signed-off-by: Loic Dachary <loic@dachary.org>

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

If given only 8GB RAM,  ceph_test_msgr may abort with buffer::bad_alloc.

http://tracker.ceph.com/issues/11260 Fixes: #11260

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit eac791c664ceed4bd3ac41b0d81ffc337b55e047)

Merge pull request #631 from dachary/wip-isa-distro-hammer

erasure-code: isolate isa plugin arch constraint

erasure-code: isolate isa plugin arch constraint

Link the distro directory to the directory containing all supported
distros. Add the x86_64 arch contraint required by the isa plugin to an
isolated file that is combined with all jobs.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 65565de50dbc841cc2a8254317140131348b65d9)

Merge pull request #620 from ceph/wip-13366

Removed centos 6.5 from config, fixes #13366

Merge pull request #629 from dachary/wip-13454-openstack-flavor-hammer

buildpackages: implement notcmalloc flavor

buildpackages: implement notcmalloc flavor

http://tracker.ceph.com/issues/13454 Fixes: #13454

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit baa8ea327e136197072faffedbe23087c9db2cdb)

tasks/admin_socket.py: wait 120 seconds instead of 60

When running on virtual machines, it may take more than one minute for a
daemon to create the admin socket.

http://tracker.ceph.com/issues/13449 Fixes: #13449

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c0828cae19a49d73de02e5cc4a1ac51403b98dec)

openstack: rados/thrash: allocate three disks, always

The thrasher needs disk attached to run against xfs, ext4 or btrfs. And
some jobs use more disks and do not fit in 40GB.

http://tracker.ceph.com/issues/13450 Fixes: #13450

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #625 from dachary/wip-buildpackages-submodules-hammer

buildpackages: get ceph submodules

buildpackages: get ceph submodules

Prior to v0.80.9, autogen.sh did not get submodules. Copy/paste the
submodule initialization from newer autogen.sh in common.sh so that
v0.80.8 and below can be rebuilt from sources. It does not hurt to
update the submodules twice.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 81f5c3ad0374039280030fba239bf46934584829)

Removed centos 6.5 from config, fixes #13366

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #617 from dachary/wip-buildpackages-centos-hammer

buildpackages: use _get_version instead of os_version

buildpackages: use _get_version instead of os_version

os_version is from the remote and will be 7.1.23 for CentOS 7
instead of the expected 7.0 for all 7.* CentOS.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #612 from ceph/wip-13376-buildpackages-upgrade-hammer

buildpackages: walk the whole config tree to find sha1 to build

buildpackages: enforce ceph-object-corpus canonical URL

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0045dfc8a5364d20cc4ea5f1f0b3947f05543af9)

buildpackages: walk the whole config tree to find sha1 to build

It is not enough to look for the first install task. In upgrade tests,
the install.upgrade task requires more packages to be built. In more
complicated tests using sequential and parallel tasks, the actual
install or install.upgrade task may be deeper in the config tree.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4d89c9e4dee3ededc403509010093a55be54a95a)

buildpackages: honour install priorities tags, branch, sha1

The install config may have contradicting tag/branch and sha1.  When
suite.py prepares the jobs, it always overrides the sha1 with whatever
default is provided on the command line with --distro and what is found
in the gitbuilder. If it turns out that the tag or the branch in the
install config task is about another sha1, it will override anyway.

Instead of obtaining the tag, branch and sha1 directly from the
packaging.GitbuilderProject object, compute them from the returned
uri_reference data member. The uri_reference is used by the install task
to fetch packages in the gitbuilders and this is what buildpackages
needs to build.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c103579a55cff3bd90e2d8d7b8df03bf29560961)

buildpackages: branches are found in refs/remotes/origin

Note in refs/heads which is populated for local branches only and not
after a fresh clone.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 810caaf55f2439efbc0bec7ff21332d40d4dff8e)

buildpackages: do not override the sha1

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit e54c916891a3a8b61a856773a70650f387e290a5)

buildpackages: allow for concurrent build if != sha1

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit a630da722008278969bcdbbdf8a1e26169653fa7)

buildpackages: prefer GitbuilderProject.os_{type,version} over config.

The config['os_type'] and config['os_version'] are not always set for a given
job (for instance, in the rbd suite). When a suite runs, it relies on
default values, depending on the target Operating System and internal,
hard coded values associating ubuntu to 14.04 etc.

Instead of using config['os_{type,version}'] use the GitbuilderProject
equivalent which is set with the appropriate defaults.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 9db757b811a08e07c8c68866af1d41aaf7641de1)

Merge pull request #610 from ceph/wip-13393

Removed rhel distro, fixes #13393 as rhel is not supported in sepia lab

Removed rhel distro, fixes #13393 as rhel is not supported in sepia lab

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/cephfs: fix mount_timeout config loading

Previously were defaulting to a string, which
always compared greater than elapsed, so never
timed out.

Fixes: #12820
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8e275049a0a8b976fc9eb5c63dcefe01bedcc711)

tasks/ceph_fuse.py: virtual machines need flexible mount timeout

1) add a wait time before the mount attempt to let the cluster get set up.
By default this should be skipped, but for VMs and known-slow systems we
can give them 60 seconds.
2) Make the timeout configurable, with a 30-second default, but override it
for VM tests.

http://tracker.ceph.com/issues/12320 Fixes: #12320

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit ec12f21a7ead9d82b575c4f927d5e7b73f85c3d5)

Merge pull request #605 from dachary/wip-buildpackages-test-hammer

buildpackages: implicit for OpenStack

buildpackages: implicit for OpenStack

When using the teuthology-openstack command, buildpackages is implicit
and does not need to be included when running the test suite. Move the
stanza to the debug directory as a reminder when debuging.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f133725ba37d1f48b5f80fcf733264b60f788051)

Merge pull request #601 from dachary/wip-git-ceph-com

more hammer git ceph com updates

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a14a951640f79fd7ed217c423dcc9f453c830556)

Conflicts:
suites/rbd/qemu/workloads/qemu_bonnie.yaml
suites/rbd/qemu/workloads/qemu_fsstress.yaml
suites/rbd/qemu/workloads/qemu_iozone.yaml.disabled
suites/rbd/qemu/workloads/qemu_xfstests.yaml
suites/rbd/singleton/all/formatted-output.yaml
the conflicts are because https is used in master instead of http,
the rest is otherwise identical.

tasks/chef.py
obsolete, replaced by ansible.

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f062b171f4ab191986cea525c831573109aba2f9)

teuthology: some suites still use http://ceph.newdream.net

This probably redirects to http://ceph.com but ceph.newdream.net still appears in some places

http://tracker.ceph.com/issues/9922 Fixes: #9922

Signed-off-by: Armando Segnini <armando.segnini@telecom-bretagne.eu>
(cherry picked from commit 9ace238e874b45afbdcef97fe656d9a707e6105a)

Merge pull request #594 from dachary/wip-13250-ceph-deploy-branch-hammer

ceph-deploy: fix ceph-deploy-branch config options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-deploy: fix ceph-deploy-branch config options

The config paramter of download_ceph_deploy does not have a ceph-deploy
item, therefore the ceph-deploy-branch parameter is always assumed to be
master.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit fdcfb79b3c57765441a6d15bea8ef2fcbcebf5f8)

ceph-deploy: always zap disk before creating an osd

The existing logic is to ceph-deploy osd create --zap-disk which will
zap the data device before preparing it. However it will not zap the
journal device (see http://tracker.ceph.com/issues/13291).

If ceph-deploy osd create fails, a fall back will zap both the data
device and the journal and try prepare again. This could work if
the device preparation and activation was synchronous and catch all
errors that could be caused by an unclean journal device. However,
the activation is asynchronous and it is entirely possible for a device
to be prepared successfully and fail to activate in the background.

The data and journal device are always zapped before calling ceph-deploy
osd create. The logic is simpler and the overhead is low.

http://tracker.ceph.com/issues/13000 Fixes: #13000

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 01d48a270ad526492dccbb0e363162f0befb0f9e)

rados: add test for 13234.yaml

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #582 from ceph/wip-buildpackages-hammer

hammer backport: buildpackages task

buildpackages: user-data must be per os-type/os-version

CentOS 6.5 needs to install a package and reboot to grow the root file
system. Instead of assuming a common user-data.txt file can fit all
Operating Systems, make one user data per os-type/os-version combination.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 05228e1e5c95654ce6e1c81f557002cb0c966a2f)

buildpackages: flock must --close

The process run by flock must not inherit the file descriptor because
this will cause the lock to be held forever should the command survive
the call to flock. This is for instance the case for the ssh-agent.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 204e27fecda583c312c35e905b6c864a5c8a1925)

buildpackages: get Ceph git url via get_ceph_git_url()

Instead of relying on git_base_url, use the get_ceph_git_url() to obtain
the URL of the Ceph repository to use with git clone. This allows the
user to override it via the git_ceph_url configuration option and the
--git-ceph-url command line option to teuthology-openstack.

http://tracker.ceph.com/issues/11883 Refs: #11883

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 78b2a9aa740e6780890944f90f8a2615f08e3d33)

buildpackages: make rpm packages

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7e3a271a89d4e7794a1ee547a3105a871f76f9f1)

buildpackages: build the repository and the packages

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit e38a6d69dfb25fae11902f537b6f2b752689236f)

buildpackages: re-use config from install task

So that the packages built match what the install task expects.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8c618e564501112fc5751888991aab87dac22fb2)

buildpackages: pep8 compliance

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d40d8ff50186a552cbc6be8649c660b1534835db)

buildpackages: build ceph packages and upload them

Import install from task.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0eeb441bb092f0ba7b4af866d58243a917c0d3f1)

buildpackages: build ceph packages and upload them

Build Ceph packages from source for the required revision, os_type,
os_version and architecture and upload them to the gitbuilder
repository.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 2b25080d4faf0ae9ddb0000808830cd4d5c7bea1)

Merge pull request #539 from ceph/wip-12715

Whitelisted "bad backtrace on dir ino" to fis #12715

Whitelisted "bad backtrace on dir ino" to fis #12715

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #507 from ceph/wip-12068-hammer

Moved point-to-point test to upgrade/hammer-x

Reviewed-by: Samuel Just <sjust@redhat.com>

Moved point-to-point test to upgrade/hammer-x

RE: http://tracker.ceph.com/issues/12068

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #502 from ceph/wip-dumplig-firefly-x-hammer

Fixes #12021 for upgrade/dumpling-firefly-x suite

Fixes #12021 for upgrade/dumpling-firefly-x suite
added "mon debug unsafe allow tier with nonempty snaps: true"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Fixes #12425, moved versions-steps-x suite from upgrade/firefly to upgrade/firefly-x
as the old location was wrong and never ran correctely to upgrade tp -x

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit d413124dddc85876e1da2c8d7acc1b3bfefc08a4)

Fixed s3test branch ref to hammer (firefly seemed incorrect)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Added config settings to resolve and test #12021
"mon debug unsafe allow tier with nonempty snaps: true"
(only for firefly-x, giant-x and upgrade/hammer/point-to-point running againts hammer branch)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit e09f761ac49d56cb1069d629943f90010790fa9a)

Drop chef from the teuthology suite

Chef has been replaced by ansible, which will be added to jobs at
schedule time.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit afe148820a96e503fec65b902cdbec33fb407273)

Drop chef from tgt suite

Chef has been replaced by ansible, which will be added to jobs at
schedule time.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 5b57b81e90393d42494c7b02fb263f1601421f16)

Merge pull request #494 from ceph/wip-hammer-2

Changed s3tests ref to firefly

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Changed s3tests ref to firefly
Per IRC chat by Yehuda, Josh and Dan:

"the purpose of firefly-original in general was for upgrades from prior point releases of firefly, where newer tests on the firefly branch would fail
it shouldn't be used if it's just a firefly->hammer upgrade though, the plain old firefly branch should be fine for that"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Fixes #11570
Removed test_cls_rbd.sh per Josh's suggestion
Removed 'branch: hammer' from 3-upgrade-sequence/upgrade-*

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 52fa9a32c8dd5cf527838cd92ca8eec312ab3d41)

Merge pull request #488 from ceph/hammer-11756

tasks/cephfs: mount fusectl before listing fuse connections

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: mount fusectl before listing fuse connections

Fixes: #11756
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit f6bba29fe9c54f6d4909833eab855450879ee192)

Merge pull request #482 from ceph/wip-fuse-umount-hammer

tasks/cephfs: fix/improve fuse force umount (again)

Removed rhel* configurations with a goal that they will be covered in the Octo lab
and won't be used by vps runs in Sepia lab

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit adfa1ee393c0ea0a36c0dfb8d7fda1a82f4b29ea)

tasks/cephfs: fix/improve fuse force umount (again)

teuthology helpfully escapes things for us so
the \; didn't need the backslash.  The logic
was still falling over in some cases too.

Additionally, make the FUSE /sys/ abort operation
more surgical by working out the connection name
of our own mount during mount().

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit fc4e90930b2c6ba0d395ae7d6edb8f8daba3fd64)

Merge pull request #469 from ceph/wip_remove_test_cls_rbd_2

Wip remove test_cls_rbd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed test_cls_rbd.sh as it conflicts with giant osds

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Initial checking for #12023
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 029fdfd741a280bf5f2712fcd706f8d80a199782)

rbd: use formatted output test from the hammer branch

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Added script to schedule 1/14th part of rados

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit a7064a7b3d8a865dead3800a4d97bd70da6dec33)

Merge pull request #439 from ceph/wip-11594

Make sure that ulimits are adjusted for ceph-objectstore-tool

Make sure that ulimits are adjusted for ceph-objectstore-tool

Fixes: #11204
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit e6ce90fdb16dcda6902ecf7857d96a86d6e3fbe5)

Replaced work units with ceph-deploy/ceph-deploy_hello_world.sh re: #11547
Removed per Josh's suggestion fs as redundant and renamed rados to bacis

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit dcfd6a3b099fd0e14fbfd47a8678a411387e01da)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #432 from dachary/wip-11429-hammer

suites/rados: add test for 11429 (hammer)

Reviewed-by: Samuel Just <sjust@redhat.com>

Initial check in for upgrades/hammer stable upgrade suite

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 8c823e7813d23f71bb8a7586c968c3e035f1d4b2)

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

squash: use sleep instead of utility.sleep

(cherry picked from commit cc107525bca02ca44fabe8513203680e0d1e422b)

squash: use != not is not in radosbench

(cherry picked from commit 91bc0e91864903f6635879432e6dc7a0b36c76c2)

squash: remove utility, moved to teuthology.git

(cherry picked from commit 73577f1077dcfb059d7a96d7502a147a943cd261)

squash: ceph_manager: add utility_task doc string

(cherry picked from commit 2a60852a1d0b0b7980004cf262d4af8ea4011667)

suites/rados: add test for 11429

This patch also adds some convenience facilities for making
some of the ceph_manager methods into tasks usable from a
yaml file.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 015ed70f8a46a44e35433d1f701e37f68af31cf9)

Added hammer-client-x to the mix
Renamed dirs to better reflect what is being tested
Removed old dirs
Added logging
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 3c8730d035a90f221c05526983c40f0a11da1ba6)

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #430 from ceph/wip-11495-hammer

#11495/hammer: backport ceph_deploy task fix