Merge pull request #720 from dachary/wip-12748-hammer

rados/singleton-nomsgr/all: add export-after-evict.yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #724 from dachary/wip-openstack-rbd-hammer

openstack: rbd/thrash: allocate three disks, always

openstack: rbd/{thrash,qemu}: allocate three disks, always

The thrasher and qemu need disk attached to run against xfs or btrfs,
otherwise it will use the local file system which is always ext4. It
will succeed but this is a kind of false positive since no xfs or btrfs
related tests were run.

http://tracker.ceph.com/issues/13270 Refs: #13270

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 29973822f8f21044026055a5fd216c3a96ba7eaf)

Merge pull request #722 from dachary/wip-13581-openstack-hints-hammer

openstack: convert to array hints

openstack: convert to array hints (hammer specific)

So that it is more flexible.

http://tracker.ceph.com/issues/13581 Fixes: #13581

Signed-off-by: Loic Dachary <loic@dachary.org>

openstack: convert to array hints

So that it is more flexible.

http://tracker.ceph.com/issues/13581 Fixes: #13581

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8f2af5949c8ed4e24cedd8a6df0d169733e14926)

Conflicts:
suites/smoke/1node/clusters/openstack.yaml (does not exist)

rados/singleton-nomsgr/all: add export-after-evict.yaml

add a test for http://tracker.ceph.com/issues/12748

Fixes: #12748
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a3442cf07841a7269713e30ca270f2f9c356a598)

Merge pull request #717 from dachary/wip-dumpling-parallel-hammer

buildpackages: do not -jX on dumpling

buildpackages: do not -jX on dumpling

http://tracker.ceph.com/issues/13845 Refs: #13845

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f398b71e339b4617db180c8140f23d004f445a24)

Merge pull request #713 from dachary/wip-buildpackages-refactor-hammer

  buildpackages: implement get_pkg_type & avoid duplicate packages-repository

buildpackages: avoid duplicate packages-repository

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 87e50d394a26682614ab3ec8f566bda9f30e8416)

buildpackages: implement get_pkg_type

It can be deprectated when it is implemented in teuthology.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 13fb23ed69e6b7dfa5c005e81abc6b983c019ba4)

Merge pull request #711 from dachary/wip-buildpackages-refactor-hammer

buildpackages: refactor to not require remote

buildpackages: refactor to not require remote

Most of the flavor, sha1, tag etc. selection logic as implemented in the
packaging module of teuthology relies on remote hosts. This is complex
to tests and inconvenient because hosts must be provisionned even before
trying to figure out which packages need to be installed.

Using remote hosts is necessary when bare metal targets are used because
teuthology must adapt to the operating system already installed. The
selection logic in the context of dynamically provisionned targets is
simpler because it is defined by the job being run.

The buildpackages is refactored to use only the job configuration to
figure out which packages must be built. It makes it specific to targets
that are dynamically provisionned. It would have to be modified to query
the remote host in the case of bare metal targets.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 69e9b22e4baa06705119cc818f3b2c1a472c31f6)

Merge pull request #705 from dachary/wip-buildpackages-packaging-hammer

buildpackages: protect packages-repository

buildpackages: protect packages-repository

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4825f64a7c22320b034b020093489bf86a953fca)

Merge pull request #702 from dachary/wip-buildpackages-giant-hammer

buildpackages: do not fails for unpackaged files

buildpackages: do not fails for unpackaged files

Older versions of Ceph have that problem failing would break all upgrade suites.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8d798c4eec508a5e2433fc55ab072c9105855a18)

Merge pull request #696 from dachary/wip-buildpackages-flavor-hammer

buildpackages: 10GB is too small most of the time

buildpackages: 10GB is too small most of the time

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 129c5db6c9bc41faa5c52df340776f00ab2db2b6)

Merge pull request #685 from dachary/wip-12973-hammer

failed to encode map e428 with expected crc

buildpackages: giant has some unpackaged files

And there really is nothing we can do about it expect ignore them.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados: 'failed to encode ...' warnings are normal on upgrades (part 2)

http://tracker.ceph.com/issues/12973 Fixes: #12973

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4cbf1770397e4c97aa1d0486ac7f7a3e1da98771)

Conflicts:
suites/upgrade/hammer/point-to-point/point-to-point.yaml does not exist in hammer

rados: 'failed to encode ...' warnings are normal on upgrades

Fixes: #12973
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3084453b047ca8b6bd0ce56e4e5a7ecce71753bc)

Conflicts:
suites/rados/upgrade/hammer-x-singleton/3-thrash/default.yaml does not exist in hammer

Merge pull request #683 from dachary/wip-13451-hammer

rados/objectstore/alloc-hint: need three attached disks

rados/objectstore/alloc-hint: need three attached disks

So they can be mkfs xfs.

http://tracker.ceph.com/issues/13451 Fixes: #13451

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1a63fd73de3c3de8cff16d486496b1fc5c4cd3b4)

Merge pull request #673 from dachary/wip-buildpackage-ipv6-hammer

buildpackages: silently ignore IPv6 addresses

buildpackages: silently ignore IPv6 addresses

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7b8f205952d5e49609efbd77a1becae3aff4e531)

Merge pull request #671 from ceph/wip-13234-hammer

rados: add test for 13234.yaml

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #653 from ceph/wip-13567-hammer

suites/rbd/qemu: excercise CoW support during QEMU testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #606 from dachary/wip-12320-ceph-fuse-hammer

tasks/ceph_fuse.py: virtual machines need flexible mount timeout

Merge pull request #659 from dachary/wip-buildpackages-tags-hammer

buildpackages: fetch tags from the official Ceph repository

buildpackages: fetch tags from the official Ceph repository

A clone of Ceph is not automagically updated with the tags from the
official Ceph repository. For a pull request based on master, git
describe will use whatever tags existed at the time the clone was made,
unless the author pull them from the official Ceph repository and later
git push --tags them.

The output of git describe is used to name the packages and if the
official tags are not present, the packages will be incorrectly
named. For instance instead of 9.0.3-34 the packages could be named
0.87-8433 because the v0.87 tag is the most recent tag in the
repository. That confuses the install task that will fail with:

   'ceph version 0.87 was not installed, found 9.0.3.'

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 970a37226f884c7f109e48c90373b7fd2697f6d0)

Merge pull request #593 from dachary/wip-13000-centos-activate-hammer

ceph-deploy: always zap disk before creating an osd

suites/rbd/qemu: excercise CoW support during QEMU testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 59e768bb29960b4ca411a71342ef982bb2499354)

Merge pull request #651 from ceph/wip-fix-hammer-stable

Added v0.94.4 to the mix

Added v0.94.4 to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #650 from ceph/wip-openstack-fixes-fs_hammer

Added openstack.yaml config

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #626 from ceph/wip-13450-openstack-rados-thrash

openstack: rados/thrash: allocate three disks, always

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #627 from dachary/wip-13449-admin-socket-hammer

tasks/admin_socket.py: wait 120 seconds instead of 60

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #634 from dachary/wip-11260-openstack-singleton-nomsgr-hammer

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

Reviewed-by: Kefu Chai <kchai@redhat.com>

Added openstack.yaml config
Moved it to tasks/cfuse_workunit_kernel_untar_build.yaml

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #637 from dachary/wip-openstack-ceph-deploy-hammer

ceph-deploy: hints for OpenStack

ceph-deploy: hints for OpenStack

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 3da86724c3db305f395bce5ba2175258294dadb6)

Merge pull request #636 from dachary/wip-distro-supported

hammer now only supports CentOS 7 and Ubuntu 14.04

hammer now only supports CentOS 7 and Ubuntu 14.04

Signed-off-by: Loic Dachary <loic@dachary.org>

rados/singleton-nomsgr/all/msgr: needs 15GB RAM

If given only 8GB RAM,  ceph_test_msgr may abort with buffer::bad_alloc.

http://tracker.ceph.com/issues/11260 Fixes: #11260

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit eac791c664ceed4bd3ac41b0d81ffc337b55e047)

Merge pull request #631 from dachary/wip-isa-distro-hammer

erasure-code: isolate isa plugin arch constraint

erasure-code: isolate isa plugin arch constraint

Link the distro directory to the directory containing all supported
distros. Add the x86_64 arch contraint required by the isa plugin to an
isolated file that is combined with all jobs.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 65565de50dbc841cc2a8254317140131348b65d9)

Merge pull request #620 from ceph/wip-13366

Removed centos 6.5 from config, fixes #13366

Merge pull request #629 from dachary/wip-13454-openstack-flavor-hammer

buildpackages: implement notcmalloc flavor

buildpackages: implement notcmalloc flavor

http://tracker.ceph.com/issues/13454 Fixes: #13454

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit baa8ea327e136197072faffedbe23087c9db2cdb)

tasks/admin_socket.py: wait 120 seconds instead of 60

When running on virtual machines, it may take more than one minute for a
daemon to create the admin socket.

http://tracker.ceph.com/issues/13449 Fixes: #13449

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c0828cae19a49d73de02e5cc4a1ac51403b98dec)

openstack: rados/thrash: allocate three disks, always

The thrasher needs disk attached to run against xfs, ext4 or btrfs. And
some jobs use more disks and do not fit in 40GB.

http://tracker.ceph.com/issues/13450 Fixes: #13450

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #625 from dachary/wip-buildpackages-submodules-hammer

buildpackages: get ceph submodules

buildpackages: get ceph submodules

Prior to v0.80.9, autogen.sh did not get submodules. Copy/paste the
submodule initialization from newer autogen.sh in common.sh so that
v0.80.8 and below can be rebuilt from sources. It does not hurt to
update the submodules twice.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 81f5c3ad0374039280030fba239bf46934584829)

Removed centos 6.5 from config, fixes #13366

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #617 from dachary/wip-buildpackages-centos-hammer

buildpackages: use _get_version instead of os_version

buildpackages: use _get_version instead of os_version

os_version is from the remote and will be 7.1.23 for CentOS 7
instead of the expected 7.0 for all 7.* CentOS.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #612 from ceph/wip-13376-buildpackages-upgrade-hammer

buildpackages: walk the whole config tree to find sha1 to build

buildpackages: enforce ceph-object-corpus canonical URL

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0045dfc8a5364d20cc4ea5f1f0b3947f05543af9)

buildpackages: walk the whole config tree to find sha1 to build

It is not enough to look for the first install task. In upgrade tests,
the install.upgrade task requires more packages to be built. In more
complicated tests using sequential and parallel tasks, the actual
install or install.upgrade task may be deeper in the config tree.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 4d89c9e4dee3ededc403509010093a55be54a95a)

buildpackages: honour install priorities tags, branch, sha1

The install config may have contradicting tag/branch and sha1.  When
suite.py prepares the jobs, it always overrides the sha1 with whatever
default is provided on the command line with --distro and what is found
in the gitbuilder. If it turns out that the tag or the branch in the
install config task is about another sha1, it will override anyway.

Instead of obtaining the tag, branch and sha1 directly from the
packaging.GitbuilderProject object, compute them from the returned
uri_reference data member. The uri_reference is used by the install task
to fetch packages in the gitbuilders and this is what buildpackages
needs to build.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit c103579a55cff3bd90e2d8d7b8df03bf29560961)

buildpackages: branches are found in refs/remotes/origin

Note in refs/heads which is populated for local branches only and not
after a fresh clone.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 810caaf55f2439efbc0bec7ff21332d40d4dff8e)

buildpackages: do not override the sha1

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit e54c916891a3a8b61a856773a70650f387e290a5)

buildpackages: allow for concurrent build if != sha1

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit a630da722008278969bcdbbdf8a1e26169653fa7)

buildpackages: prefer GitbuilderProject.os_{type,version} over config.

The config['os_type'] and config['os_version'] are not always set for a given
job (for instance, in the rbd suite). When a suite runs, it relies on
default values, depending on the target Operating System and internal,
hard coded values associating ubuntu to 14.04 etc.

Instead of using config['os_{type,version}'] use the GitbuilderProject
equivalent which is set with the appropriate defaults.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 9db757b811a08e07c8c68866af1d41aaf7641de1)

Merge pull request #610 from ceph/wip-13393

Removed rhel distro, fixes #13393 as rhel is not supported in sepia lab

Removed rhel distro, fixes #13393 as rhel is not supported in sepia lab

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/cephfs: fix mount_timeout config loading

Previously were defaulting to a string, which
always compared greater than elapsed, so never
timed out.

Fixes: #12820
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8e275049a0a8b976fc9eb5c63dcefe01bedcc711)

tasks/ceph_fuse.py: virtual machines need flexible mount timeout

1) add a wait time before the mount attempt to let the cluster get set up.
By default this should be skipped, but for VMs and known-slow systems we
can give them 60 seconds.
2) Make the timeout configurable, with a 30-second default, but override it
for VM tests.

http://tracker.ceph.com/issues/12320 Fixes: #12320

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit ec12f21a7ead9d82b575c4f927d5e7b73f85c3d5)

Merge pull request #605 from dachary/wip-buildpackages-test-hammer

buildpackages: implicit for OpenStack

buildpackages: implicit for OpenStack

When using the teuthology-openstack command, buildpackages is implicit
and does not need to be included when running the test suite. Move the
stanza to the debug directory as a reminder when debuging.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit f133725ba37d1f48b5f80fcf733264b60f788051)

Merge pull request #601 from dachary/wip-git-ceph-com

more hammer git ceph com updates

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a14a951640f79fd7ed217c423dcc9f453c830556)

Conflicts:
suites/rbd/qemu/workloads/qemu_bonnie.yaml
suites/rbd/qemu/workloads/qemu_fsstress.yaml
suites/rbd/qemu/workloads/qemu_iozone.yaml.disabled
suites/rbd/qemu/workloads/qemu_xfstests.yaml
suites/rbd/singleton/all/formatted-output.yaml
the conflicts are because https is used in master instead of http,
the rest is otherwise identical.

tasks/chef.py
obsolete, replaced by ansible.

git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f062b171f4ab191986cea525c831573109aba2f9)

teuthology: some suites still use http://ceph.newdream.net

This probably redirects to http://ceph.com but ceph.newdream.net still appears in some places

http://tracker.ceph.com/issues/9922 Fixes: #9922

Signed-off-by: Armando Segnini <armando.segnini@telecom-bretagne.eu>
(cherry picked from commit 9ace238e874b45afbdcef97fe656d9a707e6105a)

Merge pull request #594 from dachary/wip-13250-ceph-deploy-branch-hammer

ceph-deploy: fix ceph-deploy-branch config options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-deploy: fix ceph-deploy-branch config options

The config paramter of download_ceph_deploy does not have a ceph-deploy
item, therefore the ceph-deploy-branch parameter is always assumed to be
master.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit fdcfb79b3c57765441a6d15bea8ef2fcbcebf5f8)

ceph-deploy: always zap disk before creating an osd

The existing logic is to ceph-deploy osd create --zap-disk which will
zap the data device before preparing it. However it will not zap the
journal device (see http://tracker.ceph.com/issues/13291).

If ceph-deploy osd create fails, a fall back will zap both the data
device and the journal and try prepare again. This could work if
the device preparation and activation was synchronous and catch all
errors that could be caused by an unclean journal device. However,
the activation is asynchronous and it is entirely possible for a device
to be prepared successfully and fail to activate in the background.

The data and journal device are always zapped before calling ceph-deploy
osd create. The logic is simpler and the overhead is low.

http://tracker.ceph.com/issues/13000 Fixes: #13000

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 01d48a270ad526492dccbb0e363162f0befb0f9e)

rados: add test for 13234.yaml

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #582 from ceph/wip-buildpackages-hammer

hammer backport: buildpackages task

buildpackages: user-data must be per os-type/os-version

CentOS 6.5 needs to install a package and reboot to grow the root file
system. Instead of assuming a common user-data.txt file can fit all
Operating Systems, make one user data per os-type/os-version combination.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 05228e1e5c95654ce6e1c81f557002cb0c966a2f)

buildpackages: flock must --close

The process run by flock must not inherit the file descriptor because
this will cause the lock to be held forever should the command survive
the call to flock. This is for instance the case for the ssh-agent.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 204e27fecda583c312c35e905b6c864a5c8a1925)

buildpackages: get Ceph git url via get_ceph_git_url()

Instead of relying on git_base_url, use the get_ceph_git_url() to obtain
the URL of the Ceph repository to use with git clone. This allows the
user to override it via the git_ceph_url configuration option and the
--git-ceph-url command line option to teuthology-openstack.

http://tracker.ceph.com/issues/11883 Refs: #11883

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 78b2a9aa740e6780890944f90f8a2615f08e3d33)

buildpackages: make rpm packages

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7e3a271a89d4e7794a1ee547a3105a871f76f9f1)

buildpackages: build the repository and the packages

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit e38a6d69dfb25fae11902f537b6f2b752689236f)

buildpackages: re-use config from install task

So that the packages built match what the install task expects.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8c618e564501112fc5751888991aab87dac22fb2)

buildpackages: pep8 compliance

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d40d8ff50186a552cbc6be8649c660b1534835db)

buildpackages: build ceph packages and upload them

Import install from task.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 0eeb441bb092f0ba7b4af866d58243a917c0d3f1)

buildpackages: build ceph packages and upload them

Build Ceph packages from source for the required revision, os_type,
os_version and architecture and upload them to the gitbuilder
repository.

http://tracker.ceph.com/issues/13031 Fixes: #13031

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 2b25080d4faf0ae9ddb0000808830cd4d5c7bea1)

Merge pull request #539 from ceph/wip-12715

Whitelisted "bad backtrace on dir ino" to fis #12715

Whitelisted "bad backtrace on dir ino" to fis #12715

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #507 from ceph/wip-12068-hammer

Moved point-to-point test to upgrade/hammer-x

Reviewed-by: Samuel Just <sjust@redhat.com>

Moved point-to-point test to upgrade/hammer-x

RE: http://tracker.ceph.com/issues/12068

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #502 from ceph/wip-dumplig-firefly-x-hammer

Fixes #12021 for upgrade/dumpling-firefly-x suite

Fixes #12021 for upgrade/dumpling-firefly-x suite
added "mon debug unsafe allow tier with nonempty snaps: true"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Fixes #12425, moved versions-steps-x suite from upgrade/firefly to upgrade/firefly-x
as the old location was wrong and never ran correctely to upgrade tp -x

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit d413124dddc85876e1da2c8d7acc1b3bfefc08a4)

Fixed s3test branch ref to hammer (firefly seemed incorrect)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Added config settings to resolve and test #12021
"mon debug unsafe allow tier with nonempty snaps: true"
(only for firefly-x, giant-x and upgrade/hammer/point-to-point running againts hammer branch)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit e09f761ac49d56cb1069d629943f90010790fa9a)