buildpackages: no need to sudo in bootcmd

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: log the output of make

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #756 from dachary/wip-13624-workunit-git

workunits: fetch from --ceph-git-url

Merge pull request #758 from dreamhost/wip-openstack-dhc

Fix runs on OpenStack clusters like DreamHost Compute

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buildpackages: pkgrepo host also needs user-data setup.

The packages repo host fails in environments where networking setup is
needed in VMs. Use the same user-data as the buildhosts to ensure this
is the case.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

buildpackages: copy teuthology user-data.txt changes

7b27e1db7: openstack: support /etc/network/intefaces injection
2358562cf: ensure VMs always have /etc/hosts set up
4378a505d: always allow unsigned deb packages
50b2db521: openstack: encode instance name with the full IP
6e828a33b: openstack: add 8.8.8.8 as a last resort resolver

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

buildpackages: catch instance creation failure.

Split the sleep from the server creation, so we catch 'server create'
failures (eg due to quota):
> Quota exceeded for cores: Requested 16, but already used 10 of 20 cores
> (HTTP 403) (Request-ID: req-6467934e-db50-4479-995c-4d44dedf553a)

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

buildpackages: Be robust about addresses

OpenStack could tell us the VM has multiple networks, and offers no
guarantee about the order of addresses either (the old code failed if
the v4 IP was first).

For now, take the first listed network, and the first listed IPv4
address therein. Comments contain more detailed examples of possible
output from openstack tool.

Also remove the need for using jq to parse the output.

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #757 from dachary/wip-dummy

Revert "dummy: only use the user playbook"

Revert "dummy: only use the user playbook"

This reverts commit a016f29e211e378a50fd833d940abbb49449c37b.

Merge pull request #744 from ceph/wip-inode-claim

tasks/cephfs: add TestJournalRepair.test_table_tool_take_inos

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #703 from ceph/wip-layout-restrict

tasks/cephfs: test for auth caps pool-setting restriction

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #728 from ceph/wip-snap-strays

tasks/cephfs: cover snaps in test_strays

Merge pull request #709 from ceph/wip-scrub-basic

tasks/cephfs: add TestForwardScrub

workunit: fetch from --ceph-git-url

The commit from which workunits are fetched must be retrieved
from --ceph-git-url via teuth_config.get_ceph_git_url() instead of
assuming it is available via git://git.ceph.com/ceph.git.

Using git://git.ceph.com/ceph.git is convenient because it supports git
archive. In the general case, some git servers such as github do not
support git archive and a full git clone must be done instead.

Although it would be possible to

    git clone --branch=master --depth=1 --single-branch

to reduce the amount of data being retrieved, it would require a

    git fetch origin SHA1

but git version >= 1.7 do not support fetching a commit.

http://tracker.ceph.com/issues/13624 Fixes: #13624

Signed-off-by: Loic Dachary <loic@dachary.org>

workunit: keep the path to the list of tests in a variable

Signed-off-by: Loic Dachary <loic@dachary.org>

workunit: allow tag to override sha1

The sha1 for the workunit task is always set by the suite.py task. The
tag must be checked before the sha1 othewise it cannot be used to
override the sha1.

Signed-off-by: Loic Dachary <loic@dachary.org>

tasks/cephfs: avoid using bare remote.run in scrub_checks

Use the Mount.* wrappers for filesystem operations,
so that changes like making run_shell use sudo just work.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: update for run_shell vs. sudo

(TestStrays.test_snapshot_remove blocked on the sudo change)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: always sudo in run_shell

This was causing permissions issues when
running inside teuthology, as run_python
was using sudo and run_shell wasn't.

Would be nice to get rid of all the rootishness,
but for the moment just make it more uniform.

tasks/cephfs: cover snaps in test_strays

Signed-off-by: John Spray <john.spray@redhat.com>

suites: add yaml for TestForwardScrub

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add TestForwardScrub

This tests the forward scrub's ability to traverse
some metadata and tag it, and the corresponding
functionality in cephfs-data-scan to filter based
on tag and inject orphaned items.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #737 from ceph/wip-enable-data-scan

suites: yaml for TestDataScan

Merge pull request #751 from dachary/wip-teuthology-integration

teuthology-integration: run teuthology integration tests

teuthology-integration: run teuthology integration tests

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #664 from dachary/wip-nop

nop: suite that does not even lock targets

Merge pull request #748 from dachary/wip-13910-buildpackages-images

buildpackages: create the image if it does not exist

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

buildpackages: create the image if it does not exist

Since buildpackages runs before target provisioning, it is possible that
the desired image does not yet exist on a newly provisionned tenant (or
region).

http://tracker.ceph.com/issues/13910 Fixes: #13910

Signed-off-by: Loic Dachary <loic@dachary.org>

suites: yaml for TestDataScan

This was an oversight.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #679 from ceph/wip-ceph-whitespace

tasks/ceph: fix up whitespace

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/cephfs: add TestJournalRepair.test_table_tool_take_inos

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #740 from dachary/wip-13899-rbd-fio

task/rbd_fio: allow for unsigned packages

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

Merge pull request #742 from dachary/wip-openstack-morepggrow

openstack: rados/.../morepggrow.yaml may need more disk

Reviewed-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

openstack: rados/.../morepggrow.yaml may need more disk

Signed-off-by: Loic Dachary <loic@dachary.org>

task/rbd_fio: allow for unsigned packages

Similar to what the teuthology install.py task does, add --force-yes to
the apt-get install so that unsigned packages are successfully
installed. It is needed when the buildpackages task is used to create
packages on the fly.

There is no need to do the same for rpm packages because the
verification is controlled from the ceph-release package instead of from
the command line.

http://tracker.ceph.com/issues/13899 Fixes: #13899

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #736 from dachary/wip-buildpackage-delete-wait

buildpackages: wait for the build machine deletion

Merge pull request #738 from dachary/wip-openstack-debug

openstack: add 15GB for debugging purposes

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

openstack: add 15GB for debugging purposes

So that it can be used as follows:

   teuthology-openstack ... --suite mysuite ... debug/openstack-15G.yaml

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: wait for the build machine deletion

When the quotas are low, it matters to block until the build machine is
actually deleted. Otherwise target provisionning may fail because the
they exceed the quota. For instance the default on OVH is to have 32
cores and the build machine uses 16. The packages-repository machine
uses two, the teuthology cluster uses one and that leaves only 13 cores
for the targets which may be too low when running jobs that require
large instances.

Signed-off-by: Loic Dachary <loic@dachary.org>

rados/thrash/workloads/admin-socket: 60s -> 180s

Provide more time for the asok workload to run.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #729 from ceph/wip-scrub-tidy

tasks/cephfs: tidy test_scrub_checks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #674 from ceph/wip-13698

tasks/cephfs: fix race in test_sessionmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #663 from ceph/wip-client-mark-down

tasks/cephfs: quick test for `tell`...

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #647 from ceph/wip-repair-dir

tasks/cephfs: include an empty dir in journal repair

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #633 from ceph/wip-test-failover

tasks/cephfs/test_failover

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #732 from dachary/wip-openstack-debug

openstack: add 30GB for debugging purposes

openstack: add 30GB for debugging purposes

So that it can be used as follows:

   teuthology-openstack ... --suite mysuite ... debug/openstack-30G.yaml

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #718 from ceph/wip-fast-read

rados: add test for ec fast_read

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #730 from ceph/wip-repair

ceph_manager: do_pg_scrub: keep scrubbing until it's done

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks/cephfs: tidy test_scrub_checks

Use named error codes instead of numbers, and
use the helper fn for getting inode number
instead of doing it by hand.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #725 from dachary/wip-openstack-rbd

openstack: rbd/thrash and rbd/qemu resources

openstack: rbd/{thrash,qemu}: allocate three disks, always

The thrasher and qemu need disk attached to run against xfs or btrfs,
otherwise it will use the local file system which is always ext4. It
will succeed but this is a kind of false positive since no xfs or btrfs
related tests were run.

http://tracker.ceph.com/issues/13270 Refs: #13270

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #721 from dachary/wip-13581-openstack-hints

openstack: convert to array hints

openstack: convert to array hints

So that it is more flexible.

http://tracker.ceph.com/issues/13581 Fixes: #13581

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #719 from dachary/wip-dumpling-parallel

buildpackages: do not -jX on dumpling

buildpackages: do not -jX on dumpling

http://tracker.ceph.com/issues/13845 Refs: #13845

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph_manager: do_pg_scrub: keep scrubbing until it's done

The ceph pg scrub ... command isn't really guarranteed to
start a scrub, keep reissuing it until the scrub actually
happens.

Related: #12746
Signed-off-by: Samuel Just <sjust@redhat.com>

rados: add test for ec fast_read

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #707 from ceph/wip-13774-jewel

Added infernalis-client-x

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Added infernalis-client-x - fixes #13774

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #712 from dachary/wip-buildpackages-refactor

buildpackages: implement get_pkg_type & avoid duplicate packages-repository

buildpackages: avoid duplicate packages-repository

Signed-off-by: Loic Dachary <loic@dachary.org>

buildpackages: implement get_pkg_type

It can be deprectated when it is implemented in teuthology.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #710 from dachary/wip-buildpackages-refactor

buildpackages: refactor to not require remote

buildpackages: refactor to not require remote

Most of the flavor, sha1, tag etc. selection logic as implemented in the
packaging module of teuthology relies on remote hosts. This is complex
to tests and inconvenient because hosts must be provisionned even before
trying to figure out which packages need to be installed.

Using remote hosts is necessary when bare metal targets are used because
teuthology must adapt to the operating system already installed. The
selection logic in the context of dynamically provisionned targets is
simpler because it is defined by the job being run.

The buildpackages is refactored to use only the job configuration to
figure out which packages must be built. It makes it specific to targets
that are dynamically provisionned. It would have to be modified to query
the remote host in the case of bare metal targets.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #704 from dachary/wip-buildpackages-packaging

buildpackages: protect packages-repository

buildpackages: protect packages-repository

Signed-off-by: Loic Dachary <loic@dachary.org>

tasks/cephfs: test for auth caps pool-setting restriction

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #701 from dachary/wip-buildpackages-giant

buildpackages: do not fails for unpackaged files

buildpackages: do not fails for unpackaged files

Older versions of Ceph have that problem failing would break all upgrade suites.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #700 from ceph/yuriw-patch-1

Update schedule_rados.sh

Update schedule_rados.sh

Fixed the args order

Merge pull request #699 from ceph/wip_rados_subset

Require -e arg for email address

Require -e arg for email address

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #698 from ceph/wip_wip_fix_f-h-x-offline-jewel

Added %

Added %
Fixed intall.upgrade syntax
Whitelisted 'reached quota'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #697 from dachary/wip-buildpackages-giant

buildpackages: giant has some unpackaged files

buildpackages: giant has some unpackaged files

And there really is nothing we can do about it expect ignore them.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #695 from dachary/wip-buildpackages-flavor

buildpackages: 10GB is too small most of the time

buildpackages: 10GB is too small most of the time

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #684 from dachary/wip-12973

rados: 'failed to encode ...' warnings are normal on upgrades

Merge pull request #694 from ceph/wip_fix_f-h-x-offline

Fixing /hammer-x/f-h-x-offline suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Added %
Fixed intall.upgrade syntax
Whitelisted 'reached quota'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #693 from ceph/wip_fix_f-h-x-offline

Fixed install.upgrade syntax

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Added %
Fixed intall.upgrade syntax

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #692 from ceph/wip_fix_f-h-x-offline

Added %

Added %

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #691 from ceph/yuriw-patch-2

Update schedule_rados.sh

Update schedule_rados.sh

Merge pull request #689 from ceph/wip_rados_subset

Merge pull request #686 from ceph/wip_rados_subset

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #686 from ceph/wip_rados_subset

Reducing the ammount of resulting jobs scheduled
Fixed the math added comments

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #688 from ceph/jewel

Reducing the ammount of resulting jobs scheduled

Reducing the ammount of resulting jobs scheduled

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 92980c3063487b477c1bff71ebb2c403bcdfa73c)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #686 from ceph/wip_rados_subset

Reducing the ammount of resulting jobs scheduled

Reducing the ammount of resulting jobs scheduled

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #681 from ceph/wip-fio-ver

Update fio version from 2.2.9 to 2.2.11

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rados: 'failed to encode ...' warnings are normal on upgrades

http://tracker.ceph.com/issues/12973 Fixes: #12973

Signed-off-by: Loic Dachary <loic@dachary.org>

Update fio version from 2.2.9 to 2.2.11

Also removes fio-version option from yaml since its redundant and if required can be specified in
overrides

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>