ansible: Start libvirt services on slave spinup

I accidentally left these out when I combined all the slave playbooks.  It wasn't caught till now because I hadn't actually pulled the latest ceph-build.git on prado

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1654 from ceph/wip-kernel-drop-centos7

kernel: drop centos7 builds and arm64 cruft

Merge pull request #1653 from ceph/wip-cleanup-home

ceph-api-nightly: Clean up after yourself

kernel: drop arm64 cruft

Don't pretend we can build an arm64 kernel: our kernel config is for
x86_64.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

kernel: drop centos7 builds

The gcc version requirement was raised to 4.9 in kernel 4.8.  centos7
is on gcc 4.8 and these builds have been failing for the last couple of
months.

While we could switch to devtoolset-7 or devtoolset-8, both of which
are installed on our centos7 build agents, let's just drop them.  For
one-off testing of older releases, trusty and xenial would do.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-api-nightly: Clean up after yourself

This job has a bunch of templates.  Each one reaches 30-50GB in size which quickly fills up our builder home dirs.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1651 from ceph/wip-timeout

bz-mill: Scrap job

Merge pull request #1649 from ceph/wip-xterm

ansicolor jobs: Set TERM=xterm

bz-mill: Scrap job

Checked with Christina.  She no longer uses this job.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1650 from tspmelo/wip-cypress-libxss1

install-e2e-test-deps.sh: Add libxss1 to debian distros

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

install-e2e-test-deps.sh: Add libxss1 to debian distros

Cypress is failing due to a missing package.

Signed-off-by: Tiago Melo <tmelo@suse.com>

ansicolor jobs: Set TERM=xterm

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1647 from ceph/wip-cancel_on_pr_udpate-master

ceph-*pull-requests*: cancel job on PR update

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

build: cancel job on PR update

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1646 from tchaikov/wip-ceph-perf-pull-requests-ln

ceph-perf-pull-requests: do not fail if symblink already exists

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: s/source/./

turns out we have ubuntu test nodes labeled with "performance", and they
use dash for "sh", and jenkins use "sh" for running the embedded
scripts.

let's avoid bashism, and use "." instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-perf-pull-requests: install clang-10 on debian/ubuntu

and always install clang for both classic and crimson job

so we can see the warnings spit by clang when running classic perf tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-perf-pull-requests: do not fail if symblink already exists

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1645 from tchaikov/wip-remove-unexpected-token

ceph-perf-pull-requests: remove superfluous ";;"

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: remove superfluous ";;"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1644 from tchaikov/wip-crimson-clang-10

ceph-perf-pull-requests: enable llvm-toolset-10 repo for clang-10

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: enable llvm-toolset-10 repo for clang-10

we need clang for detecting build failures with this compiler, and for
performing static analysis.

recently, seastar started to require better C++17 compatible compiler,
and it dropped the support of C++14. so before llvm-toolset-10 is added
to RHEL8/CentOS8, we need to use our own copr repo for clang-10.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1643 from ceph/wip-cansiblerpm

ceph-ansible-rpm: Add missing CHACRACTL_KEY

ceph-ansible-rpm: Add missing CHACRACTL_KEY

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1642 from ceph/guits-refact_labels

ceph-ansible: only use smithi nodes for all_daemons

ceph-ansible: only use smithi nodes for all_daemons

Using `(smithi || centos7)` allows ovh slave nodes to be used for all
`all_daemons` based scenario which is not what we want since the latest
makes the playbook failing for these scenarios.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1641 from tchaikov/wip-dts-9

scripts/build_utils: add copr repo for latest gcc-toolset-9

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

scripts/build_utils: add copr repo for latest gcc-toolset-9

see also https://copr.fedorainfracloud.org/coprs/tchaikov/gcc-toolset-9

we can drop it once devtoolset-9-gcc-c++ >= 9.2.1-2.3 is available in
CentOS8/RHEL8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*build/setup_rpm: extract setup_rpm_build_deps

less repeatings this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1640 from ceph/guits-braggi

ceph-ansible: do not play 'all_daemons' scenarios on braggi

ceph-ansible: do not play 'all_daemons' scenarios on braggi

braggi nodes don't have enough resources to support these scenarios.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1639 from ceph/guits-add_braggi

ceph-ansible: add back braggi node

ceph-ansible: add back braggi node

This commit adds back braggi nodes for all_daemons based scenario.
Also, it adds back ooo_collocation scenario which was removed by
mistake.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1638 from ceph/wip-nodename

Fixes to builder playbook

ansible: Use JNLP for permanent.  SSH for ephemeral.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Set nodename=ansible_hostname on static builders

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1637 from ceph/wip-copy-perms

ceph*setup: Give ceph*build jobs permission to copy artifacts

ceph*setup: Give ceph*build jobs permission to copy artifacts

This is a new setting in the newest CopyArtifact plugin.

https://stackoverflow.com/a/32084538/6225562

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1636 from ceph/guits_refact_ca

ceph-ansible: refact project definition

ceph-ansible: refact project definition

This commit modifies the current project definitions:

1/ it uses the exclude keyword to avoid duplicating project
2/ it declares a new project to only use smithi nodes for 'large'
scenario testing (basically all scenario based on 'all_daemons'
scenario).
3/ exclude 'non_container-cephadm_adopt' for non_containerized
deployments
4/ reindent jobs section

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1633 from ceph/jan--f-patch-c-v-pr-build

Let tox.ini decide which environments to use

Merge pull request #1635 from ceph/ceph_ansible_remove_tox_dashboard

ceph-ansible: remove tox dashboard

ceph-ansible: remove tox dashboard

This file doesn't exist anymore.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1634 from ceph/ceph_ansible_shrink_osd_nightly

ceph-ansible-nightly: add new shrink_osd scenarios

ceph-ansible-nightly: add new shrink_osd scenarios

This replaces the shrink_osd scenario by shrink_osd_multiple and
shrink_osd_single.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Let tox.ini decide which environments to use

Otherwise we have two places that define envs.

Merge pull request #1632 from ceph/wip-delete_backend_api

ceph-dashboard-pr-backend: delete

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pr-backend: delete

As it's been overridden by ceph-api.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1630 from tchaikov/wip-slimmer-tests

ceph-api: do not build/run tests for api tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>

Merge pull request #1627 from tspmelo/wip-cypress-record

ceph-api-nightly: Enable recording of E2E runs

Merge pull request #1608 from tchaikov/wip-crimson-libc-allocator

ceph-*-setup: use libc allocator for crimson flavor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1631 from tchaikov/wip-slimmer-backend-tests

ceph-dashboard-pr-backend: do not build/run tests for api tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-dashboard-pr-backend: do not build/run tests for api tests

as ceph-pull-requests already covers them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-api: do not build/run tests for api tests

as ceph-pull-requests already covers them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1629 from ceph/guits-split_shrink_osds

ceph-ansible: split shrink_osd scenario

ceph-ansible: split shrink_osd scenario

This is for testing in two ways:

1/ shrinking OSDs one by one
2/ shrinking multiple OSDs with a single call of the playbook

Related ceph-ansible PR: ceph/ceph-ansible#5569

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-api-nightly: Enable recording of E2E runs

Signed-off-by: Tiago Melo <tmelo@suse.com>

ceph-api: rename dasboard-backed-api to ceph-api

For the sake of simplicity (ceph-dashboard-backend-api vs. ceph-api) and
for spreading ownership of the Ceph API across teams, rename (currently
just a clone to minimize CI breakages) dashboard-backend-api to just
ceph-api.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1625 from ceph/guits_skip_jobs

ceph-ansible: play shrink_osd_legacy on 3.2 only

ceph-ansible: play shrink_osd_legacy on 3.2 only

Skip `shrink_osd_legacy` job on all branches but stable-3.2

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1624 from ceph/guits-skip_jobs

ceph-ansible: skip some scenarios

ceph-ansible: skip some scenarios

Let's skip these scenarios, they don't make sense.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1623 from ceph/guits-inject_password

ceph-ansible: inject password in scenario job

ceph-ansible: inject password in scenario job

this commit makes this 'individual job scenario testing' using the
`DOCKER_HUB_USERNAME` and `DOCKER_HUB_PASSWORD` environment variable so
the job can run with docker authentication.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1621 from ceph/wip-whoops-mock

ansible: mock is in EPEL only apparently

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1622 from ceph/wip-labextras

ceph*build: Enable lab-extras repo on CentOS8 builders

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph*build: Enable lab-extras repo on CentOS8 builders

I discovered this was missing by reimaging a CentOS7 builder to CentOS8 and it failed the first build it tried to run.

```
+ sudo yum install -y yum-utils
Last metadata expiration check: 0:00:28 ago on Thu Jul 16 21:05:07 2020.
Package yum-utils-4.0.12-3.el8.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
+ get_rpm_dist
+ LSB_RELEASE=/usr/bin/lsb_release
+ '[' '!' -x /usr/bin/lsb_release ']'
++ /usr/bin/lsb_release --short --id
+ ID=CentOS
+ case $ID in
++ /usr/bin/lsb_release --short --release
++ cut -d. -f1
+ RELEASE=8
+ DIST=el8
+ DISTRO=centos
+ echo el8
el8
+ '[' 8 = 7 ']'
+ '[' 8 = 8 ']'
+ sudo dnf config-manager --set-enabled PowerTools
+ sed -e s/@//g
+ '[' default = crimson ']'
+ sudo yum install -y python3-devel
CentOS-8 - PowerTools                            13 kB/s | 4.3 kB     00:00
Package python36-devel-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64 is already installed.

Dependencies resolved.
Nothing to do.
Complete!
+ sudo yum-builddep -y '--setopt=*.skip_if_unavailable=true' /tmp/install-deps.49348/ceph.spec
Last metadata expiration check: 0:00:01 ago on Thu Jul 16 21:05:36 2020.

Package platform-python-devel-3.6.8-23.el8.x86_64 is already installed.
Package checkpolicy-2.9-1.el8.x86_64 is already installed.
Package cmake-3.11.4-7.el8.x86_64 is already installed.
Package cryptsetup-2.2.0-2.el8.x86_64 is already installed.
Package gcc-c++-8.3.1-5.el8.0.2.x86_64 is already installed.
Package gdbm-1:1.18-1.el8.x86_64 is already installed.
Package keyutils-libs-devel-1.5.10-6.el8.x86_64 is already installed.
No matching package to install: 'libbabeltrace-devel'
```

We're already enabling this repo in `install-deps.sh` but that's obviously not soon enough.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: mock is in EPEL only apparently

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1620 from tchaikov/wip-build_utils

ceph-dev-new-build: pass CEPH_EXTRA_RPMBUILD_ARGS with quotes

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-*-build: pass OSD_FLAVOR to build-push-ceph-container-imgs.sh

so it is able to get the correct REPO_URL and push the container image
with the correct tag

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-build: pass CEPH_EXTRA_RPMBUILD_ARGS with quotes

this change fixes an oversight in 260b24a651

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: fix "bogus date"

rpmbuild complains at seeing them:

warning: bogus date in %changelog: Tue Mar 10 2013 Gary Lowell <glowell@inktank.com> - 1-0
warning: bogus date in %changelog: Tue Aug 27 2012 Gary Lowell <glowell@inktank.com> - 1-0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1619 from tchaikov/wip-build-utils-fixes

build_utils.sh: fix regressions introduced by bac3e963a56

build_utils.sh: fix regressions introduced by bac3e963a56

* quote parameter with spaces in it with ""
* fix the variable name
* use `find` for collecting the patches, because if glob fails
  the patten won't be replaced by the matched file names by shell
* add the closing "}"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1612 from tchaikov/wip-less-errors

ceph-*-build/build/build_rpm: avoid "No such file or directory" errors

Merge pull request #1618 from tchaikov/wip-cephacheck

ceph-perf-pull-requests: pass job id and job url to check

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: pass job id and job url to check

so we can check the details of check run

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*-build,ceph-release-rpm: extract build_ceph_release_rpm

for less repeating

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: avoid "No such file or directory" errors

it's highlighted by jenkins, and it's distracting even we "|| true" this
command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*-build/build/build_rpm: extract helper functions

to avoid repeatings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1617 from tchaikov/wip-cephacheck

ceph-perf-pull-requests: paste check status with cbt/githubcheck

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: stop updating github with status

as check updates it as well

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-perf-pull-requests: paste check status with cbt/githubcheck

add integration with https://github.com/apps/cephacheck

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1616 from ceph/update_tox_for_cephadm

build_utils.sh: add tox-cephadm.ini file

build_utils.sh: add tox-cephadm.ini file

As a follow up on c9463f5 we also need to declare the dedicated tox
file for the cephadm container scenario.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1615 from ceph/ceph_ansible_cephadm

ceph-ansible-prs: add cephadm container scenario

ceph-ansible-prs: add cephadm container scenario

This adds an optional cephadm container scenario.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1614 from ceph/wip-deleteshamanrepo

nfs-ganesha: Clean up shaman repo files

nfs-ganesha: Clean up shaman repo files

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1611 from ceph/wip-nocentos6

ceph-release-rpm: Drop centos6

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-release-rpm: Drop centos6

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1609 from ceph/ceph_ansible_cephadm_adopt

ceph-ansible-prs: add cephadm_adopt scenario

Merge pull request #1610 from ceph/ceph-ansible-docker-login

ceph-ansible: add docker hub credentials

ceph-ansible: add docker hub credentials

To avoid to be rated limited by the docker.io registry when pulling the
ceph container image, we need to use an authenticated user.

https://docs.docker.com/docker-hub/download-rate-limit/

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-ansible-prs: add cephadm_adopt scenario

This adds an optional cephadm_adopt scenario.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-*-setup: use libc allocator for crimson flavor

to avoid the issues caused by allocator. we cannot use seastar allocator
in non-seastar thread or threads not managed by ThreadPool.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1607 from tchaikov/wip-no-nautilus-images

ceph-dev-*-build: reset CI_CONTAINER when building nautilus

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1605 from tchaikov/wip-build-crimson-on-master

ceph-dev-*: add crimson flavor for master build

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev-*-build: reset CI_CONTAINER when building nautilus

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-*: add crimson flavor for master build

Signed-off-by: Kefu Chai <kchai@redhat.com>