ceph-perf-pull-requests: install clang-10 on debian/ubuntu

and always install clang for both classic and crimson job

so we can see the warnings spit by clang when running classic perf tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-perf-pull-requests: do not fail if symblink already exists

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1645 from tchaikov/wip-remove-unexpected-token

ceph-perf-pull-requests: remove superfluous ";;"

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: remove superfluous ";;"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1644 from tchaikov/wip-crimson-clang-10

ceph-perf-pull-requests: enable llvm-toolset-10 repo for clang-10

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: enable llvm-toolset-10 repo for clang-10

we need clang for detecting build failures with this compiler, and for
performing static analysis.

recently, seastar started to require better C++17 compatible compiler,
and it dropped the support of C++14. so before llvm-toolset-10 is added
to RHEL8/CentOS8, we need to use our own copr repo for clang-10.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1643 from ceph/wip-cansiblerpm

ceph-ansible-rpm: Add missing CHACRACTL_KEY

ceph-ansible-rpm: Add missing CHACRACTL_KEY

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1642 from ceph/guits-refact_labels

ceph-ansible: only use smithi nodes for all_daemons

ceph-ansible: only use smithi nodes for all_daemons

Using `(smithi || centos7)` allows ovh slave nodes to be used for all
`all_daemons` based scenario which is not what we want since the latest
makes the playbook failing for these scenarios.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1641 from tchaikov/wip-dts-9

scripts/build_utils: add copr repo for latest gcc-toolset-9

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

scripts/build_utils: add copr repo for latest gcc-toolset-9

see also https://copr.fedorainfracloud.org/coprs/tchaikov/gcc-toolset-9

we can drop it once devtoolset-9-gcc-c++ >= 9.2.1-2.3 is available in
CentOS8/RHEL8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*build/setup_rpm: extract setup_rpm_build_deps

less repeatings this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1640 from ceph/guits-braggi

ceph-ansible: do not play 'all_daemons' scenarios on braggi

ceph-ansible: do not play 'all_daemons' scenarios on braggi

braggi nodes don't have enough resources to support these scenarios.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1639 from ceph/guits-add_braggi

ceph-ansible: add back braggi node

ceph-ansible: add back braggi node

This commit adds back braggi nodes for all_daemons based scenario.
Also, it adds back ooo_collocation scenario which was removed by
mistake.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1638 from ceph/wip-nodename

Fixes to builder playbook

ansible: Use JNLP for permanent.  SSH for ephemeral.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Set nodename=ansible_hostname on static builders

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1637 from ceph/wip-copy-perms

ceph*setup: Give ceph*build jobs permission to copy artifacts

ceph*setup: Give ceph*build jobs permission to copy artifacts

This is a new setting in the newest CopyArtifact plugin.

https://stackoverflow.com/a/32084538/6225562

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1636 from ceph/guits_refact_ca

ceph-ansible: refact project definition

ceph-ansible: refact project definition

This commit modifies the current project definitions:

1/ it uses the exclude keyword to avoid duplicating project
2/ it declares a new project to only use smithi nodes for 'large'
scenario testing (basically all scenario based on 'all_daemons'
scenario).
3/ exclude 'non_container-cephadm_adopt' for non_containerized
deployments
4/ reindent jobs section

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1633 from ceph/jan--f-patch-c-v-pr-build

Let tox.ini decide which environments to use

Merge pull request #1635 from ceph/ceph_ansible_remove_tox_dashboard

ceph-ansible: remove tox dashboard

ceph-ansible: remove tox dashboard

This file doesn't exist anymore.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1634 from ceph/ceph_ansible_shrink_osd_nightly

ceph-ansible-nightly: add new shrink_osd scenarios

ceph-ansible-nightly: add new shrink_osd scenarios

This replaces the shrink_osd scenario by shrink_osd_multiple and
shrink_osd_single.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Let tox.ini decide which environments to use

Otherwise we have two places that define envs.

Merge pull request #1632 from ceph/wip-delete_backend_api

ceph-dashboard-pr-backend: delete

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pr-backend: delete

As it's been overridden by ceph-api.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1630 from tchaikov/wip-slimmer-tests

ceph-api: do not build/run tests for api tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>

Merge pull request #1627 from tspmelo/wip-cypress-record

ceph-api-nightly: Enable recording of E2E runs

Merge pull request #1608 from tchaikov/wip-crimson-libc-allocator

ceph-*-setup: use libc allocator for crimson flavor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1631 from tchaikov/wip-slimmer-backend-tests

ceph-dashboard-pr-backend: do not build/run tests for api tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

ceph-dashboard-pr-backend: do not build/run tests for api tests

as ceph-pull-requests already covers them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-api: do not build/run tests for api tests

as ceph-pull-requests already covers them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1629 from ceph/guits-split_shrink_osds

ceph-ansible: split shrink_osd scenario

ceph-ansible: split shrink_osd scenario

This is for testing in two ways:

1/ shrinking OSDs one by one
2/ shrinking multiple OSDs with a single call of the playbook

Related ceph-ansible PR: ceph/ceph-ansible#5569

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-api-nightly: Enable recording of E2E runs

Signed-off-by: Tiago Melo <tmelo@suse.com>

ceph-api: rename dasboard-backed-api to ceph-api

For the sake of simplicity (ceph-dashboard-backend-api vs. ceph-api) and
for spreading ownership of the Ceph API across teams, rename (currently
just a clone to minimize CI breakages) dashboard-backend-api to just
ceph-api.

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #1625 from ceph/guits_skip_jobs

ceph-ansible: play shrink_osd_legacy on 3.2 only

ceph-ansible: play shrink_osd_legacy on 3.2 only

Skip `shrink_osd_legacy` job on all branches but stable-3.2

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1624 from ceph/guits-skip_jobs

ceph-ansible: skip some scenarios

ceph-ansible: skip some scenarios

Let's skip these scenarios, they don't make sense.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1623 from ceph/guits-inject_password

ceph-ansible: inject password in scenario job

ceph-ansible: inject password in scenario job

this commit makes this 'individual job scenario testing' using the
`DOCKER_HUB_USERNAME` and `DOCKER_HUB_PASSWORD` environment variable so
the job can run with docker authentication.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #1621 from ceph/wip-whoops-mock

ansible: mock is in EPEL only apparently

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1622 from ceph/wip-labextras

ceph*build: Enable lab-extras repo on CentOS8 builders

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph*build: Enable lab-extras repo on CentOS8 builders

I discovered this was missing by reimaging a CentOS7 builder to CentOS8 and it failed the first build it tried to run.

```
+ sudo yum install -y yum-utils
Last metadata expiration check: 0:00:28 ago on Thu Jul 16 21:05:07 2020.
Package yum-utils-4.0.12-3.el8.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
+ get_rpm_dist
+ LSB_RELEASE=/usr/bin/lsb_release
+ '[' '!' -x /usr/bin/lsb_release ']'
++ /usr/bin/lsb_release --short --id
+ ID=CentOS
+ case $ID in
++ /usr/bin/lsb_release --short --release
++ cut -d. -f1
+ RELEASE=8
+ DIST=el8
+ DISTRO=centos
+ echo el8
el8
+ '[' 8 = 7 ']'
+ '[' 8 = 8 ']'
+ sudo dnf config-manager --set-enabled PowerTools
+ sed -e s/@//g
+ '[' default = crimson ']'
+ sudo yum install -y python3-devel
CentOS-8 - PowerTools                            13 kB/s | 4.3 kB     00:00
Package python36-devel-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64 is already installed.

Dependencies resolved.
Nothing to do.
Complete!
+ sudo yum-builddep -y '--setopt=*.skip_if_unavailable=true' /tmp/install-deps.49348/ceph.spec
Last metadata expiration check: 0:00:01 ago on Thu Jul 16 21:05:36 2020.

Package platform-python-devel-3.6.8-23.el8.x86_64 is already installed.
Package checkpolicy-2.9-1.el8.x86_64 is already installed.
Package cmake-3.11.4-7.el8.x86_64 is already installed.
Package cryptsetup-2.2.0-2.el8.x86_64 is already installed.
Package gcc-c++-8.3.1-5.el8.0.2.x86_64 is already installed.
Package gdbm-1:1.18-1.el8.x86_64 is already installed.
Package keyutils-libs-devel-1.5.10-6.el8.x86_64 is already installed.
No matching package to install: 'libbabeltrace-devel'
```

We're already enabling this repo in `install-deps.sh` but that's obviously not soon enough.

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: mock is in EPEL only apparently

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1620 from tchaikov/wip-build_utils

ceph-dev-new-build: pass CEPH_EXTRA_RPMBUILD_ARGS with quotes

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-*-build: pass OSD_FLAVOR to build-push-ceph-container-imgs.sh

so it is able to get the correct REPO_URL and push the container image
with the correct tag

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-new-build: pass CEPH_EXTRA_RPMBUILD_ARGS with quotes

this change fixes an oversight in 260b24a651

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils: fix "bogus date"

rpmbuild complains at seeing them:

warning: bogus date in %changelog: Tue Mar 10 2013 Gary Lowell <glowell@inktank.com> - 1-0
warning: bogus date in %changelog: Tue Aug 27 2012 Gary Lowell <glowell@inktank.com> - 1-0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1619 from tchaikov/wip-build-utils-fixes

build_utils.sh: fix regressions introduced by bac3e963a56

build_utils.sh: fix regressions introduced by bac3e963a56

* quote parameter with spaces in it with ""
* fix the variable name
* use `find` for collecting the patches, because if glob fails
  the patten won't be replaced by the matched file names by shell
* add the closing "}"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1612 from tchaikov/wip-less-errors

ceph-*-build/build/build_rpm: avoid "No such file or directory" errors

Merge pull request #1618 from tchaikov/wip-cephacheck

ceph-perf-pull-requests: pass job id and job url to check

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: pass job id and job url to check

so we can check the details of check run

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*-build,ceph-release-rpm: extract build_ceph_release_rpm

for less repeating

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/build_utils.sh: avoid "No such file or directory" errors

it's highlighted by jenkins, and it's distracting even we "|| true" this
command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-*-build/build/build_rpm: extract helper functions

to avoid repeatings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1617 from tchaikov/wip-cephacheck

ceph-perf-pull-requests: paste check status with cbt/githubcheck

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: stop updating github with status

as check updates it as well

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-perf-pull-requests: paste check status with cbt/githubcheck

add integration with https://github.com/apps/cephacheck

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1616 from ceph/update_tox_for_cephadm

build_utils.sh: add tox-cephadm.ini file

build_utils.sh: add tox-cephadm.ini file

As a follow up on c9463f5 we also need to declare the dedicated tox
file for the cephadm container scenario.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1615 from ceph/ceph_ansible_cephadm

ceph-ansible-prs: add cephadm container scenario

ceph-ansible-prs: add cephadm container scenario

This adds an optional cephadm container scenario.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1614 from ceph/wip-deleteshamanrepo

nfs-ganesha: Clean up shaman repo files

nfs-ganesha: Clean up shaman repo files

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1611 from ceph/wip-nocentos6

ceph-release-rpm: Drop centos6

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-release-rpm: Drop centos6

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1609 from ceph/ceph_ansible_cephadm_adopt

ceph-ansible-prs: add cephadm_adopt scenario

Merge pull request #1610 from ceph/ceph-ansible-docker-login

ceph-ansible: add docker hub credentials

ceph-ansible: add docker hub credentials

To avoid to be rated limited by the docker.io registry when pulling the
ceph container image, we need to use an authenticated user.

https://docs.docker.com/docker-hub/download-rate-limit/

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-ansible-prs: add cephadm_adopt scenario

This adds an optional cephadm_adopt scenario.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-*-setup: use libc allocator for crimson flavor

to avoid the issues caused by allocator. we cannot use seastar allocator
in non-seastar thread or threads not managed by ThreadPool.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1607 from tchaikov/wip-no-nautilus-images

ceph-dev-*-build: reset CI_CONTAINER when building nautilus

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1605 from tchaikov/wip-build-crimson-on-master

ceph-dev-*: add crimson flavor for master build

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev-*-build: reset CI_CONTAINER when building nautilus

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-*: add crimson flavor for master build

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1604 from ceph/wip-rmrfcontainer

ceph-dev*build: Delete ceph-container dir on failure

ceph-dev*build: Delete ceph-container dir on failure

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1602 from rishabh-d-dave/cv-dont-run-simple-func-tests

ceph-volume: don't run simple func tests against PRs

Merge pull request #1596 from tchaikov/wip-chacracl-0.0.21

s/chacractl>=0.0.4/chacractl>=0.0.21/

Reviewed-by: Justin Caratzas <justin.caratzas@gmail.com>

Merge pull request #1603 from sebastian-philipp/ceph-dev-new-build-mandatory-containers

ceph-dev-new-build: fail, if building containers fail

ceph-dev-new-build: fail, if building containers fail

This avoids:

* unnecessary breakages in the master branch
* unnecessary pulpito runs due to failing containers

Let's fail early, instead of later in the pulpilto runs.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: don't run simple func tests for PRs

Since "simple" functional tests are meant to test that ceph-volume
successfully takes over ceph-disk deployed OSDs.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1601 from ceph/revert_container_nightly_changes

container: revert flavors/releases update

container: revert flavors/releases update

We can do this kind of changes directly in ceph-container instead of
doing some kind of hack.

https://github.com/ceph/ceph-container/pull/1696

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1600 from ceph/container_nightly_flavors

container: update nighly flavors

container: update nighly flavors

In addition of 61b7b619 we also need to update the FLAVORS env variable.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1599 from ceph/container_nightly_update

container: update devel nightly list

container: update devel nightly list

We don't need to build luminous and mimic devel anymore.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1598 from tchaikov/wip-stop-building-wip-branches-on-el7

ceph-dev-trigger: stop building wip-* on el7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-dev-trigger: stop building wip-* on el7

it's a follow-up of f57a522325a61ab8de3a2817729013f7f4a96177

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1597 from tchaikov/wip-no-more-master-on-el7

ceph-dev-trigger: stop building master on el7