build_utils.sh: Don't build containers for luminous either

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1611 from ceph/wip-nocentos6

ceph-release-rpm: Drop centos6

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-release-rpm: Drop centos6

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1609 from ceph/ceph_ansible_cephadm_adopt

ceph-ansible-prs: add cephadm_adopt scenario

Merge pull request #1610 from ceph/ceph-ansible-docker-login

ceph-ansible: add docker hub credentials

ceph-ansible: add docker hub credentials

To avoid to be rated limited by the docker.io registry when pulling the
ceph container image, we need to use an authenticated user.

https://docs.docker.com/docker-hub/download-rate-limit/

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-ansible-prs: add cephadm_adopt scenario

This adds an optional cephadm_adopt scenario.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1607 from tchaikov/wip-no-nautilus-images

ceph-dev-*-build: reset CI_CONTAINER when building nautilus

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1605 from tchaikov/wip-build-crimson-on-master

ceph-dev-*: add crimson flavor for master build

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dev-*-build: reset CI_CONTAINER when building nautilus

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dev-*: add crimson flavor for master build

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1604 from ceph/wip-rmrfcontainer

ceph-dev*build: Delete ceph-container dir on failure

ceph-dev*build: Delete ceph-container dir on failure

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1602 from rishabh-d-dave/cv-dont-run-simple-func-tests

ceph-volume: don't run simple func tests against PRs

Merge pull request #1596 from tchaikov/wip-chacracl-0.0.21

s/chacractl>=0.0.4/chacractl>=0.0.21/

Reviewed-by: Justin Caratzas <justin.caratzas@gmail.com>

Merge pull request #1603 from sebastian-philipp/ceph-dev-new-build-mandatory-containers

ceph-dev-new-build: fail, if building containers fail

ceph-dev-new-build: fail, if building containers fail

This avoids:

* unnecessary breakages in the master branch
* unnecessary pulpito runs due to failing containers

Let's fail early, instead of later in the pulpilto runs.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: don't run simple func tests for PRs

Since "simple" functional tests are meant to test that ceph-volume
successfully takes over ceph-disk deployed OSDs.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1601 from ceph/revert_container_nightly_changes

container: revert flavors/releases update

container: revert flavors/releases update

We can do this kind of changes directly in ceph-container instead of
doing some kind of hack.

https://github.com/ceph/ceph-container/pull/1696

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1600 from ceph/container_nightly_flavors

container: update nighly flavors

container: update nighly flavors

In addition of 61b7b619 we also need to update the FLAVORS env variable.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1599 from ceph/container_nightly_update

container: update devel nightly list

container: update devel nightly list

We don't need to build luminous and mimic devel anymore.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #1598 from tchaikov/wip-stop-building-wip-branches-on-el7

ceph-dev-trigger: stop building wip-* on el7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-dev-trigger: stop building wip-* on el7

it's a follow-up of f57a522325a61ab8de3a2817729013f7f4a96177

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1597 from tchaikov/wip-no-more-master-on-el7

ceph-dev-trigger: stop building master on el7

ceph-dev-trigger: stop building master on el7

as we already have octopus build for el7, which is suffice for preparing
user of older release on el7 to upgrade to octopus, and laster octopus
on el8 or containerized deployment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

s/chacractl>=0.0.4/chacractl>=0.0.21/

chacractl v0.0.21 is python3 compatible, let's use chacractl>=0.0.21, so
no need to install python2 just for using chacractl!

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1595 from ceph/private-tserlin-nfs-ganesha-PowerTools-remove-sed

nfs-ganesha: enable PowerTools repo for CentOS 8; remove sed hack

nfs-ganesha: enable PowerTools repo for CentOS 8; remove sed hack

The CentOS 8 PowerTools repo is disabled by default, but we need
it for a few *-devel packages.

Also, the sed hack for adding the libacl-devel package is no longer
needed. The change was merged upstream:

https://github.com/nfs-ganesha/nfs-ganesha/commit/8f01a5df9dbcdad6c501efb929a711c3ba6a6b4c

Merge pull request #1592 from ceph/wip-newquay

ceph-dev*/quay*: Use our own quay.ceph.io registry

Merge pull request #1590 from tchaikov/wip-nautilus-el8

ceph-dev-new-trigger: build nautilus on el8 also

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-dev*/quay*: Use our own quay.ceph.io registry

This will have to be overridden in ceph.git/qa as well.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1589 from ceph/private-tserlin-nfs-ganesha-enable-el8

nfs-ganesha: enable CentOS 8 builds, other fixes

Merge pull request #1591 from ceph/private-tserlin-nfs-ganesha-stable-enable-el8

nfs-ganesha-stable: Enable CentOS 8 builds of the stable branches of nfs-ganesha

nfs-ganesha-stable: Enable CentOS 8 builds; other fixes

We also needed other fixes to the build script to account for both
el8/el7 builds, as well as handling for the newer version of "mock"
which now uses templates.

ceph-dev-new-trigger: build nautilus on el8 also

Signed-off-by: Kefu Chai <kchai@redhat.com>

nfs-ganesha: enable CentOS 8 builds, other fixes

Enable CentOS 8 builds of the "next" branch of nfs-ganesha.

We also needed other fixes to the build script to account for both
el8/el7 builds, as well as handling for the newer version of "mock"
which now uses templates.

Merge pull request #1587 from ceph/wip-fixrsync

ceph-website: Fix typo in rsync command

ceph-website: Fix typo in rsync command

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1586 from ceph/wip-adam

ceph-website: white-list adamduncan

ceph-website: white-list adamduncan

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1585 from ceph/wip-slashes

ceph-website: Support GIT_BRANCH with multiple slashes

ceph-website: Support GIT_BRANCH with multiple slashes

The SoftIron folks like to use branch names like 'origin/feature/multingual' so let's support that.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1583 from callithea/ceph-dashboard-pr-backend

ceph-dashboard-pr-backend: Target nautilus too

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pr-backend: Target nautilus too

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1582 from tchaikov/wip-apt-present

ceph-deploy-pull-requests: s/installed/present/

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-deploy-pull-requests: drop state parameter

since state is "present" by default, there is no need to specify it
explicitly:
see
https://docs.ansible.com/ansible/latest/modules/apt_module.html#parameters

the desired package state should be "present" instead of "installed".

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1577 from ceph/wip-dash-ansi

ceph-dashboard: Enable ansicolor plugin

Merge pull request #1579 from ceph/drop-focal

ceph-*-build: remove focal for crimson flavor

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-*-build: remove focal for crimson flavor

we have now shifted to using centos 8 as base for building crimson
instead of using focal.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-dashboard/ceph-prs: Enable ansicolor plugin

Ernesto requested this via e-mail

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1580 from ceph/wip-db-m

ceph-dashboard-pr-backend: Target master too

ceph-dashboard-pr-backend: Target master too

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1578 from tchaikov/wip-ceph-deploy-pr-playbook-sudo

ansible: replace sudo with become

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ansible: replace sudo by become

sudo has be deprecated since Ansible 1.9 [0] and was removed in 2.9 [1], let's
use "become" instead.

---
[0]
https://github.com/ansible/ansible/blob/stable-2.0/CHANGELOG.md#19-dancing-in-the-street---mar-25-2015
[1] https://github.com/ansible/ansible/blob/stable-2.9/changelogs/CHANGELOG-v2.9.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1576 from tchaikov/wip-crimson-clang

ceph-perf-pull-requests: use clang for crimson perf test

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-perf-pull-requests: install clang for crimson build

so we can detect build failures on clang

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-perf-pull-requests: prefer clang over gcc when building crimson

for detecting test failures on clang

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1575 from ceph/wip-backend-octopus

ceph-dashboard-pr-backend: Auto-run on octopus PRs only

ceph-dashboard-pr-backend: Auto-run on octopus PRs only

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1574 from ceph/wip-cephio

Add ceph-website job

Merge pull request #1573 from ceph/wip-one-playbook

Refactor slave playbook

Merge pull request #1549 from ceph/wip-syncpull

scripts: Fix sync-pull so it still supports older than Octopus

ansible: Add some shebangs to the jenkins_node module

This module wasn't working on python3-only distros.  This fixes it and is apparently fine.

https://blogs.gnome.org/mcatanzaro/2018/02/16/on-python-shebangs/

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: One playbook to rule them all

I/we got into a bad habit of updating one playbook to support X new distro or X new package but wouldn't update the others.

I think having all the tasks for all the types of slaves in one playbook will help keep things homogenous.  And the playbooks are still idempotent (they can't be run while a job is running of course).

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Scrap all individual playbooks

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1572 from ceph/wip-optional-sudo

ansible: Optionally disable sudo access for jenkins-build user

Add ceph-website job

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Update slave playbooks for python2-less Ubuntu distros

python2 is completely unavailable in Focal Fossa so it's time to split out those deps by Ubuntu version

Signed-off-by: David Galloway <dgallowa@redhat.com>

ansible: Optionally disable sudo access for jenkins-build user

Sometimes that user just doesn't need sudo access y'know?

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1569 from lxbsz/new

kernel: update KConfig to 5.7.0-rc5-ish

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

kernel: update Kconfig templates to 5.7-rc5

Because of how "make olddefconfig" works, we were missing netfilter
options and couldn't configure INPUT/FORWARD/OUTPUT chains needed for
kcephfs suite (unshare_ns_mount.sh, mount.py).

Update our Kconfig templates to 5.7-rc5.  This has been generated
from the actual config (/boot/config-*) of one of the latest kernel
builds in shaman (as opposed to e.g. taking a distro config).

Here is the diff against the actual deb config:

14a18
> CONFIG_CC_HAS_ASM_INLINE=y
1199,1201c1203,1205
< # CONFIG_NF_TABLES_INET is not set
< # CONFIG_NF_TABLES_NETDEV is not set
< # CONFIG_NFT_NUMGEN is not set
---
> CONFIG_NF_TABLES_INET=y
> CONFIG_NF_TABLES_NETDEV=y
> CONFIG_NFT_NUMGEN=m
1202a1207
> # CONFIG_NFT_FLOW_OFFLOAD is not set
1208a1214
> CONFIG_NFT_NAT=m
1213a1220
> CONFIG_NFT_REJECT_INET=m
1215a1223,1224
> CONFIG_NFT_FIB=m
> # CONFIG_NFT_FIB_INET is not set
1217,1221c1226,1235
< # CONFIG_NFT_SOCKET is not set
< # CONFIG_NFT_OSF is not set
< # CONFIG_NFT_TPROXY is not set
< # CONFIG_NFT_SYNPROXY is not set
< # CONFIG_NF_FLOW_TABLE is not set
---
> CONFIG_NFT_SOCKET=m
> CONFIG_NFT_OSF=m
> CONFIG_NFT_TPROXY=m
> CONFIG_NFT_SYNPROXY=m
> CONFIG_NF_DUP_NETDEV=m
> CONFIG_NFT_DUP_NETDEV=m
> CONFIG_NFT_FWD_NETDEV=m
> # CONFIG_NFT_FIB_NETDEV is not set
> CONFIG_NF_FLOW_TABLE_INET=m
> CONFIG_NF_FLOW_TABLE=m
1386,1387c1400,1405
< # CONFIG_NF_TABLES_IPV4 is not set
< # CONFIG_NF_TABLES_ARP is not set
---
> CONFIG_NF_TABLES_IPV4=y
> CONFIG_NFT_REJECT_IPV4=m
> CONFIG_NFT_DUP_IPV4=m
> CONFIG_NFT_FIB_IPV4=m
> CONFIG_NF_TABLES_ARP=y
> CONFIG_NF_FLOW_TABLE_IPV4=m
1423c1441,1445
< # CONFIG_NF_TABLES_IPV6 is not set
---
> CONFIG_NF_TABLES_IPV6=y
> CONFIG_NFT_REJECT_IPV6=m
> CONFIG_NFT_DUP_IPV6=m
> CONFIG_NFT_FIB_IPV6=m
> CONFIG_NF_FLOW_TABLE_IPV6=m
1459a1482
> CONFIG_NFT_BRIDGE_REJECT=m
1661a1685
> # CONFIG_NET_ACT_CT is not set

Here is the diff against the actual rpm config:

1167,1169c1170,1172
< # CONFIG_NF_TABLES_INET is not set
< # CONFIG_NF_TABLES_NETDEV is not set
< # CONFIG_NFT_NUMGEN is not set
---
> CONFIG_NF_TABLES_INET=y
> CONFIG_NF_TABLES_NETDEV=y
> CONFIG_NFT_NUMGEN=m
1170a1174
> # CONFIG_NFT_FLOW_OFFLOAD is not set
1177,1178c1181,1183
< # CONFIG_NFT_TUNNEL is not set
< # CONFIG_NFT_OBJREF is not set
---
> CONFIG_NFT_NAT=m
> CONFIG_NFT_TUNNEL=m
> CONFIG_NFT_OBJREF=m
1180c1185
< # CONFIG_NFT_QUOTA is not set
---
> CONFIG_NFT_QUOTA=m
1181a1187
> CONFIG_NFT_REJECT_INET=m
1183a1190,1191
> CONFIG_NFT_FIB=m
> # CONFIG_NFT_FIB_INET is not set
1185c1193
< # CONFIG_NFT_SOCKET is not set
---
> CONFIG_NFT_SOCKET=m
1189c1197,1202
< # CONFIG_NF_FLOW_TABLE is not set
---
> CONFIG_NF_DUP_NETDEV=m
> CONFIG_NFT_DUP_NETDEV=m
> CONFIG_NFT_FWD_NETDEV=m
> # CONFIG_NFT_FIB_NETDEV is not set
> CONFIG_NF_FLOW_TABLE_INET=m
> CONFIG_NF_FLOW_TABLE=m
1354,1355c1367,1372
< # CONFIG_NF_TABLES_IPV4 is not set
< # CONFIG_NF_TABLES_ARP is not set
---
> CONFIG_NF_TABLES_IPV4=y
> CONFIG_NFT_REJECT_IPV4=m
> CONFIG_NFT_DUP_IPV4=m
> CONFIG_NFT_FIB_IPV4=m
> CONFIG_NF_TABLES_ARP=y
> # CONFIG_NF_FLOW_TABLE_IPV4 is not set
1391c1408,1412
< # CONFIG_NF_TABLES_IPV6 is not set
---
> CONFIG_NF_TABLES_IPV6=y
> CONFIG_NFT_REJECT_IPV6=m
> CONFIG_NFT_DUP_IPV6=m
> CONFIG_NFT_FIB_IPV6=m
> CONFIG_NF_FLOW_TABLE_IPV6=m
1420a1442
> CONFIG_NFT_BRIDGE_REJECT=m
1422c1444
< # CONFIG_NF_CONNTRACK_BRIDGE is not set
---
> CONFIG_NF_CONNTRACK_BRIDGE=m
1623a1646
> # CONFIG_NET_ACT_CT is not set

Fixes: https://tracker.ceph.com/issues/45531
Signed-off-by: Xiubo Li <xiubli@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1571 from ceph/wip-sepia-defaults2

sepia-fog-images: Update defaults

sepia-fog-images: Update defaults

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1570 from ceph/wip-obs

Support OpenSUSE Leap static slaves too

Support OpenSUSE Leap static slaves too

These variables should be passed on the command line when configuring or reconfiguring a static Jenkins slave.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1567 from ceph/wip-kernel-ubuntu-focal

kernel: turn on ubuntu focal builds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

kernel: turn on ubuntu focal builds

Commit 458332be6244 ("Add support for Ubuntu Focal Fossa globally")
added support for focal builds but didn't turn them on.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1559 from tspmelo/wip-cypress-key

ceph-dashboard-pull-requests: Add Cypress key credential

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1566 from tchaikov/wip-crimson-builddeps-fix

ceph-*-*: s/esac/fi/

ceph-*-*: s/esac/fi/

fix syntax error introduced by f934188b6cb95a56fd799ff2806a3f4ce30a9c58

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1564 from tchaikov/wip-crimson-builddeps

ceph-*-build: enable crimson build dependencies

ceph-*-build: enable crimson build dependencies

if $FLAVOR is crimson, crimson specific build dependencies should be
installed, otherwise we could have following failure when trying to
build crimson flavor of ceph packages:

error: Failed build dependencies:
fmt-devel is needed by ceph-2:16.0.0-1456.g311da94.el8.x86_64

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-dashboard-pull-requests: Add Cypress key credential

We need to have access to this key, to be able to record our e2e runs.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #1563 from ceph/wip-jjb-latest

jenkins-job-builder: Use latest pip

jenkins-job-builder: Use latest pip

I guess jjb needs a newer pip.  This job broke a few weeks ago after https://github.com/ceph/ceph-build/commit/21e7a7b2d3aa4d49b3a38c8eab494adb8f928a4b

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1562 from ceph/wip-unpin

ceph*setup: Unpin to trusty slaves

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph*setup: Unpin to trusty slaves

Not sure why this was needed in the first place but not only do we not have any actual 'trusty' slaves anymore.  Any Ubuntu slave should do.  I tried CentOS but `dch` is required and conflicts with RPM packaging tools.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1560 from ideepika/wip-crimson-el8

support crimson flavor for centos 8

add crimson flavor build for centos8

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1557 from ceph/wip-focal

Add support for Ubuntu Focal Fossa globally

ceph-dev*trigger: Build crimson on Focal instead of Bionic

Signed-off-by: David Galloway <dgallowa@redhat.com>

Add support for Ubuntu Focal Fossa globally

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1552 from tspmelo/wip-xvfb

ceph-dashboard-pull-requests: Install xvfb

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #1556 from ceph/wip-no-easyinstall

Get rid of easy_install and add functionality to install any pip version

scripts: Add ability to install a different pip version

@guits was seeing an issue in ceph-ansible jobs with pip 10.0.0.  This commit will allow individual jobs to specify "latest" or "pip==X.X.X" or "pip<20.0.0" as a second parameter to the `install_python_packages` functions.  The default behavior is still no second parameter and pip=10.0.0 will still be installed until we're ready to unpin that.

Signed-off-by: David Galloway <dgallowa@redhat.com>

scripts: Stop using easy_install to upgrade pip

A recent pip release caused `easy_install --upgrade pip` to break all builds on non-CentOS7 machines.  Let's assume it's safe to stop using easy_install to upgrade pip since we're not building on any ancient distros anymore.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1553 from ceph/guits-ca_doc_stable-5.0

ceph-ansible-docs: build doc for stable-5.0