David Galloway [Thu, 21 May 2020 16:52:14 +0000 (12:52 -0400)]
ansible: One playbook to rule them all
I/we got into a bad habit of updating one playbook to support X new distro or X new package but wouldn't update the others.
I think having all the tasks for all the types of slaves in one playbook will help keep things homogenous. And the playbooks are still idempotent (they can't be run while a job is running of course).
Signed-off-by: David Galloway <dgallowa@redhat.com>
Xiubo Li [Fri, 15 May 2020 00:54:42 +0000 (20:54 -0400)]
kernel: update Kconfig templates to 5.7-rc5
Because of how "make olddefconfig" works, we were missing netfilter
options and couldn't configure INPUT/FORWARD/OUTPUT chains needed for
kcephfs suite (unshare_ns_mount.sh, mount.py).
Update our Kconfig templates to 5.7-rc5. This has been generated
from the actual config (/boot/config-*) of one of the latest kernel
builds in shaman (as opposed to e.g. taking a distro config).
Here is the diff against the actual deb config:
14a18
> CONFIG_CC_HAS_ASM_INLINE=y
1199,1201c1203,1205
< # CONFIG_NF_TABLES_INET is not set
< # CONFIG_NF_TABLES_NETDEV is not set
< # CONFIG_NFT_NUMGEN is not set
---
> CONFIG_NF_TABLES_INET=y
> CONFIG_NF_TABLES_NETDEV=y
> CONFIG_NFT_NUMGEN=m 1202a1207
> # CONFIG_NFT_FLOW_OFFLOAD is not set 1208a1214
> CONFIG_NFT_NAT=m 1213a1220
> CONFIG_NFT_REJECT_INET=m 1215a1223,1224
> CONFIG_NFT_FIB=m
> # CONFIG_NFT_FIB_INET is not set
1217,1221c1226,1235
< # CONFIG_NFT_SOCKET is not set
< # CONFIG_NFT_OSF is not set
< # CONFIG_NFT_TPROXY is not set
< # CONFIG_NFT_SYNPROXY is not set
< # CONFIG_NF_FLOW_TABLE is not set
---
> CONFIG_NFT_SOCKET=m
> CONFIG_NFT_OSF=m
> CONFIG_NFT_TPROXY=m
> CONFIG_NFT_SYNPROXY=m
> CONFIG_NF_DUP_NETDEV=m
> CONFIG_NFT_DUP_NETDEV=m
> CONFIG_NFT_FWD_NETDEV=m
> # CONFIG_NFT_FIB_NETDEV is not set
> CONFIG_NF_FLOW_TABLE_INET=m
> CONFIG_NF_FLOW_TABLE=m
1386,1387c1400,1405
< # CONFIG_NF_TABLES_IPV4 is not set
< # CONFIG_NF_TABLES_ARP is not set
---
> CONFIG_NF_TABLES_IPV4=y
> CONFIG_NFT_REJECT_IPV4=m
> CONFIG_NFT_DUP_IPV4=m
> CONFIG_NFT_FIB_IPV4=m
> CONFIG_NF_TABLES_ARP=y
> CONFIG_NF_FLOW_TABLE_IPV4=m 1423c1441,1445
< # CONFIG_NF_TABLES_IPV6 is not set
---
> CONFIG_NF_TABLES_IPV6=y
> CONFIG_NFT_REJECT_IPV6=m
> CONFIG_NFT_DUP_IPV6=m
> CONFIG_NFT_FIB_IPV6=m
> CONFIG_NF_FLOW_TABLE_IPV6=m 1459a1482
> CONFIG_NFT_BRIDGE_REJECT=m 1661a1685
> # CONFIG_NET_ACT_CT is not set
Here is the diff against the actual rpm config:
1167,1169c1170,1172
< # CONFIG_NF_TABLES_INET is not set
< # CONFIG_NF_TABLES_NETDEV is not set
< # CONFIG_NFT_NUMGEN is not set
---
> CONFIG_NF_TABLES_INET=y
> CONFIG_NF_TABLES_NETDEV=y
> CONFIG_NFT_NUMGEN=m 1170a1174
> # CONFIG_NFT_FLOW_OFFLOAD is not set
1177,1178c1181,1183
< # CONFIG_NFT_TUNNEL is not set
< # CONFIG_NFT_OBJREF is not set
---
> CONFIG_NFT_NAT=m
> CONFIG_NFT_TUNNEL=m
> CONFIG_NFT_OBJREF=m 1180c1185
< # CONFIG_NFT_QUOTA is not set
---
> CONFIG_NFT_QUOTA=m 1181a1187
> CONFIG_NFT_REJECT_INET=m 1183a1190,1191
> CONFIG_NFT_FIB=m
> # CONFIG_NFT_FIB_INET is not set 1185c1193
< # CONFIG_NFT_SOCKET is not set
---
> CONFIG_NFT_SOCKET=m 1189c1197,1202
< # CONFIG_NF_FLOW_TABLE is not set
---
> CONFIG_NF_DUP_NETDEV=m
> CONFIG_NFT_DUP_NETDEV=m
> CONFIG_NFT_FWD_NETDEV=m
> # CONFIG_NFT_FIB_NETDEV is not set
> CONFIG_NF_FLOW_TABLE_INET=m
> CONFIG_NF_FLOW_TABLE=m
1354,1355c1367,1372
< # CONFIG_NF_TABLES_IPV4 is not set
< # CONFIG_NF_TABLES_ARP is not set
---
> CONFIG_NF_TABLES_IPV4=y
> CONFIG_NFT_REJECT_IPV4=m
> CONFIG_NFT_DUP_IPV4=m
> CONFIG_NFT_FIB_IPV4=m
> CONFIG_NF_TABLES_ARP=y
> # CONFIG_NF_FLOW_TABLE_IPV4 is not set 1391c1408,1412
< # CONFIG_NF_TABLES_IPV6 is not set
---
> CONFIG_NF_TABLES_IPV6=y
> CONFIG_NFT_REJECT_IPV6=m
> CONFIG_NFT_DUP_IPV6=m
> CONFIG_NFT_FIB_IPV6=m
> CONFIG_NF_FLOW_TABLE_IPV6=m 1420a1442
> CONFIG_NFT_BRIDGE_REJECT=m 1422c1444
< # CONFIG_NF_CONNTRACK_BRIDGE is not set
---
> CONFIG_NF_CONNTRACK_BRIDGE=m 1623a1646
> # CONFIG_NET_ACT_CT is not set
Fixes: https://tracker.ceph.com/issues/45531 Signed-off-by: Xiubo Li <xiubli@redhat.com> Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Kefu Chai [Sat, 9 May 2020 08:20:45 +0000 (16:20 +0800)]
ceph-*-build: enable crimson build dependencies
if $FLAVOR is crimson, crimson specific build dependencies should be
installed, otherwise we could have following failure when trying to
build crimson flavor of ceph packages:
error: Failed build dependencies:
fmt-devel is needed by ceph-2:16.0.0-1456.g311da94.el8.x86_64
David Galloway [Thu, 30 Apr 2020 16:20:51 +0000 (12:20 -0400)]
ceph*setup: Unpin to trusty slaves
Not sure why this was needed in the first place but not only do we not have any actual 'trusty' slaves anymore. Any Ubuntu slave should do. I tried CentOS but `dch` is required and conflicts with RPM packaging tools.
Signed-off-by: David Galloway <dgallowa@redhat.com>
David Galloway [Tue, 21 Apr 2020 19:00:49 +0000 (15:00 -0400)]
scripts: Add ability to install a different pip version
@guits was seeing an issue in ceph-ansible jobs with pip 10.0.0. This commit will allow individual jobs to specify "latest" or "pip==X.X.X" or "pip<20.0.0" as a second parameter to the `install_python_packages` functions. The default behavior is still no second parameter and pip=10.0.0 will still be installed until we're ready to unpin that.
Signed-off-by: David Galloway <dgallowa@redhat.com>
David Galloway [Tue, 21 Apr 2020 18:57:17 +0000 (14:57 -0400)]
scripts: Stop using easy_install to upgrade pip
A recent pip release caused `easy_install --upgrade pip` to break all builds on non-CentOS7 machines. Let's assume it's safe to stop using easy_install to upgrade pip since we're not building on any ancient distros anymore.
Signed-off-by: David Galloway <dgallowa@redhat.com>
* add "ceph-perf-classic" job. this job is defined using the job
template of "ceph-perf-{osd_flavor}"
* change the whitelist of "ceph-perf-crimson" and "ceph-perf-classic"
to "performance", so both of them can be enabled as long as the
"performance" label is attached to a PR.
please note, "ceph-perf-classic" job is using the test case and
acceptance critiera defined in `src/test/crimson/cbt/radosbench_4K_read.yaml`
for testing the performance. we need to adapt it for the needs of
classic OSD.
Because teuthology has functional and integration tests which
needs a connection to some services running in sepia and
some of the jenkins workers are running far away from the lab,
it takes significantly more time to run tests when using those
nodes. Like 6-7 minutes for running test on nodes in sepia,
and about 55-60 minutes outside.
so we need to remove `ceph-container` after pushing the created image
using `sudo` as well. otherwise, jenkins is not able to cleanup the
workspace using the "jenkins-build" user.
Sage Weil [Wed, 1 Apr 2020 22:46:25 +0000 (17:46 -0500)]
ceph-dev-*build: build container as root
Hopefully this avoids all of the podman/docker issues like
+ make clean.all
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: could not get runtime: open /proc/13238/ns/user: no such file or directory
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: could not get runtime: open /proc/13238/ns/user: no such file or directory
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: could not get runtime: open /proc/13238/ns/user: no such file or directory
+ login_docker_hub
+ echo 'Login in the Docker Hub'
Login in the Docker Hub
+ docker login -u **** -p **** quay.io
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: could not get runtime: open /proc/13238/ns/user: no such file or directory
Dimitri Savineau [Tue, 31 Mar 2020 18:47:15 +0000 (14:47 -0400)]
ceph-ansible-prs: run podman on stable-4.0 only
We don't need to run an extra podman scenario job on newer branches than
stable-4.0 because we're already using podman by default (CentOS 8) for
all container scenarios.
For older branches we don't support podman.
Nathan Cutler [Tue, 31 Mar 2020 10:21:11 +0000 (12:21 +0200)]
build_utils.sh: add pacific
I hope this will fix the FTBFS currently seen in Shaman on openSUSE Leap
15.2 builds of "master" branch:
++ case $ver in
++ rel=unknown
++ echo 'ERROR: Unknown release for version '\''16.0.0'\'''
ERROR: Unknown release for version '16.0.0'
++ echo unknown
++ exit 1
+ RELEASE_BRANCH=unknown
Dimitri Savineau [Fri, 27 Mar 2020 14:16:24 +0000 (10:16 -0400)]
quay-pruner: add CentOS 8 container support
Since Ceph Octopus release, the container image are based on CentOS 8
instead of 7 for previous releases.
The current quay pruner script only matches tags for CentOS 7.
This adds the CentOS 8 support.
Dimitri Savineau [Fri, 27 Mar 2020 13:29:09 +0000 (09:29 -0400)]
ceph-ansible-nightly: parametrize centos release
Since octopus the container image are based on CentOS 8 instead of 7 for
the previous Ceph releases.
The current regex to find the latest stable container image per Ceph
releases is hardcoding 7.
projects / ceph-build.git / log