Add sync-images job

This adds a nightly job in order to mirror some container
images (grafana image only for now) from docker.io to quay.io

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2207 from ceph/wip-bpf-cgroup

kernel: enable CONFIG_CGROUP_BPF

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #2202 from ceph/ceph-devstack

ceph-devstack: Add more parameters, and a trigger

kernel: enable CONFIG_CGROUP_BPF

It's required by podman.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2204 from ceph/wip-rpm-bpf-syscall

kernel: enable CONFIG_BPF_SYSCALL for rpm

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #2203 from ceph/wip-include-to-include-tasks

ansible: replace include: by include_tasks: and undo pin of ansible version

Unpin ansible-core in ceph-build-pull-requests too

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-setup, ceph-tag: undo ansible pin to <9

ansible roles used by these jobs have had include: changed to
include_tasks:, which was the reason the pin had been done in the
first place

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible: stop using deprecated include:

Signed-off-by: Dan Mick <dmick@redhat.com>

kernel: enable CONFIG_BPF_SYSCALL for rpm

It's already enabled for deb and is required by podman.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2200 from ceph/whitelist-squid

build: whitelist squid branch for pr tests

Merge pull request #2201 from rkachach/fix_issue_deps_script

ceph-build: fixing dependencies script

ceph-build: fixing dependencies script

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

build: whitelist squid branch for pr tests

saw that some tests were not getting triggered in my squid PR
https://github.com/ceph/ceph/pull/55685

Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-devstack: Add more parameters, and a trigger

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2190 from rosinL/wip-pr-arm64

ceph-pull-requests-arm64: set test jobs number equal to build jobs number

Merge pull request #2197 from Matan-B/wip-matanb-crimson-remove-c8-revert

Revert "ceph-dev*: remove centos8 crimson builds"

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2198 from petrutlucian94/libvirt

windows: restart libvirtd.socket if the socket is missing

windows: restart libvirtd.socket if the socket is missing

The Windows job is failing because of libvirt failures, specifically
the libvirt unix socket is missing.

After some trial and error, looks like the libvirtd.socket service
needs to be restarted.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Revert "ceph-dev*: remove centos8 crimson builds"

This reverts commit 1984464b9a80f6857877aac0620737cbc40fd67c.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2196 from ceph/ceph-container-ceph-base-prerelease

ceph-container: set PRERELEASE env var

ceph-container: set PRERELEASE env var

otherwise, the build-ceph-base.sh script complains because this
variable has no default value.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2195 from petrutlucian94/libvirt

windows: explicitly enable libvirtd and install qemu-kvm

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: explicitly enable libvirtd and install qemu-kvm

Libvirt is still not available after installing the libvirt-daemon-system
package.

We'll explicitly enable the libvirtd service and install qemu-kvm.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2194 from petrutlucian94/libvirt

windows: install libvirt

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: install libvirt

The Windows jobs use Libvirt VMs to build Ceph and run the tests.

The issue is that we're assuming Libvirt to be preinstalled, which
isn't necessarily the case.

This change will explicitly install Libvirt.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2192 from dmick/wip-builder

Update builder to work better on centos7: local xml parser, python3 on centos7

Merge pull request #2193 from dmick/wip-pin-jjb

ceph-build-pull-requests: pin jenkins-job-builder to <6.0.0

ceph-build-pull-requests: pin jenkins-job-builder to <6.0.0

Something's happened in the new release of jjb that breaks our
yml templates.  There are hints here, perhaps.

https://groups.google.com/g/jenkins-job-builder/c/FQOIorLgi9g

For now, 5.1.0 seems to work.

Also pin it in jenkins-job-builder.

Signed-off-by: Dan Mick <dmick@redhat.com>

examples/builder.yml: Add 'update-ca-trust" step for CentOS 7

get_url to update agent.jar was failing with a cert error,
but jenkins.ceph.com's cert was fine; on a hunch I updated the
dynamic ca trust database and that resolved the error.  I can't
see that it would do any damage to do so unconditionally.

Signed-off-by: Dan Mick <dmick@redhat.com>

library/jenkins_node: Allow to work on centos7 by adding xml parser

xmltodict is exactly what I needed, but it's too hard to dig up
for centos7 (and overkill anyway).  Supply an implementation that
allows jenkins XML parsing and use it instead.

Also, use python3 on centos7, and tweak the packages a bit in
examples/builder.yml

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible.cfg: Help debugging by setting logger

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-pull-requests-arm64: set test jobs number equal to build jobs number

On arm nodes confusa*, there are enough memory to run `nproc` build jobs,
but the test jobs number `$(nproc) / 4` are too small, and make check test run pretty
slow.

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #2186 from rkachach/fix_issue_64013

project config: adding logic to create kvm default network

ceph-build: behave is now installed as part of run-tests script

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

project config: improving deps installation script stability
Fixes: https://tracker.ceph.com/issues/64013
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #2189 from dmick/wip-ceph-iscsi-stable-centos9

ceph-iscsi-stable: make centos9 work

ceph-iscsi-stable: make centos9 work

also remove centos7 builds

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2188 from dmick/wip-remove-debugsolver

build_utils.sh: remove --debugsolver

build_utils.sh: remove --debugsolver

It leaves root-owned files around which causes cleanup issues

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2178 from cbodley/wip-unblock-c9-containers

enable centos9 builds for ceph-iscsi, tcmu-runner

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Dan Mick <dan.mick@redhat.com>

tcmu-runner: disable glfs for lack of glusterfs-api-devel

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2185 from Matan-B/wip-matanb-crimson-remove-c8

ceph-dev*: remove centos8 crimson builds

ceph-dev*: remove centos8 crimson builds

remove any quincy crimson buils (not a supported release)
remove reef c8 builds
remove main c8 builds

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2184 from dmick/wip-debug-solver

build_utils.sh: try using dnf, adding debug, and using dnf clean all

build_utils.sh Add "dnf clean all" as a Big Hammer

There's no reason this should be necessary, but maybe it
works around the mysterious issue of "Problem: cannot install
both python3-devel-3.9.18-1.el9.x86_64 from appstream and
python3-devel-3.9.18-1.el9.x86_64 from @System"

Signed-off-by: Dan Mick <dmick@redhat.com>

build_utils.sh: try using dnf, adding -d 10 --debugsolver for more info

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2183 from dmick/wip-verbose-builddep

Add verbosity to yum-builddep to try to get more info about failure

Add verbosity to yum-builddep to try to get more info about failure

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2182 from dmick/wip-ceph-container-prerelease-build

Add ceph-container-prerelease-build

Add ceph-container-prerelease-build

Based on ceph-container-build-ceph-base-push-imgs, but
for building containers for testing before the release is
validated and committed.  Differences:

1) pulls packages from download.ceph.com/prerelease
2) does not attempt to build aarch64
3) pushes built container to quay.ceph.io:ceph/prerelease

The prerelease directories are password-protected to avoid any
early exposure to in-progress builds.

This requires corresponding changes to ceph-container.git.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2181 from dmick/wip-release-tools

sync-push: "staging" -> "prerelease", and allow specifying releasename-version

Allow specifying '<release>-<version>' to sync only one version

Signed-off-by: Dan Mick <dmick@redhat.com>

Change from "staging" to "prerelease"

ceph-container already uses the word "staging" for a specific
function; avoid confusion by standardizing on "prerelease" here

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2180 from ceph/wip-pin-ansible

ceph-setup, ceph-tag: pin ansible to < 9

ceph-setup, ceph-tag: pin ansible to < 9

This is a temporary hack until we have time to properly test
changing include to include_tasks, required by ansible 9 (include
has been deprecated for some time, and now it's officially gone)

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2179 from rhcs-dashboard/upgrade-grafana-9.4.12

ceph-grafana-trigger: update the grafana version to 9.4.12

ceph-grafana-trigger: update the grafana version to 9.4.12

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2160 from rzarzynski/wip-kickoff-squid

kickoff v19 squid

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

tcmu-runner: enable centos9

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-iscsi: enable centos9

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2170 from ceph/wip-kernel-trigger-timeout

kernel-trigger: increase clone timeout

Reviewed-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2169 from ceph/wip-kernel-rpmbuild-fix

kernel: unbreak RPM builds

Reviewed-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2177 from ceph/teuth-pr-disable

Remove teuthology-pull-requests

Remove teuthology-pull-requests

This job represents a small subset of what we test with GitHub Actions. It's
also unreliable, for reasons including - but not limited to - the fact that
it's not possible to request an agent with a particular OS version.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2175 from dmick/wip-ceph-build-pull-requests

ceph-build-pull-requests: rather than ansible, pin ansible-core

Merge pull request #2176 from tserlin/wip-tserlin-pin-to-V5.7

nfs-ganesha: pin the daily build to V5.7

nfs-ganesha: pin the daily build to V5.7

The latest upstream nfs-ganesha 5 tag was released. Bump from
V5.5 to V5.7.

Merge pull request #2166 from rkachach/fix_issue_63019

project config: adding rook orchestrator e2e testing job

ceph-build: adding quincy, pacific and octopus to blacklist

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

test: changing trigger-phrase

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

project config: adding rook orchestrator e2e testing job
Fixes: https://tracker.ceph.com/issues/63019
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #2172 from dmick/wip-sync-push

sync-push et al: add -ex to shebang, add staging area to sync-push

ceph-build-pull-requests: rather than ansible, pin ansible-core

Last change pinned ansible to 8.4.0 to workaround a bug.  The
buggy code is actually in ansible-core, so pin that instead.
See https://github.com/ceph/ceph-build/pull/2173

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2173 from dmick/wip-ceph-build-pull-requests

ceph-build-pull-requests/build/build: pin ansible to 8.4.0

Merge pull request #2174 from ceph/wip-bookworm

ceph-build, etc: add debian bookworm

ceph-build, etc: add debian bookworm

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ceph-build-pull-requests/build/build: pin ansible to 8.4.0

ansible 8.5.0 includes a change to ansible-galaxy that breaks
the installation of sensu and rabbitmq:

https://github.com/ansible/ansible/issues/82073

Work around that by pinning ansible to 8.4.0

Fixes: https://tracker.ceph.com/issues/63223
Signed-off-by: Dan Mick <dmick@redhat.com>

sync-push: add staging_dir for releases under construction

Signed-off-by: Dan Mick <dmick@redhat.com>

sync-pull, sign-rpms, sync-push: add -ex to shebang line

Show what we're doing, and exit on error

Signed-off-by: Dan Mick <dmick@redhat.com>

kernel-trigger: increase clone timeout

From time to time Jenkins kills "git index-pack" due to a timeout:

 > git fetch --no-tags --progress -- https://github.com/ceph/ceph-client.git +refs/heads/*:refs/remotes/origin/* # timeout=20
ERROR: Timeout after 20 minutes
...
remote: Enumerating objects: 9769655, done.
remote: Counting objects:   0% (1/368573)
...
remote: Counting objects: 100% (368573/368573), done.
remote: Compressing objects:   0% (1/73615)
...
remote: Compressing objects: 100% (73615/73615), done.
Receiving objects:   0% (1/9769655)
...
Receiving objects:  99% (9762992/9769655), 2.61 GiB | 19.22 MiB/s
remote: Total 9769655 (delta 294178), reused 368051 (delta 293885), pack-reused 9401082
Receiving objects: 100% (9769655/9769655), 2.62 GiB | 19.00 MiB/s
Receiving objects: 100% (9769655/9769655), 2.63 GiB | 19.17 MiB/s, done.
Resolving deltas:   0% (0/8264128)
...
Resolving deltas:  15% (1312818/8264128)
error: index-pack died of signal 15
fatal: index-pack failed

This step is CPU intensive, but it's very hard to tell what was going
on the worker node at the time and what made it slow.  As kernel builds
are pretty infrequent, just bump the timeout drastically.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

kernel: unbreak RPM builds

A bunch of kbuild changes went into 6.6-rc1.  In particular, the
location of rpmbuild structure has changed and BuildRequires clauses
are now applied universally so we need dwarves to be installed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2167 from batrick/debug-options

ceph-dev-new-setup: enable debug options for dev builds

ceph-dev-new-setup: enable debug options for dev builds

Note: it has been this way since at least 35e1a715. It's difficult to tell when
or even if ceph was ever properly built with debugging configurations for QA as
there are corresponding changes in ceph with the switch to cmake which makes
this challenging to evaluate.

It's likely that it was wrongly assumed that cmake would set the build type to
Debug because the ".git" directory would be present. This is not the case
because the "make-dist" script (executed below) creates a git tarball that is
used for the actual untar/build. See also:

https://github.com/ceph/ceph/pull/53800

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #2168 from rhcs-dashboard/node-version-upgrade

deps: upgrade the node version to 16

deps: upgrade the node version to 16

the reasoning for choosing the node 16 is because it supports angular
version 12, 13, 14 and 15. We have angular 12 in the pacific

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2164 from petrutlucian94/whitelist_ceph

windows: add Ceph dir to Windows Defender exclusion list

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: add Ceph dir to Windows Defender exclusion list

We're getting occasional false positives from Windows Defender.
For this reason, we'll add the Ceph dir to the exclusion list.

  ERROR - Test exception: [WinError 225] Operation did not complete
  successfully because the file contains a virus or potentially unwanted
  software. Total exceptions: 22

   Windows Defender Antivirus has taken action to protect this machine from
   malware or other potentially unwanted software.
    For more information please see the following:
   https://go.microsoft.com/fwlink/?linkid=37020&name=Trojan:Win32/Bearfoos.A!m
   l&threatid=2147731250&enterprise=0
        Name: Trojan:Win32/Bearfoos.A!ml
        ID: 2147731250
        Severity: Severe
        Category: Trojan
        Path: file:_C:\ceph\rbd-wnbd.exe
        Detection Origin: Local machine
        Detection Type: Concrete
        Detection Source: Real-Time Protection
        User: NT AUTHORITY\SYSTEM
        Process Name: C:\Program Files\Python311\python.exe
        Action: Quarantine
        Action Status:  No additional actions required
        Error Code: 0x80508023
        Error description: The program could not find the malware and other
                       potentially unwanted software on this device.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2163 from dmick/wip-sepia-fog-images-build

sepia-fog-images/build/build: bootstrap teuthology unconditionally

Whether or not we will teuthology-lock machines, we always need
teuthology-queue --pause

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2162 from petrutlucian94/win32_sv_start

windows: use "net.exe start" instead of Start-Service

windows: use "net.exe start" instead of Start-Service

The "Start-Serice" command doesn't really say why the service could
not be started:

  Start-Service : Service 'ceph-rbd (ceph-rbd)' cannot be started
  due to the following error: Cannot start service

We'll use the "net.exe start" command instead, which provides
additional information.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2161 from tserlin/wip-tserlin-pin-to-V5.5

nfs-ganesha: pin the daily build to V5.5

nfs-ganesha: pin the daily build to V5.5

Instead of just building the tip of whatever is in the "next"
branch, let's pin the daily nfs-ganesha build to the latest
stable tag: V5.5.

ceph-dev-build: update {build,setup}_osc for squid

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

scripts/build_utils: map 19.* to squid

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #2159 from ljflores/wip-lower-concurrency

scripts: lower concurrency for many-core machines

scripts: lower concurrency for many-core machines

adami nodes have 96 processing units (nproc). Based on the old script
logic, we were compiling 90 build jobs with 96 processing units. This
combination (96/90) ends up causing many instances of memory
overconsumption on adami nodes.

With this new logic, we take into account that jobs take up more memory
than we expect. This will make it so adami nodes will compile 67 build
jobs on 96 processing units, which will hopefully avoid so many instances
of memory overconsumption. (Total memory on adami nodes is generally
~270036 MiB, so 270036 / 4000 = 67)

braggi nodes do not have this problem; they have 48 processing units.
It has been working so far to compile 48 build jobs with 48 processing
units on the braggi nodes, and this new logic will not change that or
any other node with nproc <= 50.

Fixes: https://tracker.ceph.com/issues/57296
Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #2157 from dmick/wip-jenkins-builders

Update jenkins-builder support

Merge pull request #2158 from ceph/fix-ca-flake8

ceph-container-flake8: migrate to alpine/flake8 image