scripts/build_utils.sh: set DOCS_ONLY for PRs with only admin/ files, too

DOCS_ONLY is set if the filenames all start with docs/, but it's
also true that if the filenames start with admin/, it's strictly
docs related (discovered for https://github.com/ceph/ceph/pull/57649)

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2241 from dmick/wip-allow-arm64-for-setup

Allow all the ceph -setup jobs to run on arm64.

Merge pull request #2240 from petrutlucian94/apt_timeout

windows: increase package install timeout

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Allow all the ceph -setup jobs to run on arm64.

They only do file operations to build a source tarball and debian source
package things, so the architecture isn't important.

Signed-off-by: Dan Mick <dmick@redhat.com>

windows: increase package install timeout

One "apt-get install" command times out after the default 30 seconds.
We'll increase the timeout to 10m.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2238 from dmick/wip-remove-deb-mirror

build_utils.sh: remove pbuilder's --mirror option for Debian

build_utils.sh: remove pbuilder's --mirror option for Debian

The existing mirror is defunct, and hard-stopping bookworm builds.
For now, let's just revert to not specifying a mirror (as we have
in the past for other repo configurations) and solve any problems
that result if they pop up.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2222 from dmick/wip-add-c9-containers

ceph-dev-build,ceph-dev-new-build: build c9 containers

ceph-dev-build,ceph-dev-new-build: build c9 containers

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2234 from dmick/wip-chacra-postgres-version

chacra-pull-requests: set postgres version for jammy

chacra-pull-requests: set postgres version for jammy

next time: can't we just use a generic "install postgresql"

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2233 from dmick/wip-deprecated-include

{chacra,shaman}-pull-requests: replace deprecated include:

{chacra,shaman}-pull-requests: replace deprecated include:

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2232 from petrutlucian94/log_timeout

windows: increase log collection timeout

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: increase log collection timeout

Some failed Windows jobs are missing the build artifacts and it
seems like the log collection function times out.

https://jenkins.ceph.com/job/ceph-windows-pull-requests/39380/

We'll increase the timeout from 30s to 10m.

While at it, we're updating the scp_download and scp_upload helpers,
checking for timeout errors.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2231 from ceph/wip-nuke-ceph-doc-releases-rtd

remove ceph-doc-releases-rtd job

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

remove ceph-doc-releases-rtd job

Rebuilding doc/releases in release branches was replaced with
redirects:

  https://github.com/ceph/ceph/pull/48800
  https://github.com/ceph/ceph/pull/48945

Also, this job never actually ran because shallow cloning is
incompatible with included-regions filter.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2230 from ceph/wip-lusov-build-timestamps

ceph-dev-new-build: add timestamps to the build log

ceph-dev-new-build: add timestamps to the build log

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #2229 from petrutlucian94/increase_timeouts

windows: increase timeouts

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: increase timeouts

We've decreased the amount of resources allocated to Windows PR test
jobs but as a result we're now experiencing timeouts.

We'll double the timeouts used by our test scripts.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2228 from petrutlucian94/win_reduce_vcpus

windows: limit number of Linux VM vcpus

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: limit number of Linux VM vcpus

We reduced the amount of memory allocated to Linux VMs used by the
Windows PR jobs to 16GB but we're now hitting OOM errors while
building Ceph.

To mitigate this, we'll limit the number of parallel jobs and
allocated vcpus.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2227 from petrutlucian94/reduce_win_mem

windows: reduce Linux VM memory

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: reduce Linux VM memory

There have been a lot of Windows job failures lately due to
qemu errors caused by insufficient Jenkins slave memory:

  ERROR    internal error: process exited while connecting to
  monitor: 2024-04-21T11:37:28.922183Z qemu-system-x86_64: cannot
  set up guest memory 'pc.ram': Cannot allocate memory

Each Windows job spins up two virtual machines: a Linux one with
64GB of ram and a Windows VM with 8GB of ram. Since the Windows job
no longer uses memstore, we can probably reduce the Linux vm memory
to 16GB.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2226 from ceph/cv-drop-pacific

ceph-volume: drop pacific nightly testing

ceph-volume: drop pacific nightly testing

This drops the pacific nightly ceph-volume testing

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2225 from ceph/cv-test-suite

ceph-volume: update 'test suite' definition

ceph-volume: update 'test suite' definition

corresponding PR: ceph/ceph#56827

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2224 from ceph/ceph-volume_drop_fs

ceph-volume: drop filestore jobs

ceph-volume: drop filestore jobs

filestore is no longer supported.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2223 from ceph/ceph-volume-update-tests

ceph-volume: update jobs definition

ceph-volume: update jobs definition

drop filestore testing
make centos distro value more generic (el8 and el9)

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2221 from cbodley/wip-squid-no-focal

ceph-dev-new-trigger: stop building focal packages on squid

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #2220 from cbodley/wip-main-no-focal

ceph-dev-new-trigger: stop building focal packages on main

ceph-dev-new-trigger: stop building focal packages on squid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-dev-new-trigger: stop building focal packages on main

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2216 from ceph/wip-squid

Add squid to ceph-dev-cron and other jobs (except for ceph-volume)

Reviewed-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2214 from dmick/wip-add-arm64-centos9

ceph-dev-new-build: allow arm64 builds of centos9

Merge pull request #2206 from cbodley/wip-quincy-on-jammy

ceph-dev-new-trigger: start building quincy packages on jammy

Merge pull request #2215 from Svelar/arm_ci

ceph-pull-requests-arm64: make publishers same as ceph-pull-requests

Merge pull request #2217 from Svelar/zbd

ceph-pull-requests: remove WITH_ZBD

Merge pull request #2218 from Svelar/arm64_RWL

ceph-pull-requests-arm64: enable RWL mode on arm64

ceph-pull-requests-arm64: enable RWL mode on arm64

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

ceph-pull-requests: remove WITH_ZBD

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Update releases - add squid, remove pacific and older

... with the exception of ceph-volume jobs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-dev-trigger: fix a typo a regex for reef

It should be .*reef.*, not .*main.*.  This was introduced in commit
a19480349163 ("Update releases - add reef, remove octopus and older"),
but didn't really affect anything because the build matrix happens to
be the same.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-pull-requests-arm64: add kill tests when aborted to arm64

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

ceph-pull-requests-arm64: add publishers to fix ci result inaccuracy

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #2213 from clwluvw/collect-logs

build_utils: move collect lgos playbook into the root dir

ceph-dev-new-build: allow arm64 builds of centos9

Signed-off-by: Dan Mick <dmick@redhat.com>

build_utils: move collect lgos playbook into the root dir

The playbook needs to be in the root directory so it can discover the roles

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #2212 from clwluvw/ceph-ansible-collect-logs

build_utils: use custom playbook for collect_ceph_logs

build_utils: use custom playbook for collect_ceph_logs

If provided, use the custom playbook to collect logs.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #2191 from baum/centos8_build_deps

centos8: add yum Configs for  ceph/grpc copr https://copr.fedorainfracloud.org/coprs/ceph/grpc/

Merge pull request #2211 from ceph/update-sync-images3

sync-images: address permission denied issue

sync-images: address permission denied issue

adding `--security-opt label=disable` to workaround the following error:

```
time="2024-02-29T12:04:24Z" level=fatal msg="open /sync.yml: permission denied"
```

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2210 from ceph/update-sync-images2

sync-images: write sync.yml in workspace

sync-images: write sync.yml in workspace

I might be lacking JJB expertise but this is the only
way I could figure out in order to get `sync.yml` in the
workspace.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2209 from ceph/update-sync-images

sync-images: fix the inclusion section

sync-images: fix the inclusion section

according to the documentation [1], my understanding is that
`!include-raw` should be used instead of `!include-raw-escape`.

[1] https://jenkins-job-builder.readthedocs.io/en/latest/definition.html#inclusion-tags

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2208 from ceph/sync-images

Add sync-images job

Add sync-images job

This adds a nightly job in order to mirror some container
images (grafana image only for now) from docker.io to quay.io

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2207 from ceph/wip-bpf-cgroup

kernel: enable CONFIG_CGROUP_BPF

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #2202 from ceph/ceph-devstack

ceph-devstack: Add more parameters, and a trigger

kernel: enable CONFIG_CGROUP_BPF

It's required by podman.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2204 from ceph/wip-rpm-bpf-syscall

kernel: enable CONFIG_BPF_SYSCALL for rpm

Reviewed-by: Xiubo Li <xiubli@redhat.com>

ceph-dev-new-trigger: start building quincy packages on jammy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2203 from ceph/wip-include-to-include-tasks

ansible: replace include: by include_tasks: and undo pin of ansible version

Unpin ansible-core in ceph-build-pull-requests too

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-setup, ceph-tag: undo ansible pin to <9

ansible roles used by these jobs have had include: changed to
include_tasks:, which was the reason the pin had been done in the
first place

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible: stop using deprecated include:

Signed-off-by: Dan Mick <dmick@redhat.com>

centos8: grpc-devel

  - see https://copr.fedorainfracloud.org/coprs/ceph/grpc/

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

kernel: enable CONFIG_BPF_SYSCALL for rpm

It's already enabled for deb and is required by podman.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2200 from ceph/whitelist-squid

build: whitelist squid branch for pr tests

Merge pull request #2201 from rkachach/fix_issue_deps_script

ceph-build: fixing dependencies script

ceph-build: fixing dependencies script

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

build: whitelist squid branch for pr tests

saw that some tests were not getting triggered in my squid PR
https://github.com/ceph/ceph/pull/55685

Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-devstack: Add more parameters, and a trigger

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2190 from rosinL/wip-pr-arm64

ceph-pull-requests-arm64: set test jobs number equal to build jobs number

Merge pull request #2197 from Matan-B/wip-matanb-crimson-remove-c8-revert

Revert "ceph-dev*: remove centos8 crimson builds"

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2198 from petrutlucian94/libvirt

windows: restart libvirtd.socket if the socket is missing

windows: restart libvirtd.socket if the socket is missing

The Windows job is failing because of libvirt failures, specifically
the libvirt unix socket is missing.

After some trial and error, looks like the libvirtd.socket service
needs to be restarted.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Revert "ceph-dev*: remove centos8 crimson builds"

This reverts commit 1984464b9a80f6857877aac0620737cbc40fd67c.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2196 from ceph/ceph-container-ceph-base-prerelease

ceph-container: set PRERELEASE env var

ceph-container: set PRERELEASE env var

otherwise, the build-ceph-base.sh script complains because this
variable has no default value.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2195 from petrutlucian94/libvirt

windows: explicitly enable libvirtd and install qemu-kvm

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: explicitly enable libvirtd and install qemu-kvm

Libvirt is still not available after installing the libvirt-daemon-system
package.

We'll explicitly enable the libvirtd service and install qemu-kvm.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2194 from petrutlucian94/libvirt

windows: install libvirt

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: install libvirt

The Windows jobs use Libvirt VMs to build Ceph and run the tests.

The issue is that we're assuming Libvirt to be preinstalled, which
isn't necessarily the case.

This change will explicitly install Libvirt.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2192 from dmick/wip-builder

Update builder to work better on centos7: local xml parser, python3 on centos7

Merge pull request #2193 from dmick/wip-pin-jjb

ceph-build-pull-requests: pin jenkins-job-builder to <6.0.0

ceph-build-pull-requests: pin jenkins-job-builder to <6.0.0

Something's happened in the new release of jjb that breaks our
yml templates.  There are hints here, perhaps.

https://groups.google.com/g/jenkins-job-builder/c/FQOIorLgi9g

For now, 5.1.0 seems to work.

Also pin it in jenkins-job-builder.

Signed-off-by: Dan Mick <dmick@redhat.com>

examples/builder.yml: Add 'update-ca-trust" step for CentOS 7

get_url to update agent.jar was failing with a cert error,
but jenkins.ceph.com's cert was fine; on a hunch I updated the
dynamic ca trust database and that resolved the error.  I can't
see that it would do any damage to do so unconditionally.

Signed-off-by: Dan Mick <dmick@redhat.com>

library/jenkins_node: Allow to work on centos7 by adding xml parser

xmltodict is exactly what I needed, but it's too hard to dig up
for centos7 (and overkill anyway).  Supply an implementation that
allows jenkins XML parsing and use it instead.

Also, use python3 on centos7, and tweak the packages a bit in
examples/builder.yml

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible.cfg: Help debugging by setting logger

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-pull-requests-arm64: set test jobs number equal to build jobs number

On arm nodes confusa*, there are enough memory to run `nproc` build jobs,
but the test jobs number `$(nproc) / 4` are too small, and make check test run pretty
slow.

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #2186 from rkachach/fix_issue_64013

project config: adding logic to create kvm default network

ceph-build: behave is now installed as part of run-tests script

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

project config: improving deps installation script stability
Fixes: https://tracker.ceph.com/issues/64013
Signed-off-by: Redouane Kachach <rkachach@redhat.com>