ceph-container-build-ceph-base-push-imgs*: allow specifying FLAVOR

Add parameters to set variables that the job scripts will look for
to choose which flavors to build.  This will override the defaults in
ceph-container.git; the defaults are set to the same as
ceph-container.git at this moment, but are probably soon to change to
centos,9.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2247 from dmick/wip-fix-include-with-gh-clone

Replace obsolete "include:' in upstream ansible tasks

Merge pull request #2249 from dmick/wip-fix-ceph-container-yml-names

ceph-container-*: make .yml filenames match job names

ceph-container-*: make .yml filenames match job names

Why intentionally confuse people?

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2248 from dmick/wip-fix-container-job-display-names

ceph-container-build-*: make display-name the job name

Merge pull request #2242 from Svelar/extend_arm_build_time

ceph-pull-requests-arm64: extend timeout to 4h

Merge pull request #2243 from dmick/wip-scripts-ceph-iscsi

Modify scripts/{sync-pull,sign-rpms,sync-push} to handle ceph-iscsi

ceph-container-build-*: make display-name the job name

Some of these were using the *same display-name for different jobs*.
O_o

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-build-pull-requests: listen for a '.*retest.*' PR comment

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible/requirements: Use private github repo to fix include:

ansible has removed include:.  Create clones of the two
repos in sensu-requirements.yml to change it to import_tasks:.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2239 from dmick/wip-use-base-deb-mirror

build_utils.sh: re: last commit: one must specify a mirror

Merge pull request #2244 from dmick/wip-docs-only-for-admin-dir

scripts/build_utils.sh: set DOCS_ONLY for PRs with only admin/ files,…

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

scripts/build_utils.sh: set DOCS_ONLY for PRs with only admin/ files, too

DOCS_ONLY is set if the filenames all start with docs/, but it's
also true that if the filenames start with admin/, it's strictly
docs related (discovered for https://github.com/ceph/ceph/pull/57649)

Signed-off-by: Dan Mick <dmick@redhat.com>

scripts/sync-push: add projectname to handle ceph-iscsi

ceph-iscsi also needs a repo file created
Drop el7
Add usage

Signed-off-by: Dan Mick <dan.mick@redhat.com>

scripts/sync-pull: changes for ceph-iscsi, drop el7

Several sections are done only for the 'ceph' project (to
accommodate ceph-iscsi, which has no debs, so doesn't need
to find the most-populated distroversion or create debs,
pull a tarball, etc.).

Also drop el7.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

sign-rpms: add 'project' argument

This allows signing ceph-iscsi packages

Signed-off-by: Dan Mick <dan.mick@redhat.com>

scripts: add vim modelines for editing ease

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2241 from dmick/wip-allow-arm64-for-setup

Allow all the ceph -setup jobs to run on arm64.

ceph-pull-requests-arm64: extend timeout to 4h

Because performance of omani series machine are not so well which caused make check often failed, especially when cache hit was low, extend timeout to 4h seems reasonable.

Ref build duration: https://jenkins.ceph.com/job/ceph-pull-requests-arm64/buildTimeTrend

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #2240 from petrutlucian94/apt_timeout

windows: increase package install timeout

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Allow all the ceph -setup jobs to run on arm64.

They only do file operations to build a source tarball and debian source
package things, so the architecture isn't important.

Signed-off-by: Dan Mick <dmick@redhat.com>

windows: increase package install timeout

One "apt-get install" command times out after the default 30 seconds.
We'll increase the timeout to 10m.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

build_utils.sh: re: last commit: one must specify a mirror

To pbuild Debian on Ubuntu (as we do), one must specify a mirror
that contains the Debian environments.  Use the most-generic one.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2238 from dmick/wip-remove-deb-mirror

build_utils.sh: remove pbuilder's --mirror option for Debian

build_utils.sh: remove pbuilder's --mirror option for Debian

The existing mirror is defunct, and hard-stopping bookworm builds.
For now, let's just revert to not specifying a mirror (as we have
in the past for other repo configurations) and solve any problems
that result if they pop up.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2222 from dmick/wip-add-c9-containers

ceph-dev-build,ceph-dev-new-build: build c9 containers

ceph-dev-build,ceph-dev-new-build: build c9 containers

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2234 from dmick/wip-chacra-postgres-version

chacra-pull-requests: set postgres version for jammy

chacra-pull-requests: set postgres version for jammy

next time: can't we just use a generic "install postgresql"

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2233 from dmick/wip-deprecated-include

{chacra,shaman}-pull-requests: replace deprecated include:

{chacra,shaman}-pull-requests: replace deprecated include:

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2232 from petrutlucian94/log_timeout

windows: increase log collection timeout

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: increase log collection timeout

Some failed Windows jobs are missing the build artifacts and it
seems like the log collection function times out.

https://jenkins.ceph.com/job/ceph-windows-pull-requests/39380/

We'll increase the timeout from 30s to 10m.

While at it, we're updating the scp_download and scp_upload helpers,
checking for timeout errors.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2231 from ceph/wip-nuke-ceph-doc-releases-rtd

remove ceph-doc-releases-rtd job

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

remove ceph-doc-releases-rtd job

Rebuilding doc/releases in release branches was replaced with
redirects:

  https://github.com/ceph/ceph/pull/48800
  https://github.com/ceph/ceph/pull/48945

Also, this job never actually ran because shallow cloning is
incompatible with included-regions filter.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2230 from ceph/wip-lusov-build-timestamps

ceph-dev-new-build: add timestamps to the build log

ceph-dev-new-build: add timestamps to the build log

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #2229 from petrutlucian94/increase_timeouts

windows: increase timeouts

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: increase timeouts

We've decreased the amount of resources allocated to Windows PR test
jobs but as a result we're now experiencing timeouts.

We'll double the timeouts used by our test scripts.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2228 from petrutlucian94/win_reduce_vcpus

windows: limit number of Linux VM vcpus

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: limit number of Linux VM vcpus

We reduced the amount of memory allocated to Linux VMs used by the
Windows PR jobs to 16GB but we're now hitting OOM errors while
building Ceph.

To mitigate this, we'll limit the number of parallel jobs and
allocated vcpus.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2227 from petrutlucian94/reduce_win_mem

windows: reduce Linux VM memory

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

windows: reduce Linux VM memory

There have been a lot of Windows job failures lately due to
qemu errors caused by insufficient Jenkins slave memory:

  ERROR    internal error: process exited while connecting to
  monitor: 2024-04-21T11:37:28.922183Z qemu-system-x86_64: cannot
  set up guest memory 'pc.ram': Cannot allocate memory

Each Windows job spins up two virtual machines: a Linux one with
64GB of ram and a Windows VM with 8GB of ram. Since the Windows job
no longer uses memstore, we can probably reduce the Linux vm memory
to 16GB.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #2226 from ceph/cv-drop-pacific

ceph-volume: drop pacific nightly testing

ceph-volume: drop pacific nightly testing

This drops the pacific nightly ceph-volume testing

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2225 from ceph/cv-test-suite

ceph-volume: update 'test suite' definition

ceph-volume: update 'test suite' definition

corresponding PR: ceph/ceph#56827

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2224 from ceph/ceph-volume_drop_fs

ceph-volume: drop filestore jobs

ceph-volume: drop filestore jobs

filestore is no longer supported.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2223 from ceph/ceph-volume-update-tests

ceph-volume: update jobs definition

ceph-volume: update jobs definition

drop filestore testing
make centos distro value more generic (el8 and el9)

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2221 from cbodley/wip-squid-no-focal

ceph-dev-new-trigger: stop building focal packages on squid

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #2220 from cbodley/wip-main-no-focal

ceph-dev-new-trigger: stop building focal packages on main

ceph-dev-new-trigger: stop building focal packages on squid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-dev-new-trigger: stop building focal packages on main

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2216 from ceph/wip-squid

Add squid to ceph-dev-cron and other jobs (except for ceph-volume)

Reviewed-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2214 from dmick/wip-add-arm64-centos9

ceph-dev-new-build: allow arm64 builds of centos9

Merge pull request #2206 from cbodley/wip-quincy-on-jammy

ceph-dev-new-trigger: start building quincy packages on jammy

Merge pull request #2215 from Svelar/arm_ci

ceph-pull-requests-arm64: make publishers same as ceph-pull-requests

Merge pull request #2217 from Svelar/zbd

ceph-pull-requests: remove WITH_ZBD

Merge pull request #2218 from Svelar/arm64_RWL

ceph-pull-requests-arm64: enable RWL mode on arm64

ceph-pull-requests-arm64: enable RWL mode on arm64

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

ceph-pull-requests: remove WITH_ZBD

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Update releases - add squid, remove pacific and older

... with the exception of ceph-volume jobs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-dev-trigger: fix a typo a regex for reef

It should be .*reef.*, not .*main.*.  This was introduced in commit
a19480349163 ("Update releases - add reef, remove octopus and older"),
but didn't really affect anything because the build matrix happens to
be the same.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-pull-requests-arm64: add kill tests when aborted to arm64

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

ceph-pull-requests-arm64: add publishers to fix ci result inaccuracy

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #2213 from clwluvw/collect-logs

build_utils: move collect lgos playbook into the root dir

ceph-dev-new-build: allow arm64 builds of centos9

Signed-off-by: Dan Mick <dmick@redhat.com>

build_utils: move collect lgos playbook into the root dir

The playbook needs to be in the root directory so it can discover the roles

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #2212 from clwluvw/ceph-ansible-collect-logs

build_utils: use custom playbook for collect_ceph_logs

build_utils: use custom playbook for collect_ceph_logs

If provided, use the custom playbook to collect logs.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #2191 from baum/centos8_build_deps

centos8: add yum Configs for  ceph/grpc copr https://copr.fedorainfracloud.org/coprs/ceph/grpc/

Merge pull request #2211 from ceph/update-sync-images3

sync-images: address permission denied issue

sync-images: address permission denied issue

adding `--security-opt label=disable` to workaround the following error:

```
time="2024-02-29T12:04:24Z" level=fatal msg="open /sync.yml: permission denied"
```

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2210 from ceph/update-sync-images2

sync-images: write sync.yml in workspace

sync-images: write sync.yml in workspace

I might be lacking JJB expertise but this is the only
way I could figure out in order to get `sync.yml` in the
workspace.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2209 from ceph/update-sync-images

sync-images: fix the inclusion section

sync-images: fix the inclusion section

according to the documentation [1], my understanding is that
`!include-raw` should be used instead of `!include-raw-escape`.

[1] https://jenkins-job-builder.readthedocs.io/en/latest/definition.html#inclusion-tags

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2208 from ceph/sync-images

Add sync-images job

Add sync-images job

This adds a nightly job in order to mirror some container
images (grafana image only for now) from docker.io to quay.io

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2207 from ceph/wip-bpf-cgroup

kernel: enable CONFIG_CGROUP_BPF

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #2202 from ceph/ceph-devstack

ceph-devstack: Add more parameters, and a trigger

kernel: enable CONFIG_CGROUP_BPF

It's required by podman.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2204 from ceph/wip-rpm-bpf-syscall

kernel: enable CONFIG_BPF_SYSCALL for rpm

Reviewed-by: Xiubo Li <xiubli@redhat.com>

ceph-dev-new-trigger: start building quincy packages on jammy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2203 from ceph/wip-include-to-include-tasks

ansible: replace include: by include_tasks: and undo pin of ansible version

Unpin ansible-core in ceph-build-pull-requests too

Signed-off-by: Dan Mick <dmick@redhat.com>

ceph-setup, ceph-tag: undo ansible pin to <9

ansible roles used by these jobs have had include: changed to
include_tasks:, which was the reason the pin had been done in the
first place

Signed-off-by: Dan Mick <dmick@redhat.com>

ansible: stop using deprecated include:

Signed-off-by: Dan Mick <dmick@redhat.com>

centos8: grpc-devel

  - see https://copr.fedorainfracloud.org/coprs/ceph/grpc/

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

kernel: enable CONFIG_BPF_SYSCALL for rpm

It's already enabled for deb and is required by podman.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2200 from ceph/whitelist-squid

build: whitelist squid branch for pr tests

Merge pull request #2201 from rkachach/fix_issue_deps_script

ceph-build: fixing dependencies script

ceph-build: fixing dependencies script

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

build: whitelist squid branch for pr tests

saw that some tests were not getting triggered in my squid PR
https://github.com/ceph/ceph/pull/55685

Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-devstack: Add more parameters, and a trigger

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2190 from rosinL/wip-pr-arm64

ceph-pull-requests-arm64: set test jobs number equal to build jobs number

Merge pull request #2197 from Matan-B/wip-matanb-crimson-remove-c8-revert

Revert "ceph-dev*: remove centos8 crimson builds"

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2198 from petrutlucian94/libvirt

windows: restart libvirtd.socket if the socket is missing