Merge pull request #2327 from ceph/wip-skip-interp

ansible: Add tag to python interpreter var setting tasks

ansible: Add tag to python interpreter var setting tasks

I'm working on a combination of terraform and ansible to quickly
spin up instances in IBM Cloud so they can join Jenkins as builders.

Part of this process involves creating a virtualenv and running ansible
from it.  python-jenkins needs to be installed in the virtualenv and
setting the python interpreters inside the playbook breaks that.

This PR will let me `--skip-tags interpreter`

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2325 from dmick/ceph-dev-new-trigger-7-days

ceph-dev-new-trigger: use new feature, 7-day-old or newer branches

ceph-dev-new-trigger: use new feature, 7-day-old or newer branches

Actually modify the trigger job to use choosing-strategy: ancestry and
maximum-age: 7, so that only branches newer than 7 days old get built.

Also set build-name to the branch being triggered, for clarity.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2324 from dmick/jjb6

update all jobs to jenkins-job-builder v6; add 'choose from 7 days old' to ceph-dev-new-trigger

jenkins-job-builder,ceph-build-pull-requests: fetch and use jjb 6

currently it's actually a custom version of jjb6, to add
new choosing-strategy: ancestry, to be able to use
maximum-age in days to choose a branch to build.  This will
change to 6.5 once that branch is released.

Signed-off-by: Dan Mick <dan.mick@redhat.com>
sq

Update all jobs with ${VAR} syntax to ${{VAR}} for jjb6 compatibility

jenkins-job-builder 6 changed the interpretation of {var} to *always*
refer to a jjb macro; this is true even for ${VAR}, which we use in a
lot of place for Jenkins variable expansion (although not all; it's
unclear whether the {} are or ever were necessary, or in what exact
contexts).  So far, we've pinned jjb to <6, but we need new
functionality.

In any event, rather than changing all of them to the same {}-less style
that might work, change the ${} style to ${{}} and leave $ alone.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Update all jobs for jjb6: !include-raw -> !include-raw-verbatim

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2323 from ceph/pipeline-branch

ceph-dev-pipeline: Add CEPH_BUILD_BRANCH param

ceph-dev-pipeline: Add CEPH_BUILD_BRANCH param

So that we can easily test in-development changes to this pipeline.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2320 from rhcs-dashboard/node-install-for-windows-test

ceph-website: setup nodejs for building the website

ceph-website: setup nodejs for building the website

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2322 from dmick/wip-quiet-submodule-update

ceph-{dev-,}setup: Make git submodule operations much quieter.

ceph-{dev-,}setup: Make git submodule operations much quieter.

After f381e3f05fcb08a7b1a3a1aa8f86e066bd4718ba, which is quite the
win for ceph-dev-new-setup, credit to zack@redhat.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2321 from ceph/pipeline-branch

Correct ceph-dev-pipeline branch

Correct ceph-dev-pipeline branch

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2310 from ceph/jenkinsfile

Add ceph-dev-pipeline

Merge pull request #2316 from dmick/wip-remove-quincy

Remove quincy builds/mentions from most builds/build configs.

ceph-dev-pipeline: Add more info to build desc

BRANCH, DISTROS, ARCHS, SETUP_BUILD_ID with a link to the setup build.

Signed-off-by: Zack Cerza <zack@cerza.org>

Add ceph-dev-pipeline-trigger

This job serves to trigger ceph-dev-pipeline via webhook payloads, but is
opt-in. This implementation initially targets only GitHub push event payloads.
On each push, it will inspect the head commit for [git trailers][0]. The trailer
required to opt in is "CI-PIPELINE: true". If that trailer is found, the job
looks for these others:

ARCHS, CI_COMPILE, CI_CONTAINER, CI_PIPELINE, DISTROS, DWZ, FLAVOR, SCCACHE

Any that are found are passed as parameters.

[0] https://git-scm.com/docs/git-interpret-trailers

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: optionally skip package build

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Report build status to shaman

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: build container separately

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-*build: move container build to separate script

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2319 from ceph/wip-update-java-alternatives

Updating java openjdk to version 17 on rhel and debian builders, I ha…

Updating java openjdk to version 17 on rhel and debian builders, I have added the update-alternatives command to make sure we set the OS to use the new java version

Signed-off-by: Adam Kraitman <akraitma@li-8b09b2cc-35b7-11b2-a85c-cd1dbade58f9.ibm.com>

ceph-*build: Update status in separate script

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-setup: allow cd-pipeline to copy artifacts

Signed-off-by: Zack Cerza <zack@redhat.com>

Add ceph-dev-pipeline

This may ultimately replace the existing ceph-*build jobs; for now, it is a
work in-progress.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2318 from dmick/wip-ceph-build-quay-creds

ceph-build.yml: fix the name of the Jenkins cred

ceph-build.yml: fix the name of the Jenkins cred

(this is why it's a bad idea to switch between . and _)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2317 from dmick/wip-ceph-build-quay-creds

ceph-build: use correct quay.ceph.io credentials

ceph-build: use correct quay.ceph.io credentials

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2314 from ceph/prs-no-smithi

Do not run make check or API tests on smithi due to OOM

Remove quincy builds/mentions from most builds/build configs.

Intentionally leaving the map from number to name in build_utils.sh for
historical purposes.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Do not run make check or API tests on smithi due to OOM

Saw many jobs fail on smithi004.  dmesg shows lots of OOM kills.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2311 from ceph/fix-cdnb

ceph-dev-new-build: Fix parameter name

Merge pull request #2312 from dmick/wip-jenkins-builder

ansible/examples/builder.yml: "vars" should be a dict

it always should have been, but this syntax is now an error.  Make it
a nested dict (rather than a list of dicts).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-dev-new-build: Fix parameter name

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2295 from ceph/keep-more

Merge pull request #2309 from ljflores/wip-revert-pr-2306

Revert "ceph-website-prs/config/definitions: remove org filter for site compile check

ceph-website-prs/config/definitions: remove whitelist section

This section allows three people to bypass the org restriction.
However, zdover23 is already part of the org, and the other two
were originally contractors who are no longer involved with
the project.

Signed-off-by: Laura Flores <lflores@ibm.com>

Revert "ceph-website-prs/config/definitions: remove org filter for site compile check"

This reverts commit a227cbd01c628e9d98cf5cb3d96f0a299c4d12c3.
We decided that this is too much of a security risk and are looking
into better options.

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #2307 from dmick/wip-sepia-fog-images

Change default distroversions, improve task-detection code

sepia-fog-images: look for deploy tasks only of the image being captured

It doesn't matter if there are deploy tasks for other images, only for
the one we're about to capture

Signed-off-by: Dan Mick <dmick@redhat.com>

sepia-fog-images: change defaults for distrovers to capture

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2306 from ljflores/wip-ceph-website

ceph-website-prs/config/definitions: remove org filter for site compile check

ceph-website-prs/config/definitions: remove org filter for site compile check

These filters block anyone outside the ceph org from having the site compile
check run on their PRs. We do want people from outside the org to be able to
raise PRs, so removing these filters will allow that check to run.

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #2305 from dmick/wip-remove-ceph-container-jobs

remove all jobs referring to ceph-container.

Merge pull request #2304 from dmick/ceph-use-container

ceph,ceph-build: build prerelease containers (using container/)

remove all jobs referring to ceph-container.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph,ceph-build: build prerelease containers (using container/)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2273 from ceph/dvanders_allow_zac_to_render

ceph-website: zac is the docs maintainer

Merge pull request #2299 from dmick/wip-attic

move disabled jobs to 'attic/' subdirectory

Merge pull request #2302 from dmick/wip-fix-ceph-dev-build

ceph-dev-build: fix another error (stray '+' in build/build_rpm)

ceph-dev-build: fix another error (stray '+' in build/build_rpm)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2301 from dmick/wip-fix-ceph-dev-build

ceph-dev-build/build/setup_rpm: fix syntax error

ceph-dev-build/build/setup_rpm: fix syntax error

Reversed the =~ operator when porting change from ceph-dev-new-build

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2300 from dmick/wip-containerfile-ceph-dev-build

ceph-dev-build: change to use container/Containerfile

ceph-dev-build: change to use container/Containerfile

This is the same change as done by 6361c5c4bac7467893b1f629090712c8c0d736a8
and 435515fe481bd1752b630d23986249e90c0fa4f8 in ceph-dev-new-build, but
for ceph-dev-build.  ceph-container is no longer used.

Also remove CONTAINER_BRANCH parameter from ceph-dev

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2298 from dmick/wip-skip-testing-container-prs

build_utils.sh, various PR build scripts: skip if container/

Merge pull request #2292 from dmick/wip-builder-ansible

Clean up problematic rpm macro, install skopeo package for centos9

move ceph-medic* and diamond* to attic/

long-obsolete jobs

Signed-off-by: Dan Mick <dan.mick@redhat.com>

move disabled jobs to 'attic/' subdirectory

Reduce clutter, avoid mistaken interpretations; it's easy to miss
'disabled: true' when you're searching for relevant jobs

Signed-off-by: Dan Mick <dan.mick@redhat.com>

build_utils.sh, various PR build scripts: skip if container/

Parallel to the docs-only test, skip PR checks (api, make check, etc.)
if the only changes are to the container/ subdirectory

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2297 from dmick/get-tarballs-prerelease

scripts/get-tarballs.sh: pull to prerelease area

scripts/get-tarballs.sh: pull to prerelease area

I've got a bunch of doc work to do to the release process;
this is one of them.  Maybe we should also add this, and
its copy, to the ceph-cm-ansible role for d.c.c

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2296 from dmick/add-get-tarballs.sh

Keep track of extra script get-tarballs.sh

Keep track of extra script get-tarballs.sh

This is used by sync-pull to fetch the tarballs when
constructing a release, but needs modification for the
prerelease/ directory structure.  It's invoked by ssh
to download.ceph.com to run from /home/signer/bin, because
it needs to copy large files from chacra to download.ceph.com
and doing that on download.ceph.com is much more efficient.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-dev-new-build: Keep more builds

Builds of this job take up 30KB; we can keep far more.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-trigger: Keep more builds

Builds of this job take up 20MB.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-setup: Keep more builds

I've seen ceph builds fail because they took so long to claim a build node
that the artifacts created by their ceph-dev-new-setup build was
garbage-collected.

Artifacts for a given build of this job use less than 1G; we should keep more
artifacts around. Metadata for builds of this job are negligible; we should
keep far more sets of metadata around.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2294 from ceph/podman-perms

ceph-dev-new-setup: Correct podman storage group

ceph-dev-new-setup: Correct podman storage group

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2290 from rhcs-dashboard/sync-multi-aarch

sync-images: sync all images including their different aarch

examples/builder.yml: add skopeo

arguably we should fix 'epel_packages' because I think none of those
are in epel anymore, but in this file it really means 'packages
that are release-specific'

Signed-off-by: Dan Mick <dan.mick@redhat.com>

examples/builder.yml: don't add definition for dist past el7

This caused some really weird breakage where el8 was being set
on el9 machines, and it's no longer necessary (I don't know why it
was ever there)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2291 from ceph/cdn-distro-default

ceph-dev-new: Update default distro value

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2285 from ceph/xz

sync-images: sync all images including their different aarch

The sync is not syncing the different architectures supported by the
images atleast for grafana. And hence only the amd64 gets mirrored to
the quay repo and causes issues to arm64 builds. I saw that skopeo
supports an `--all` opton that copies every aarch.

Refer: https://github.com/containers/skopeo/blob/main/docs/skopeo-sync.1.md#options

Fixes: https://tracker.ceph.com/issues/68627
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2289 from ceph/sccache

ceph-dev-new-setup: Use sccache if requested

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-build: Set up sccache if requested

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new: Add sccache toggle

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-build: Use creds for sccache bucket

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2288 from ceph/dwz

ceph-dev-new: Add dwz toggle

Merge pull request #2287 from ceph/yaml-formatting

Merge pull request #2284 from ceph/unbottleneck

ceph-dev-new-setup: Run on any node

Merge pull request #2286 from ceph/quiet-submodule

ceph-dev-new-setup: Run on any node

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-setup: Disable dwz if requested

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new: Add dwz toggle

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new: Fix YAML errors in job definition

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-setup: Quiet submodule operations

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph-dev-new-setup: Support tar.xz

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2280 from dmick/wip-container

ceph-dev-new-*: change for in-tree container builds

ceph-dev-new-build/build/setup_rpm: add podman login

ceph.git's build.sh does not handle authentication in its latest
version, but assumes it's been done.  Add the authentication
step (podman login) here, in setup_rpm, with the side benefit
that it'll fail faster if the credentials are incorrect.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #2283 from ceph/cephadm-move-to-quay

sync-images: sync all cephadm required images to quay.io