scripts/build_utils: remove Debug build from Crimson

See: https://github.com/ceph/ceph-build/pull/2167/files
Builds should be set to Debug only when ending in "-debug".

https://github.com/ceph/ceph-build/pull/1805 set Crimson to always use Debug.
That made sense in earlier development phases.
However, today we should align with Classical builds and use Debug only when needed.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2365 from ceph/builder-ansible

FIx up builder.yml

examples/builder.yml: update comments with new usage info

Signed-off-by: Dan Mick <dan.mick@redhat.com>

slave.yml: remove old symlink

Signed-off-by: Dan Mick <dan.mick@redhat.com>

examples/builder.yml: set api token based on api_uri

This is really the ceph-jenkins ssh user's password, but they're
now stored in ceph-sepia-secrets so they don't have to be found
and entered on the cmdline here

This keeps 'one ansible-vault encrypt output' in one file, and
should be easier to edit/view/etc.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

examples/builder.yml: fix up subuid/subgid handling code

1) not necessary to make 'find highest uid' conditional (and the
   check was checking 0 instead of !0 anyway)
2) fix up choice of "calculated or default" using "| default()"

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2363 from ceph/pipeline-container

ceph-dev-pipeline: Build in containers

ceph-dev-pipeline: Fix extract/upload of cephadm

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: chown without sudo

And also at the very end of the job.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Store sccache logs

There is an issue where invocations of archiveArtifacts in parallel branches of
the pipeline overwrite each other.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build the builder less often

To build the builder image less often, we can:

1. When building, use two tags: branch.os.arch, and sha1.branch.os.arch
2. After building, push both tags
3. Before building, pull both tags

As a result, we can avoid building entirely if that exact image has already been
pushed, and avoid building at least some layers if the branch had been built
somewhat recently.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Split upload stage from build

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Re-narrow node label

smithis take far too long.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Add and use update_shaman.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build release RPM

Signed-off-by: Zack Cerza <zack@cerza.org>

setup_container_runtime.sh: Prune more images

Only leave one weeks' worth.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Fix CI_COMPILE

Setting an environment variable affects all matrix branches; this approach keeps
them isolated.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Copy even less artifacts

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use ceph-source-dist by default

But continue to offer ceph-dev-new-setup as an option.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2357 from ceph/ceph-source-dist

Add ceph-source-dist job

Add ceph-source-dist job

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2360 from cbodley/wip-doc-PendingReleaseNotes

scripts: docs_pr_only() considers PendingReleaseNotes

scripts: docs_pr_only() considers PendingReleaseNotes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-dev-pipeline: Fix syntax error

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2358 from ceph/fixapt

setup_container_runtime.sh: Add missing sudo

setup_container_runtime.sh: Add missing sudo

... to the apt update invocation.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2356 from ceph/setup-containers

Add scripts/setup_container_runtime.sh

ceph-dev-new-setup: Use setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Add scripts/setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2355 from ceph/cdp-containerize

ceph-dev-pipeline: Build in containers

ceph-dev-pipeline: Use uv to set up chacractl

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Relax node labels

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline-trigger: CEPH_BUILD_BRANCH

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Add CEPH_REPO param

So that we can build release branches from ceph.git

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-new-setup: Only use podman if >= v4

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use centos9 hosts

We want to do containerized builds with recent podman versions.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Avoid docker.io rate limits

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Copy less artifacts

This is purely to save time.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Extract less from source dist

Only what we need to build the builder container.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build in containers

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2343 from ceph/new-pip

build_utils: Use pip 24.0 by default

build_container: Look for pipeline's source dir

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use ceph-build branch param

I missed this when filing #2323

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Fix indentation

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2348 from Matan-B/wip-matanb-cleanup-crimson-builds

ceph-dev*: Remove Crimson unsupported builds

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

ceph-dev*: Remove Crimson un-supported builds

Crimson should be built only for main and T.
Remove R and S builds.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2346 from Matan-B/wip-matanb-clean-with-seastar

ceph-dev*: final transition to WITH_CRIMSON

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #2345 from dmick/ceph-release-containers-defaults

ceph-release-containers: tweak the parameter defaults

ceph-dev*: final transition to WITH_CRIMSON

Follow up to: https://github.com/ceph/ceph-build/pull/2329
Now that https://github.com/ceph/ceph/pull/62325 is merged.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

build_utils: Use pip 24.0 by default

distutils is removed from python 3.12 so pip 20.3.4 is incompatible with systems running python3.12.

Ubuntu Noble is our new target Jenkins builder OS so let's bump to 24.0 (the latest as of this PR)

https://stackoverflow.com/questions/69919970/no-module-named-distutils-util-but-distutils-is-installed

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-release-containers: tweak the parameter defaults

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2344 from dmick/fix-include-raw

PR 2315 added back in an 'include-raw' which is deprecated

PR 2315 added 'include-raw' which is deprecated

should use include-raw-verbatim.  Also, {} must be doubled.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2342 from ceph/add-ceph-release-containers

Add ceph-release-containers job

Merge pull request #2315 from batrick/tentacle-kickoff

Tentacle kickoff!

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Add ceph-release-containers job

Builds prerelease versions of ceph container images: two arch-specific
images and one 'manifest-list' image.  Pushed all to quay.ceph.io/ceph,
in a private set of repos (prerelease-amd64, prerelease-arm64, and
prerelease) for testing and signoff.  Once moved, the script
container/make-manifest-list.py can be used with the --promote flag to
promote them from quay.ceph.io to quay.io for public release.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2341 from dmick/remove-container-build-from-ceph-build

ceph-build: remove container build

*: update relevant scripts and triggers for tentacle

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

scripts: add tentacle

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Merge pull request #2340 from ceph/sepia-fog-fixes

sepia-fog-images: Mark down systems we borrow instead of lock

ceph-build: remove container build

The container must be built from signed packages, and signing
is not part of the Jenkins build.  So we'll create two new jobs
to build the prerelease containers (so that builders don't have
to have access to both amd64 and arm64 build hosts).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

sepia-fog-images: Mark down systems we borrow instead of lock

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2333 from ceph/no-setup-arm64

ceph*setup: Do not run on arm64 builders

Merge pull request #2329 from Matan-B/wip-matanb-add-with-crimson

ceph-dev*: intermediate transition to WITH_CRIMSON

ceph*setup: Do not run on arm64 builders

We're already low on ARM builders.  This is a waste of resources.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2332 from ceph/tf-fixes

ansible: Fixes to podman subuid/subgid checks

ansible: Fixes to podman subuid/subgid checks

I think this was working during my testing because sub{u,g}id_check.rc /wasn't/ 0.  I had manually modified the system so it wasn't in the same state as a fresh cloud instance.

I manually ran this playbook on an instance and confirmed it gets past the podman setup now.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2331 from ceph/2284-followup

Install and configure podman on Ubuntu builders

ansible: Install and configure podman on Ubuntu builders

Signed-off-by: David Galloway <david.galloway@ibm.com>

ansible: Remove ceph-docs deps from builder playbook

We're building docs in readthedocs.org infra now.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2328 from phlogistonjohn/jjm-bwc-container_only_pr

scripts: add build-with-container.py and friends to container_only_pr

ceph-dev*: intermediate transition to WITH_CRIMSON

See: https://github.com/ceph/ceph/pull/61672
This PR changes any WITH_SEASTAR / --with-seastar to WITH_CRIMSON /
--with crimson respectively. We should let our builders add the new flags
and once the PR is merged we could clean up the unused WITH_SEASTAR.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

scripts: add build-with-container.py and friends to container_only_pr

Add the build-with-container.py script and the helper shell script and
Dockerfile that support the builder image to the container_only_pr
patterns in build_utils.sh.  This should save time and resources when
contributing patches to only these files in the ceph/ceph repo.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #2326 from dmick/pr-only-for

Rework changed-files test (doc-only, container-only)

Merge pull request #2327 from ceph/wip-skip-interp

ansible: Add tag to python interpreter var setting tasks

ansible: Add tag to python interpreter var setting tasks

I'm working on a combination of terraform and ansible to quickly
spin up instances in IBM Cloud so they can join Jenkins as builders.

Part of this process involves creating a virtualenv and running ansible
from it.  python-jenkins needs to be installed in the virtualenv and
setting the python interpreters inside the playbook breaks that.

This PR will let me `--skip-tags interpreter`

Signed-off-by: David Galloway <david.galloway@ibm.com>

Rework changed-files test (doc-only, container-only)

Take an array of patterns to compare each changed file against.
For docs, use the current suggestions from
https://tracker.ceph.com/issues/69650

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2325 from dmick/ceph-dev-new-trigger-7-days

ceph-dev-new-trigger: use new feature, 7-day-old or newer branches

ceph-dev-new-trigger: use new feature, 7-day-old or newer branches

Actually modify the trigger job to use choosing-strategy: ancestry and
maximum-age: 7, so that only branches newer than 7 days old get built.

Also set build-name to the branch being triggered, for clarity.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2324 from dmick/jjb6

update all jobs to jenkins-job-builder v6; add 'choose from 7 days old' to ceph-dev-new-trigger

jenkins-job-builder,ceph-build-pull-requests: fetch and use jjb 6

currently it's actually a custom version of jjb6, to add
new choosing-strategy: ancestry, to be able to use
maximum-age in days to choose a branch to build.  This will
change to 6.5 once that branch is released.

Signed-off-by: Dan Mick <dan.mick@redhat.com>
sq

Update all jobs with ${VAR} syntax to ${{VAR}} for jjb6 compatibility

jenkins-job-builder 6 changed the interpretation of {var} to *always*
refer to a jjb macro; this is true even for ${VAR}, which we use in a
lot of place for Jenkins variable expansion (although not all; it's
unclear whether the {} are or ever were necessary, or in what exact
contexts).  So far, we've pinned jjb to <6, but we need new
functionality.

In any event, rather than changing all of them to the same {}-less style
that might work, change the ${} style to ${{}} and leave $ alone.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Update all jobs for jjb6: !include-raw -> !include-raw-verbatim

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2323 from ceph/pipeline-branch

ceph-dev-pipeline: Add CEPH_BUILD_BRANCH param

ceph-dev-pipeline: Add CEPH_BUILD_BRANCH param

So that we can easily test in-development changes to this pipeline.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2320 from rhcs-dashboard/node-install-for-windows-test

ceph-website: setup nodejs for building the website

ceph-website: setup nodejs for building the website

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2322 from dmick/wip-quiet-submodule-update

ceph-{dev-,}setup: Make git submodule operations much quieter.

ceph-{dev-,}setup: Make git submodule operations much quieter.

After f381e3f05fcb08a7b1a3a1aa8f86e066bd4718ba, which is quite the
win for ceph-dev-new-setup, credit to zack@redhat.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2321 from ceph/pipeline-branch

Correct ceph-dev-pipeline branch

Correct ceph-dev-pipeline branch

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2310 from ceph/jenkinsfile

Add ceph-dev-pipeline

Merge pull request #2316 from dmick/wip-remove-quincy

Remove quincy builds/mentions from most builds/build configs.

ceph-dev-pipeline: Add more info to build desc

BRANCH, DISTROS, ARCHS, SETUP_BUILD_ID with a link to the setup build.

Signed-off-by: Zack Cerza <zack@cerza.org>

Add ceph-dev-pipeline-trigger

This job serves to trigger ceph-dev-pipeline via webhook payloads, but is
opt-in. This implementation initially targets only GitHub push event payloads.
On each push, it will inspect the head commit for [git trailers][0]. The trailer
required to opt in is "CI-PIPELINE: true". If that trailer is found, the job
looks for these others:

ARCHS, CI_COMPILE, CI_CONTAINER, CI_PIPELINE, DISTROS, DWZ, FLAVOR, SCCACHE

Any that are found are passed as parameters.

[0] https://git-scm.com/docs/git-interpret-trailers

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: optionally skip package build

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Report build status to shaman

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: build container separately

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-*build: move container build to separate script

Signed-off-by: Zack Cerza <zack@cerza.org>