David Galloway [Wed, 15 Jul 2026 20:18:08 +0000 (16:18 -0400)]
ceph-dev-pipeline: pin podman auth to a persistent authfile
Same fix as aae7f2fd applied to the builder container stage. The
Jenkins agent runs as a systemd service with no login session and no
XDG_RUNTIME_DIR, so podman derives the default auth.json location from
ambient node state at each invocation: /run/user/<uid> if it exists
(which depends on linger/session state), else fallback dirs under /tmp.
All of these are runtime tmpfs paths whose lifecycle is owned by
logind, systemd-tmpfiles, or podman's fallback heuristics rather than
by the job.
We observed a job where podman login succeeded and the base image pull
inside build-with-container.py went anonymous sixty seconds later on
the same node, hitting Docker Hub's unauthenticated per-IP rate limit
(toomanyrequests). The exact event that made the credentials
unavailable is still under investigation, but pinning the authfile to
persistent storage in $HOME removes the dependency on runtime tmpfs
state entirely.
Because each Jenkins sh step runs a fresh shell, REGISTRY_AUTH_FILE is
set via env at the Groovy level so every subsequent sh step in the
matrix cell resolves the same file, including the build stage's
build-with-container.py runs (which we cannot pass --authfile to) and
the container stage's build_container.
Refs: https://tracker.ceph.com/issues/77920
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 22:51:10 +0000 (18:51 -0400)]
jenkins-job-builder: skip the CVE jobs
Skip cve-pipeline and cve-source-dist in the update loop and exclude
them from the deletion pass so they aren't removed for lacking test
XML. Changes to those jobs must be made manually by a Jenkins admin.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 22:39:39 +0000 (18:39 -0400)]
ceph-dev-pipeline: verify the chacra repo URL answers before trusting shaman
Follow-up to the shaman-based chacra existence check: a ready repo
record can outlive the repo it points at (chacra nodes get rotated or
rebuilt, and shaman has been observed reporting "no healthy chacra
nodes" while ready records remained). Skipping compilation on the
strength of a stale record would leave consumers with a dead repo URL,
so extract the record's chacra_url and require it to answer 2xx; any
failure falls back to rebuilding.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 21:13:46 +0000 (17:13 -0400)]
builder-reimage: retain network configuration when commissioning
Pass skip_networking=True to machine.commission(), the API equivalent
of checking "Retain network configuration" in the MAAS UI, so a
machine commissioned by the reimage job keeps its existing interface
configuration instead of having it replaced by whatever commissioning
discovers.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 20:58:01 +0000 (16:58 -0400)]
builder-reimage: add CEPH_BUILD_BRANCH parameter
Add the same CEPH_BUILD_BRANCH functionality ceph-dev-pipeline has: a
string parameter (default main) wired into the pipeline-scm branch
spec, so the Jenkinsfile and 'checkout scm' contents can be taken from
a ceph-build branch under test.
Unlike ceph-dev-pipeline, this job also clones ceph-build at runtime
via prepare_env.sh (ansible_runner.sh runs playbooks from
repos/main/ansible), so pass the branch through as a new optional
seventh argument and clone that branch. Also fix the usage comment,
which had the main/ansible repo argument order swapped.
Drop the top-level scm block from the JJB definition: jenkins-jobs
discards it for pipeline project-type jobs (verified the generated XML
is identical with and without it), so it was dead config.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 20:54:19 +0000 (16:54 -0400)]
builder-reimage: fix SSH key not being used for git clones
The Prepare stage copied the ansible-ssh-key credential into the
workspace but never exported GIT_SSH_COMMAND, and prepare_env.sh only
looked for a key at /tmp/jenkins_git_key, which nothing creates. The
ceph-cm-ansible clone therefore ran without a key and failed with
'Permission denied (publickey)'.
Export GIT_SSH_COMMAND in the Prepare stage (as the Ansible stage
already does) and drop the dead /tmp key path from prepare_env.sh so
clones and fetches both inherit the caller's GIT_SSH_COMMAND. Also pass
$SSH_KEY to the shell instead of interpolating it in Groovy, clearing
the insecure-interpolation warning.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 18:17:36 +0000 (14:17 -0400)]
setup_chacractl: fail when shaman has no healthy chacra nodes
shaman's /api/nodes/next/ answers 404 with an HTML error page when it
has no healthy chacra node to allocate; without -f, curl captured that
page and setup_chacractl.sh wrote it into ~/.chacractl as the url.
Fail the curl instead.
Also run setup_chacractl.sh from the chacra upload stage rather than
the check stage: the existence check asks shaman now, so chacra being
unhealthy only fails cells that actually need to upload to chacra
(the upload stage's environment{} provides the credentials).
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 16:54:56 +0000 (12:54 -0400)]
build_container: fix the shaman repo readiness wait
Two problems in the wait loop:
- It only inspected .[0] of the shaman search result, but shaman holds
several repo records for the same sha1 (one per backend and per
uploading job). Observed: .[0] was the chacra record posted by the
production ceph-dev-pipeline building the same main sha1, whose
extra.build_url could never match this build's, so the loop always
timed out even though this build's own (pulp) record was ready at
.[1]. Scan all records instead.
- The skipped-compilation escape hatch keyed off the CI_COMPILE
parameter, which stays "true" when the artifact-existence check is
what skipped compilation. The pipeline now exports UPLOADED (this
cell compiled, THROWAWAY off, at least one backend enabled); when
false, any ready record for this sha1 satisfies the wait.
Also make the timeout fatal: it used to print FAIL and build the
container anyway, which either fails later in the Containerfile or
silently bakes stale packages. Now that the readiness check is
accurate, a timeout is a real error. Note THROWAWAY=true with
CI_CONTAINER=true now fails cleanly when the sha1 was never published
anywhere, instead of building an image from whatever shaman had.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 16:48:15 +0000 (12:48 -0400)]
ceph-dev-pipeline: only skip compilation when every enabled backend has the build
shouldSkipCompilation() treated the existence checks as an OR: if
either Chacra or Pulp already had the build, compilation was skipped
entirely -- which also skipped the upload to the backend that was
missing it. Observed with PULP_UPLOAD and CHACRA_UPLOAD both enabled:
pulp had the distribution, the chacra check came back empty, and the
cell skipped straight past both uploads, so chacra never received the
build.
Skip compilation only when every enabled backend already has the build
(disabled backends don't count), and when compilation does run because
one backend was missing, skip the upload stage of the backend that
already had it (unless FORCE) so packages aren't blindly re-pushed.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 16:34:01 +0000 (12:34 -0400)]
ceph-dev-pipeline: check for existing chacra builds via shaman
setup_chacractl.sh points chacractl at whichever chacra node shaman's
/api/nodes/next/ hands out, which is a round-robin upload allocator.
`chacractl exists` against that node only finds a previous build's
binaries when the rotation happens to land on the same node the build
uploaded to: observed binaries on 2.chacra.ceph.com while the check
404'd against 1.chacra.ceph.com. This predates the stage split but now
gates compilation.
Ask shaman instead, which records which node holds each build: a
"ready" repo record for this project/ref/sha1/distro/arch/flavor with
a chacra.ceph.com chacra_url. Pulp repo records live in the same
shaman collection, so filter on chacra_url. setup_chacractl.sh still
runs here to configure ~/.chacractl for the upload stage.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Fri, 10 Jul 2026 14:36:03 +0000 (10:36 -0400)]
ceph-dev-pipeline: report the flavor-level pulp repo URL to shaman
The repo record's url pointed at the per-arch distribution
(.../flavors/default/x86_64/), but noarch/, x86_64/ and SRPMS/ are
sibling distributions under the flavor path, not subdirectories of each
other. Consumers follow chacra's convention of a flavor-level URL:
ceph.git's Containerfile appends noarch/Packages/... to the shaman url
to fetch the ceph-release RPM, which 404'd against the arch-level URL
and failed the CI container build.
The per-arch repository API href in chacra_url (from pulp_repo_info) is
unchanged, so the x86_64 and arm64 cells still post distinct records.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 9 Jul 2026 19:10:08 +0000 (15:10 -0400)]
remove the dead TEST='test' chacra ref logic
The TEST parameter was meant to push release packages to chacra under
the 'test' ref, but the logic has been dead for a while: neither
ceph-dev-pipeline.yml defines a TEST parameter nor does
ceph-release-pipeline forward one when triggering the build. It was
also broken if ever enabled (env.SHA1[0..6] on the 4-character 'test'
string throws). Remove the ref override from doUploadChacraStage and
the unused TEST parameter from ceph-release-pipeline.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 9 Jul 2026 19:09:31 +0000 (15:09 -0400)]
pulp_upload.sh: include the flavor in repository and distribution names
Repository names were branch/os/version-scoped and distribution names
derived from them, so default and debug builds of the same branch/sha1
shared repositories and produced identically named distributions: the
stale-distribution delete would clobber one flavor's distribution with
the other's, and both flavors' packages mixed in one repository even
though their base_paths are flavor-specific.
Add FLAVOR to REPO_NAME, which flows into the per-arch repository names
and the dist-<repo>-<short_sha1> distribution names, and update the
existence check in the ceph-dev-pipeline Jenkinsfile to match.
Distributions created under the old naming may still own a base_path
(base_paths are unique in pulp), so also delete any differently named
distribution occupying the base_path before creating the new one.
Existing repositories with unflavored names are left in place and can
be cleaned up manually.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 9 Jul 2026 16:28:45 +0000 (12:28 -0400)]
ceph-release-pipeline: pass boolean parameters as booleans
The ceph-dev-pipeline and second ceph-tag invocations passed env values
(strings) straight to booleanParam, unlike the first ceph-tag
invocation which converts with .toBoolean(). Downstream now compares
params.THROWAWAY == false strictly, so make sure the values arrive as
actual booleans.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 9 Jul 2026 16:28:25 +0000 (12:28 -0400)]
pulp_upload.sh: use the rpm arch for the repository handoff lookup
The final `pulp repository show --name ${REPO_NAME}-${ARCH}` uses the
Jenkins matrix arch (arm64), but rpm repositories are created per rpm
arch (aarch64) by the upload loop. For arm64 rpm builds the lookup
fails after all uploads succeeded and, under set -e, kills the script,
failing the stage.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 9 Jul 2026 16:28:09 +0000 (12:28 -0400)]
cve-pipeline: add FORCE parameter
cve-pipeline runs the ceph-dev-pipeline Jenkinsfile, which consults
FORCE to decide whether to rebuild and re-upload when artifacts already
exist in Chacra or Pulp. Without the parameter, env.FORCE is unset and
a CVE rebuild of an already-uploaded sha1 always skips compilation with
no way to override short of THROWAWAY=true.
Also sync the THROWAWAY description with ceph-dev-pipeline's.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 9 Jul 2026 16:26:26 +0000 (12:26 -0400)]
ceph-dev-pipeline: key build_matrix by flavor and fix FLAVOR axis
build_matrix was keyed by "${DIST}_${ARCH}" but the matrix has a
FLAVOR axis, so the default and debug cells of the same dist/arch
shared one key: when the artifact check found existing default
artifacts and set the key false, it cancelled the debug cell's build
too (and vice versa), even though the chacra endpoints and pulp
distributions are flavor-specific. Include FLAVOR in the key
everywhere build_matrix is read or written.
Also declare the FLAVOR axis with `axis` instead of a nested `axes`
block, which is not valid declarative matrix syntax.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 9 Jul 2026 14:43:12 +0000 (10:43 -0400)]
ceph-release-pipeline: use @Field for top-level variables
SCRIPT_SPLITTING_TRANSFORMATION (JENKINS-37984) is enabled on
jenkins.ceph.com to keep ceph-dev-pipeline under the JVM 64KB method
limit. That flag rejects Declarative pipelines with script-level local
variable declarations, so ceph-release-pipeline failed to compile:
General error during semantic analysis: [JENKINS-37984]
SCRIPT_SPLITTING_TRANSFORMATION is ... incompatible with local
variable declarations inside a Jenkinsfile.
Annotate the cross-stage variables with @Field, same as was done for
ceph-dev-pipeline-trigger in c898b349. Audited all other Declarative
Jenkinsfiles in the repo; no other job has script-level variable
declarations.
Signed-off-by: David Galloway <david.galloway@ibm.com>
Shraddha Agrawal [Mon, 22 Jun 2026 06:55:49 +0000 (12:25 +0530)]
introduce noble debug builds
This commit introduces debug builds for noble which can be used for
integration testing. Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
ceph-perf: bound CBT hangs and make comment triggers reliable
radosbench prefill has wedged bath01 for days with no job timeout.
Cap each run-cbt.sh at 1h and the job at 8h, kill leftover Ceph
processes around CBT, and require an explicit "jenkins test
{osd-flavor} perf" comment so label-only auto-builds do not queue
forever on a stuck performance node.
David Galloway [Fri, 12 Jun 2026 18:20:23 +0000 (14:20 -0400)]
ceph-dev-pipeline: Notify shaman of pulp repo URLs
Mirror chacra's shaman repo records: 'url' points at the browsable
package repo (the Pulp content distribution for this arch) and
'chacra_url' points at the repository's API endpoint
(/pulp/api/v3/repositories/...).
pulp_upload.sh hands off the repository's API href and the package
manager version to the notify script via $WORKSPACE/pulp_repo_info.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Thu, 11 Jun 2026 20:27:10 +0000 (16:27 -0400)]
cve-pipeline: Creation
This enables us to build packages and containers from ceph-private.git privately.
This includes Jenkins build logs, artifacts, packages, and containers.
Packages will be pushed to the internal-only pulp instance and containers
to the internal-only quay instance.
To avoid job duplication, each job pulls in the respective Jenkinsfiles e.g.,
- cve-pipeline -> ceph-dev-pipeline
- cve-source-dist -> ceph-source-dist
The only difference is the parameters we pass to the job. The key differences
there are CEPH_REPO points to ceph-private.git whereas ceph-dev-pipeline is
not capable of doing so. We also tell the pipeline not to use a public
chacra instance.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Tue, 7 Jul 2026 13:41:27 +0000 (09:41 -0400)]
ceph-dev-pipeline-trigger: use @Field for top-level variable
The Declarative script splitting transformation being enabled on the
controller rejects pipelines that declare top-level local variables.
Convert the params list to a @Field, renamed to build_params to avoid
clobbering Jenkins pipeline's 'params' predefined global variable.
Signed-off-by: David Galloway <david.galloway@ibm.com>
David Galloway [Tue, 7 Jul 2026 13:41:17 +0000 (09:41 -0400)]
jenkins: enable Declarative script splitting to avoid 64KB method limit
The ceph-dev-pipeline matrix fails to compile with 'Method too large:
WorkflowScript.___cps___NNN' when new cells are added (e.g. noble debug
builds, #2622), because the Declarative RuntimeASTTransformer emits the
entire pipeline model as a single method. Extracting stage bodies into
helper functions (#2621) does not help since the per-cell model
boilerplate dominates.
Enable the script splitting transformation (JENKINS-37984), which
splits the generated model into multiple methods so no single one hits
the JVM's 64KB bytecode limit.
Signed-off-by: David Galloway <david.galloway@ibm.com>
Vaibhav Mahajan [Thu, 18 Jun 2026 10:20:42 +0000 (15:50 +0530)]
enhancement: create Pulp repositories with labels before upload
Add repository creation to the Pulp upload flow so CI jobs ensure the
target repo exists and carries labels discovery metadata before packages
are uploaded.
- Add `create_pulp_repository()` to idempotently create `rpm`/`deb`
repositories, apply metadata labels as JSON and honor `retain-repo-versions`.
- Extract `get_pulp_distribution_labels()` and reuse it for both
repository and distribution label application.
- Refactor `publish_pulp_distribution()` to use the shared label helper.
- Require `BRANCH` from the environment instead of deriving it from
`CEPH_VERSION` via `release_from_version()`
David Galloway [Thu, 2 Jul 2026 22:21:21 +0000 (18:21 -0400)]
bwc: pin podman auth to a persistent authfile
podman login was writing credentials to the default location under
XDG_RUNTIME_DIR (/run/user/<uid>), which is logind-managed tmpfs. The
Jenkins agent runs without XDG_RUNTIME_DIR set, so the runtime dir can
be torn down or re-resolved between bwc_login and the podman build it
is meant to authenticate. When that happens, the base image pull inside
build-with-container.py silently falls back to anonymous and hits
Docker Hub's per-IP unauthenticated rate limit (toomanyrequests), e.g.:
Set REGISTRY_AUTH_FILE to a persistent path in $HOME and pass
--authfile to podman login so login and all subsequent podman/buildah
invocations in the job resolve the same credential file regardless of
session state.
Fixes: https://tracker.ceph.com/issues/77920 Signed-off-by: David Galloway <david.galloway@ibm.com>
Shraddha Agrawal [Mon, 22 Jun 2026 06:55:49 +0000 (12:25 +0530)]
introduce noble debug builds
This commit introduces debug builds for noble which can be used for
integration testing. Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
Shraddha Agrawal [Mon, 22 Jun 2026 06:30:39 +0000 (12:00 +0530)]
ceph-dev-pipeline: extract matrix stage script bodies into helper funcs
Problem: We were hitting JVM's 64KB bytecode limit when adding new
distros to the matrix stage script.
Solution: Add helper functions for each stage in the matrix stage
script, such that the original block is significantly reduced, thus
ensuring we do not hit the JVM's 64KB limit again. This will also ensure
we can keep extending the matrix with more distros without worrying
about this issue. Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
ceph-perf: install mdutils and run CBT under gh-venv
cbt imports mdutils but ceph/cbt master requirements.txt does not
list it. Install mdutils into gh-venv, put the venv on PATH for
run-cbt.sh, and invoke compare.py via python3 so its /usr/bin/python3
shebang is not used. Only skip CBT runs when the archive dir already
has results, and fail the step if the archive is empty after run-cbt.
ceph-perf: default unset CEPH_PERF_EXTRA_CMAKE_ARGS for set -u
run-cbt uses set -euxo pipefail, but the optional Jenkins parameter is not
always exported into the shell environment. Use ${CEPH_PERF_EXTRA_CMAKE_ARGS:-}
so PR-triggered builds do not fail before cmake.
Jenkins invokes shell steps as /bin/sh -xe when there is no shebang.
On Noble that is dash, which rejects set -o pipefail. Add #!/bin/bash
so Jenkins runs the run-cbt builder with bash.
github-check uses github3.py 4.0.1, which passes app_id as int to
jwt.encode(). PyJWT 2.11+ requires iss to be a string. Install
githubcheck first, then pin pyjwt<2.11.
Kefu Chai [Thu, 25 Jun 2026 01:41:15 +0000 (09:41 +0800)]
ceph-windows: actually disable the mgr for the vstart cluster
-WITH_MGR=OFF and -WITH_LTTNG=OFF were passed to do_cmake.sh with a single
dash, so cmake never set them. The mgr has therefore been built and started
by every vstart run since this script was added in 2022, only to fail loading
rbd_support and telemetry because their python deps were never installed.
The librbd, rados and cephfs tests run against this cluster use the direct
library API and do not need the mgr, so disable it as originally intended.
Pass -DWITH_MGR=OFF (and -DWITH_LTTNG=OFF), and set CEPH_NUM_MGR=0 for
vstart.sh so it does not try to start a ceph-mgr that is no longer built.
python3-prettytable is kept: it is a client CLI dependency (ceph_daemon.py),
not a mgr one.
ceph-perf: fix github-check and use run-make.sh for clang-19
github-check fails on fresh venvs with PyJWT 2.13+ because github3
passes app_id as int to jwt.encode(). Pin pyjwt<2.13 before installing
githubcheck, and quote app-id/install-id in github-check calls.
Stop probing for clang down to 18 in the perf job; Crimson needs
clang-19. Let run-make.sh install it via llvm.sh (get_llvm) and select
it with discover_compiler, same as make-check nodes.
The added matrix cell pushed the CPS-generated WorkflowScript method past
the JVM's 64KB bytecode limit, failing every ceph-dev-pipeline build with
MethodTooLargeException. Revert to unblock CI; noble debug builds will be
re-landed once the matrix stage bodies are extracted into helper methods.
Shraddha Agrawal [Tue, 16 Jun 2026 08:01:54 +0000 (13:31 +0530)]
introduce debug builds for noble
This commit adds debug builds for the distro noble. With this change, we
will also get debug noble builds. This is especially useful for testing
crimson in QA suites. Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
projects / ceph-build.git / log