From: David Galloway <david.galloway@ibm.com>
Date: Wed, 15 Jul 2026 20:18:08 +0000 (-0400)
Subject: ceph-dev-pipeline: pin podman auth to a persistent authfile
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=9a6ed865bbd2ed3260437800779d78d939a5df76;p=ceph-build.git

ceph-dev-pipeline: pin podman auth to a persistent authfile

Same fix as aae7f2fd applied to the builder container stage. The
Jenkins agent runs as a systemd service with no login session and no
XDG_RUNTIME_DIR, so podman derives the default auth.json location from
ambient node state at each invocation: /run/user/<uid> if it exists
(which depends on linger/session state), else fallback dirs under /tmp.
All of these are runtime tmpfs paths whose lifecycle is owned by
logind, systemd-tmpfiles, or podman's fallback heuristics rather than
by the job.

We observed a job where podman login succeeded and the base image pull
inside build-with-container.py went anonymous sixty seconds later on
the same node, hitting Docker Hub's unauthenticated per-IP rate limit
(toomanyrequests). The exact event that made the credentials
unavailable is still under investigation, but pinning the authfile to
persistent storage in $HOME removes the dependency on runtime tmpfs
state entirely.

Because each Jenkins sh step runs a fresh shell, REGISTRY_AUTH_FILE is
set via env at the Groovy level so every subsequent sh step in the
matrix cell resolves the same file, including the build stage's
build-with-container.py runs (which we cannot pass --authfile to) and
the container stage's build_container.

Refs: https://tracker.ceph.com/issues/77920

Signed-off-by: David Galloway <david.galloway@ibm.com>
---

diff --git a/ceph-dev-pipeline/build/Jenkinsfile b/ceph-dev-pipeline/build/Jenkinsfile
index 38f19844f..7744b89d6 100644
--- a/ceph-dev-pipeline/build/Jenkinsfile
+++ b/ceph-dev-pipeline/build/Jenkinsfile
@@ -346,10 +346,22 @@ def doArtifactsChecksStage() {
 // Stage 6 (builder container): log into container registries and pull/build/push the ceph-build container image for this matrix cell.
 def doBuilderContainerStage() {
   env.CEPH_BUILDER_IMAGE = "${env.CONTAINER_REPO_HOSTNAME}/${env.CONTAINER_REPO_ORGANIZATION}/ceph-build"
+  // Pin podman auth to a persistent authfile. The agent runs as a systemd
+  // service with no login session and no XDG_RUNTIME_DIR, so podman derives
+  // the default auth.json location from ambient node state at each
+  // invocation, landing in runtime tmpfs the job doesn't control.
+  // Credentials written by podman login here have been observed unavailable
+  // to podman build in the same job minutes later, so the base image pull
+  // goes anonymous and hits Docker Hub's rate limit (toomanyrequests,
+  // https://tracker.ceph.com/issues/77920). Set via env so every subsequent
+  // sh step (pull/build/push here, and build-with-container.py in the build
+  // stage) resolves the same file.
+  env.REGISTRY_AUTH_FILE = "${env.HOME}/.config/containers/auth.json"
   sh '''#!/bin/bash
     set -ex
-    podman login -u ${CONTAINER_REPO_CREDS_USR} -p ${CONTAINER_REPO_CREDS_PSW} ${CONTAINER_REPO_HOSTNAME}/${CONTAINER_REPO_ORGANIZATION}
-    podman login -u ${DOCKER_HUB_CREDS_USR} -p ${DOCKER_HUB_CREDS_PSW} docker.io
+    mkdir -p "${REGISTRY_AUTH_FILE%/*}"
+    podman login --authfile ${REGISTRY_AUTH_FILE} -u ${CONTAINER_REPO_CREDS_USR} -p ${CONTAINER_REPO_CREDS_PSW} ${CONTAINER_REPO_HOSTNAME}/${CONTAINER_REPO_ORGANIZATION}
+    podman login --authfile ${REGISTRY_AUTH_FILE} -u ${DOCKER_HUB_CREDS_USR} -p ${DOCKER_HUB_CREDS_PSW} docker.io
   '''
   def ceph_builder_tag_short = "${env.BRANCH}.${env.DIST}.${env.ARCH}.${env.FLAVOR}"
   def ceph_builder_tag = "${env.SHA1[0..6]}.${ceph_builder_tag_short}"