build_container: fix the shaman repo readiness wait

Two problems in the wait loop:

- It only inspected .[0] of the shaman search result, but shaman holds
  several repo records for the same sha1 (one per backend and per
  uploading job).  Observed: .[0] was the chacra record posted by the
  production ceph-dev-pipeline building the same main sha1, whose
  extra.build_url could never match this build's, so the loop always
  timed out even though this build's own (pulp) record was ready at
  .[1].  Scan all records instead.

- The skipped-compilation escape hatch keyed off the CI_COMPILE
  parameter, which stays "true" when the artifact-existence check is
  what skipped compilation.  The pipeline now exports UPLOADED (this
  cell compiled, THROWAWAY off, at least one backend enabled); when
  false, any ready record for this sha1 satisfies the wait.

Also make the timeout fatal: it used to print FAIL and build the
container anyway, which either fails later in the Containerfile or
silently bakes stale packages.  Now that the readiness check is
accurate, a timeout is a real error.  Note THROWAWAY=true with
CI_CONTAINER=true now fails cleanly when the sha1 was never published
anywhere, instead of building an image from whatever shaman had.

Signed-off-by: David Galloway <david.galloway@ibm.com>

diff --git a/ceph-dev-pipeline/build/Jenkinsfile b/ceph-dev-pipeline/build/Jenkinsfile
index 0a804215f9eb5e82988f3477cf46258c63735105..b5a537aa81fd01afacab32601f6f09f154aeb1db 100644 (file)
--- a/ceph-dev-pipeline/build/Jenkinsfile
+++ b/ceph-dev-pipeline/build/Jenkinsfile
@@ -587,10 +587,17 @@ def doContainerStage() {
   env.CONTAINER_REPO_PASSWORD = env.CONTAINER_REPO_CREDS_PSW
   def os = get_os_info(env.DIST)
   def cephver = env.VERSION.trim()
+  // Whether this cell pushed packages this run. If not (existence check
+  // skipped compilation, or THROWAWAY), build_container accepts any ready
+  // shaman repo for this sha1 instead of waiting for one with our BUILD_URL.
+  def uploaded = build_matrix["${DIST}_${ARCH}_${FLAVOR}"] == true &&
+      params.THROWAWAY == false &&
+      (params.CHACRA_UPLOAD || params.PULP_UPLOAD)
   sh """#!/bin/bash
     export DISTRO=${os.name}
     export RELEASE=${os.version}
     export cephver=${cephver}
+    export UPLOADED=${uploaded}
     if [ "\$PULP_REGISTRY_UPLOAD" == "true" ]; then
       export REMOVE_LOCAL_IMAGES=false
     fi

diff --git a/scripts/build_container b/scripts/build_container
index 21df0d6721406b8bbef6e9a9c8aede0d7de0bb56..5d7886cdffff80887dfb548ea16345e3c0d7c2b9 100755 (executable)
--- a/scripts/build_container
+++ b/scripts/build_container
@@ -12,27 +12,31 @@ if [[ $CI_CONTAINER == "true" && $DISTRO =~ centos|rocky|alma && "$RELEASE" =~ 8
     ready=false
     while ((loop < 15)); do
       curl -s "https://shaman.ceph.com/api/search/?project=ceph&distros=${DISTRO}/${RELEASE}/${ARCH}&sha1=${SHA1}&ref=${BRANCH}&flavor=${FLAVOR}" > shaman.status
-      if [[ ($(jq -r '.[0].status' < shaman.status) == 'ready') ]]; then
-        # If we skipped compilation, we will not have generated a shaman build,
-        # so skip validating against extra.build_url
-        if [[ ${CI_COMPILE:-true} == "false" ]]; then
-          ready=true
-          break
-        elif [[ ($(jq -r '.[0].extra.build_url' < shaman.status) == ${BUILD_URL}) ]]; then
-          ready=true
-          break
-        fi
+      # Shaman holds several repo records for the same sha1 (one per backend
+      # and per uploading job), so scan all of them instead of .[0].
+      # If this build didn't upload packages (UPLOADED=false: the artifact
+      # check found existing repos, or THROWAWAY; CI_COMPILE=false:
+      # compilation disabled), any ready record for this sha1 will do.
+      # Otherwise wait for a ready record posted by this very build.
+      if [[ ${UPLOADED:-true} == "false" || ${CI_COMPILE:-true} == "false" ]]; then
+        ready=$(jq -r '[.[] | select(.status == "ready")] | length > 0' < shaman.status)
+      else
+        ready=$(jq -r --arg build_url "${BUILD_URL}" \
+          '[.[] | select(.status == "ready" and (.extra.build_url? // "") == $build_url)] | length > 0' < shaman.status)
+      fi
+      if [[ "$ready" == "true" ]]; then
+        break
       fi
       ((loop = loop + 1))
       sleep 60
     done
 
-    if [[ "$ready" == "false" ]] ; then
+    if [[ "$ready" != "true" ]] ; then
       chacra_endpoint="ceph/${BRANCH}/${SHA1}/${DISTRO}/${RELEASE}"
       echo "FAIL: timed out waiting for shaman repo to be built:  https://shaman.ceph.com/api/repos/${chacra_endpoint}/flavors/${FLAVOR}/"
-      # don't fail the build here on purpose
-      # update_build_status "failed" "ceph" $NORMAL_DISTRO $NORMAL_DISTRO_VERSION $NORMAL_ARCH
-      # exit 1
+      # Without a ready repo the container build would install stale or
+      # missing packages, so fail instead of building a wrong image.
+      exit 1
     fi
     cd ${WORKSPACE}
     # older jobs used a versioned directory; ceph-dev-pipeline uses an unversioned dir.