ceph-dev-pipeline: Fix archive-skipping

If sccache is supposed to be enabled, but isn't - for example due to the feature
not being backported - we were still attempting to archive, and hanging. Check
for the existence of the file directly.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2442 from ceph/uv

ceph-build-pull-requests: Use uv; drop OS restriction

Merge pull request #2443 from ceph/restore-pipeline

ceph-trigger-build: Default to ceph-dev-pipeline

Merge pull request #2437 from ceph/release-with-pipeline-commits

Create ceph-release-pipeline for upstream release builds

Merge pull request #2444 from dmick/cephadm-e2e-deps

scripts/dashboard/install-cephadm-e2e-deps.sh: Put back DISTRO

scripts/dashboard/install-cephadm-e2e-deps.sh: Put back DISTRO

fcf4dfee46bfafc678a20c769d0473acb6d42af7 removed the setting
of DISTRO.  Unfortunately one usage of the variable remained
in the script.  Put back the setting.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-dev-pipeline: Support for upstream builds

 - Variable reading/writing was reordered so that job logic
   can override CEPH_REPO, SHA1, and chacra_url.
 - For now, --rpm-no-match-sha avoids https://tracker.ceph.com/issues/72626
 - Optionally archive sccache artifacts only if we used sccache. Avoids
   the job hanging at the end (yes, I know allowEmptyArchive: true should
   take care of it.. 2.jenkins didn't care.)

Signed-off-by: David Galloway <david.galloway@ibm.com>

setup_chacractl: Only ask shaman for chacra node if chacra_url's unset

In a Release Build, we want to push the packages to chacra.ceph.com.
So we manually define chacra_url=https://chacra.ceph.com in ceph-source-dist
by writing it to dist/other_envvars

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-source-dist: Support for ceph-release-pipeline

Again, the changes needed here are:
 - For a Release Build, we are building a tarball based off a
   version commit that has been created by the ceph-tag job
   then pushed to ceph-releases.git.  So instead of taking
   the BRANCH from an upstream Jenkins job, we're reading
   the version commit SHA1 from the sha1.txt file that ceph-tag
   wrote then checking /that/ out and building the tarball from it.
   So some var writing and reading was shuffled around.
 - We do not want this job to support checking out from ceph-private.git
   or ceph-releases.git directly.  Instead, if ceph-release-pipeline
   passed RELEASE_BUILD=true, only then can you clone from those private
   repos.  So we override CEPH_REPO and set chacra_url to the chacra
   instance that doesn't automatically prune repos (chacra.ceph.com).

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-tag: Support specifying different ceph-build.git branch

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-tag: Take TAG_PHASE from ceph-release-pipeline job

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-tag: Always run playbook regardless of TAG

The individual tasks in the role already have when statements.

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-release-pipeline: Add job

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-trigger-build: Default to ceph-dev-pipeline

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-build-pull-requests: Change node label

We don't specifically need trusty anymore.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-build-pull-requests: Support rpm systems

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-build-pull-requests: Improve readability

Signed-off-by: Zack Cerza <zack@cerza.org>

shellcheck_job.py: Tweak output

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-build-pull-requests: Use uv

To install Python dependencies for this job, use uv instead of pip. This is
faster and quieter.

Signed-off-by: Zack Cerza <zack@cerza.org>

Add scripts/setup_uv.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2441 from ceph/podman-tmp

setup_container_runtime.sh: Add tmpfiles config

setup_container_runtime.sh: Add tmpfiles config

This should resolve podman failures that mention "current system boot ID differs
from cached boot ID".

Signed-off-by: Zack Cerza <zack@cerza.org>

setup_container_runtime.sh: Version comparison

... using Go templates instead of jq

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2440 from ceph/wip-yuriw-typo

Fixed typo

Fixed typo

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #2438 from ceph/csd-no-arm

ceph-source-dist: Avoid arm64 machines

ceph-source-dist: Avoid arm64 machines

They are very slow when running this job.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2436 from ceph/cdp-kmsgs

ceph-dev-pipeline: Report kernel msgs in post

ceph-dev-pipeline: Report kernel msgs in post

The inspiration is to help us find instances of OOM kills, but may be helpful
with other issues.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2435 from rhcs-dashboard/fix-nvm

cephadm-dashboard-e2e: use node version from dashboard/frontend

Reviewed-by: Afreen Misbah <afreen@ibm.com>

cephadm-dashboard-e2e: use node version from dashboard/frontend

via .nvmrc

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2418 from bluikko/patch-1

dashboard-cephadm-e2e: fix and silence a few errors

Merge pull request #2431 from ceph/cdp-wo-sha1

ceph-dev-pipeline: Do not require SHA1 property

Merge pull request #2433 from ceph/ctb-no-tags

ceph-trigger-build: Do not trigger on tag pushes

ceph-dev-pipeline: Do not require SHA1 property

dd70a303 inadvertently made the SHA1 property mandatory; correct that.

Fixes: https://tracker.ceph.com/issues/72693
Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Do not trigger on tag pushes

This pipeline doesn't support tags, and it doesn't seem like shaman does either.

Fixes: https://tracker.ceph.com/issues/72693
Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2434 from ceph/cdns

setup_cont_runtime: Move podman debugging info inside command -v

setup_cont_runtime: Move podman debugging info inside command -v

Fails jobs if podman is not installed already

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2427 from ceph/revert-cdp

Revert "ceph-trigger-build: Default to ceph-dev-pipeline"

Merge pull request #2430 from ceph/container-fixes

setup_container_runtime.sh: Reset podman

ceph-tag: Write a sha1 file for later consumption

In a regular build, ceph-dev-pipeline just takes a branch name and makes a source tarball in ceph-source-dist.  ceph-dev-pipeline takes that tarball from the ceph-source-dist job and builds from it so it's able to just get the SHA1 from it.

In a /release/ build, a release branch (e.g., reef-release) is specified and we add an additional version commit on top of it.  So we don't know the SHA1 just based on the branch name.  My solution was to write the version commit's SHA1 to a file and consume it later.

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-tag: Remove throwaway|bool

That var has nothing to do with checking out a previously-existing tag.  THROWAWAY=true just means 'create the packages but don't push them to chacra'

Signed-off-by: David Galloway <david.galloway@ibm.com>

setup_container_runtime.sh: Reset podman

... if `podman system info` fails.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2429 from ceph/shellcheck

Fix more shellcheck errors

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Fix more shellcheck errors

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2428 from ceph/container-fixes

ceph-dev-pipeline: fixes for loginctl and chown

ceph-dev-pipeline: Improve podman cleanup

We're seeing a lot of instances of jobs failing to start due to left-over files
from the containerized build. When calling chown, reset the group as well. Also,
mention some details about the build to make log analysis easier.

Signed-off-by: Zack Cerza <zack@cerza.org>

setup_container_runtime.sh: Always enable linger

Signed-off-by: Zack Cerza <zack@cerza.org>

Revert "ceph-trigger-build: Default to ceph-dev-pipeline"

This reverts commit 432cc024fc03daa59b2af0523d333ed654f1a2ac.

Merge pull request #2416 from ceph/shellcheck

ceph-build-pull-requests: Use shellcheck

ceph-build-pull-requests: Use shellcheck_job.py

Signed-off-by: Zack Cerza <zack@cerza.org>

Add scripts/shellcheck_job.py

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2426 from ceph/ctb-cdp

ceph-trigger-build: Default to ceph-dev-pipeline

ceph-trigger-build: Default to ceph-dev-pipeline

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2425 from ceph/wip-kernel-bc-dependency

kernel: install bc

kernel: install bc

bc has been required for building the Linux kernel for a long time.
It was installed explicitly (albeit from the wrong script) in the deb
case, but not installed in the rpm case:

  error: Failed build dependencies:
   bc is needed by kernel-6.17.0_rc1_g0dccf5229fc7-1.x86_64
  make[2]: *** [scripts/Makefile.package:75: binrpm-pkg] Error 11

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2412 from ceph/specific-sha1

Build specific commits with new build pipelines

Merge pull request #2423 from ceph/no-admins

Remove admin lists from ghprb configs

Remove admin lists from ghprb configs

These will be replaced by proper RABC eventually

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2422 from ceph/fix-bu-sc-errs

build_utils.sh: Fix shellcheck errors

build_utils.sh: Fix shellcheck errors

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2420 from ceph/scr-fixes

setup_container_runtime.sh: Drop bc

setup_container_runtime.sh: Fix shellcheck msgs

Signed-off-by: Zack Cerza <zack@cerza.org>

setup_container_runtime.sh: Drop bc

Use bash instead for math. Thanks to John Mulligan for the suggestion.

Signed-off-by: Zack Cerza <zack@cerza.org>

dashboard-cephadm-e2e: silence a few errors

Change directory before trying to open files from it.
Silences one Jenkins Indication.

Don't try to do docker logs unless docker had something running.
Removes some error lines in Jenkins console log.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #2417 from ceph/builder-wget

builder.yml: Install wget

builder.yml: Install wget

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2415 from ceph/cdnb-scr

ceph-dev-new-build: add setup_container_runtime.sh

ceph-dev-new-build: add setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2387 from ceph/wip-builders-upgrade

builder.yml: Add tasks to update builder

Merge pull request #2411 from ceph/cdp-desc-links

ceph-dev-pipeline: Link to branch, commit

Merge pull request #2414 from dmick/revert-build-utils

Revert "removed build_utils dependency from ceph pull request job"

Revert "removed build_utils dependency from ceph pull request job"

Jobs are using more than just the 'docs check' functions in
build_utils.sh (ssh_exec, get_nr_build_jobs are two that showed up
after merge).  Revert this until it can be more-carefully evaluated.

This reverts commit 28871177b6102eb3b17e260341eaa05a861628c1.

Merge pull request #2413 from dmick/sync-pull-skip-nonexistent-deb-repos

sync-pull: in the 'ugly loop', skip nonexistent repos

Merge pull request #2393 from ceph/ceph_pr_job_refactor

removed build_utils dependency from ceph pull request job

sync-pull: in the 'ugly loop', skip nonexistent repos

This heuristic remains questionable, but continue it on
by allowing some named distro/version repos to not exist without
stopping sync-pull from working

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2405 from cbodley/wip-noble-install-gcc14

scripts/build_utils: setup_pbuilder_for_old_gcc() installs requested version

ceph-dev-pipeline: Consume SHA1 parameter

If SHA1 is passed, ensure it is reflected in the consumed artifact.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-source-dist: Consume SHA1 parameter

If SHA1 is passed, check out the commit directly as opposed to just the branch

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Pass SHA1 param to child job

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2409 from ceph/cdp-rocky9

ceph-dev-pipeline: Add rocky9 target

ceph-dev-pipeline: Link to branch, commit

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2410 from rhcs-dashboard/dashboard-node-install

dashboard-cephadm-e2e: install node via nvm

Reviewed-by: Nizamudeen A <nia@redhat.com>

dashboard-cephadm-e2e: install node via nvm

to avoid pinning the node versions

Signed-off-by: Nizamudeen A <nia@redhat.com>

ceph-dev-pipeline: Add rocky9 target

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2408 from ceph/cdp-rocky

ceph-dev-pipeline: Don't reject rocky

ceph-dev-pipeline: Don't reject rocky

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2406 from dmick/setup-container-runtime

ceph-release-container: add setup_container_runtime.sh

ceph-release-container: add setup_container_runtime.sh

A ceph-release-container run suffered from

error running container: from /usr/bin/crun creating container for
[<sh command>]: sd-bus call: Interactive authentication required.:
Permission denied

That is thought to be because of the lack of 'linger' on the
jenkins-build user, such that the /run/user directory didn't
persist.  The exact mechanism of failure isn't fully understood,
but this script has fixed similar issues for ceph-dev-pipeline.

Also, add an echo of the node for convenience.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

scripts/build_utils: setup_pbuilder_for_old_gcc() installs requested version

ubuntu noble is special in that we gcc-14 is available but gcc-13 is the
default. before using update-alternatives to prefer gcc-14, we need to
make sure it's installed

copy the relevant bit from setup_pbuilder_for_new_gcc()'s hook
D05install-new-gcc to make sure the requested version is installed
before running the D10update-gcc-alternatives hook

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2404 from cbodley/wip-noble-gcc14

scripts/build_utils: default to gcc-14 for ubuntu noble

scripts/build_utils: default to gcc-14 for ubuntu noble

noble provides both gcc-13 and 14, but its gcc 13.2.0 suffers from an
LTO bug that causes ceph daemons to crash horribly (tracked in
https://tracker.ceph.com/issues/63867)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

removed build_utils dependency from ceph pull request job

Merge pull request #2391 from ceph/doc-trlr-fix

ceph-trigger-build: Clarify build job selection

Merge pull request #2402 from ceph/podman-linger-earlier

Enable linger earlier for podman

Enable linger earlier for podman

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2397 from ceph/cdp-cw

ceph-dev-pipeline: Clean up workspace in post

ceph-dev-pipeline: Clean up workspace in post

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2396 from ceph/loginctl-linger

Enable linger for the Jenkins user

Enable linger for the Jenkins user

This fixes errors like:
RunRoot is pointing to a path (/run/user/$ID/containers) which is not writable. Most likely podman will fail.
Error: creating events dirs: mkdir /run/user/$ID: permission denied

Signed-off-by: Zack Cerza <zack@cerza.org>