setup_container_runtime.sh: refresh bash path after podman removal

If we remove podman in the "Found a very old podman; removing" function, we subsequently run `command -v podman` later in the script but we're hitting a stale cache.

e.g.,

```
+ setup_container_runtime
++ id -nu
+ loginctl enable-linger jenkins-build
+ command -v podman
/usr/bin/podman
+ podman system info
++ podman version -f '{{ lt .Client.Version "4" }}'
+ '[' true = true ']'
+ echo 'Found a very old podman; removing'
Found a very old podman; removing
+ command -v dnf
+ command -v apt
/usr/bin/apt
+ sudo apt remove -y podman

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  buildah catatonit cmake cmake-data conmon containernetworking-plugins
  cython3 dh-elpa-helper dh-exec dh-python docutils-common emacsen-common
  fonts-glyphicons-halflings fuse-overlayfs golang golang-1.18 golang-1.18-doc
  golang-1.18-go golang-1.18-src golang-doc
  golang-github-containernetworking-plugin-dnsname
  golang-github-containers-common golang-github-containers-image golang-go
  golang-src gperf javahelper jsonnet junit4 libaio-dev libavahi-glib1
  libbabeltrace-ctf-dev libbabeltrace-dev libbabeltrace1 libblkid-dev
  libc6-dbg libcap-dev libcap-ng-dev libcryptsetup-dev libcunit1 libcunit1-dev
  libcurl4-openssl-dev libdouble-conversion3 libevent-2.1-7 libevent-dev
  libevent-extra-2.1-7 libevent-openssl-2.1-7 libevent-pthreads-2.1-7
  libfmt-dev libfmt8 libfuse-dev libfuse2 libgcrypt20-dev libgmp-dev
  libgmpxx4ldbl libgnutls-dane0 libgnutls-openssl27 libgnutls28-dev
  libgnutlsxx28 libgoogle-perftools-dev libgoogle-perftools4 libgpg-error-dev
  libhamcrest-java libibverbs-dev libidn2-dev libjs-bootstrap libjs-d3
  libjs-eonasdan-bootstrap-datetimepicker libjs-jquery-hotkeys libjs-moment
  libjs-moment-timezone libjs-mustache libjs-rickshaw libjsoncpp25
  libkeyutils-dev libldap-dev libldap2-dev liblmdb-dev liblttng-ust-common1
  liblttng-ust-ctl5 liblttng-ust-dev liblttng-ust-python-agent1 liblttng-ust1
  liblua5.3-0 liblua5.3-dev liblz4-dev libnbd-dev libnbd0 libnl-3-dev
  libnl-genl-3-dev libnl-route-3-dev libnspr4-dev libnss3-dev libnuma-dev
  liboath-dev liboath0 libostree-1-1 libp11-kit-dev libpcre2-16-0
  libpcre2-32-0 libpcre2-dev libpcre2-posix3 libpython3-all-dev libqt5core5a
  libqt5dbus5 libqt5network5 librabbitmq-dev librabbitmq4 librdkafka++1
  librdkafka-dev librdkafka1 librdmacm-dev libre2-dev libreadline-dev
  librhash0 libselinux1-dev libsepol-dev libsnappy-dev libsnappy1v5
  libsqlite3-dev libtasn1-6-dev libtcmalloc-minimal4 libthrift-0.16.0
  libthrift-dev libudev-dev libunbound8 libunwind-dev liburcu-dev
  libutf8proc-dev libutf8proc2 libxmlsec1-dev libxmlsec1-gcrypt
  libxmlsec1-gnutls libxmlsec1-nss libxslt1-dev libyaml-cpp-dev libyaml-cpp0.7
  libzstd-dev nasm nettle-dev ninja-build nlohmann-json3-dev prometheus
  python3-alabaster python3-all python3-all-dev python3-cheroot
  python3-cherrypy3 python3-coverage python3-dateutil python3-docutils
  python3-grpcio python3-imagesize python3-isodate python3-jaraco.classes
  python3-jaraco.collections python3-jaraco.functools python3-jaraco.text
  python3-jmespath python3-lxml python3-natsort python3-onelogin-saml2
  python3-pluggy python3-portend python3-prettytable python3-protobuf
  python3-py python3-roman python3-snowballstemmer python3-sphinx
  python3-tempora python3-toml python3-wcwidth python3-xmlsec
  python3-xmltodict python3-zc.lockfile sgml-base slirp4netns socat
  sphinx-common tox uidmap uuid-dev valgrind xfslibs-dev xml-core xmlstarlet
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  podman
0 upgraded, 0 newly installed, 1 to remove and 6 not upgraded.
After this operation, 36.6 MB disk space will be freed.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 218554 files and directories currently installed.)
Removing podman (3.4.4+ds1-1ubuntu1.22.04.3) ...
Processing triggers for man-db (2.10.2-1) ...
+ command -v podman
/usr/bin/podman
+ command -v podman
/usr/bin/podman
++ podman version -f '{{ lt .Client.Version "5.6.1" }}'
/tmp/jenkins512750246180841280.sh: line 29: /usr/bin/podman: No such file or directory
+ '[' '' = true ']'
++ podman version -f '{{ ge .Client.Version "4" }}'
/tmp/jenkins512750246180841280.sh: line 36: /usr/bin/podman: No such file or directory
+ '[' '' = true ']'
+ podman rm -f ceph_build
/tmp/jenkins512750246180841280.sh: line 66: /usr/bin/podman: No such file or directory
```

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2558 from ceph/djgalloway-patch-1

Clarify testing changes w/ JJB in README

Clarify testing changes w/ JJB in README

Merge pull request #2556 from dmick/ceph-release-containers-version

ceph-release-containers: Add VERSION parameter to make-manifest-list invocation

Merge pull request #2557 from ceph/r10-for-tent

ceph-trigger-build: Build Rocky 10 by default

ceph-trigger-build: Build Rocky 10 by default

Except squid and reef

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-release-containers: Add VERSION parameter to make-manifest-list invocation

Signed-off-by: Dan Mick <dmick@ibm.com>

Merge pull request #2553 from phlogistonjohn/jjm-refactor-bwc

refactor bwc related code into common functions

xxx: workaround branches with slashes (again)

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

Merge pull request #2554 from ceph/dist-on-x86

Revert "ceph-source-dist: Run on any architecture"

Revert "ceph-source-dist: Run on any architecture"

This reverts commit 2a944cd9542c5748316910da97d2c10ddd674a43.

Fixes: https://tracker.ceph.com/issues/75442
I don't have the time to dig into which node dependency is the problem on arm64 but ceph-source-dist runs fine on x86 but not arm64.

ceph-pull-requests-arm64: update file to use new bwc functions

Replace the existing "open coded" implementation with calls to the
recently created bwc.sh functions. This avoids duplication and improves
reuse.

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

ceph-pull-requests: update file to use new bwc functions

Replace the existing "open coded" implementation with calls to the
recently created bwc.sh functions. This avoids duplication and improves
reuse.

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

scripts: create bwc.sh a library for funcs around build-with-containers

Copy and make generic various bits of code present in
ceph-pull-requests/build/build and ceph-pull-requests-arm64/build/build
such that they can be replaced in the future by calls to functions
in this file.

Signed-off-by: John Mulligan <phlogistonjohn@asynchrono.us>

Merge pull request #2552 from ceph/nofail

ceph-dev-cron: ls-remote failure is not fatal

ceph-dev-cron: ls-remote failure is not fatal

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2551 from ceph/stupid

sign-rpms: extra \n

sign-rpms: extra \n

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2550 from ceph/get

sync-pull: Check the actual repo readiness

sync-pull: Check the actual repo readiness

This has been broken for.. possibly ever.

--head results in 405 Method Not Allowed

Just a GET results in 200 no matter what, whether the repo exists or not.

```
++ curl -fsSo - --connect-timeout 5 --max-time 10 https://shaman.ceph.com/api/repos/ceph/reef/efac5a54607c13fa50d4822e50242b86e6e446df/rocky/10/flavors/default/
++ jq -r '.[].status // empty'
+ status=
+ '[' '' = ready ']'
+ echo 'Shaman repo for rocky/10 is not ready (status: unknown). Skipping.'
Shaman repo for rocky/10 is not ready (status: unknown). Skipping.
```

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2540 from ceph/sign-256

sign-rpms: Sign using SHA256 algo

Merge pull request #2549 from ceph/branch-name

Make get-tarballs.sh executable

Make get-tarballs.sh executable

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2499 from jamiepryde/umbrella-release

Umbrella kickoff!

Merge pull request #2546 from shraddhaag/wip-shraddhaag-improve-ci-for-docs

disable CI checks for PRs with only docs/QA changes

disable CI checks for PRs with only docs/QA changes

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Merge pull request #2548 from dmick/add-arm64-c9-main

Add centos9/arm64 to main builds

Add centos9/arm64 to main builds

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-dev-cron: Add umbrella to Jenkinsfile

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>

*: Update scripts and triggers for Umbrella

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>

Merge pull request #2547 from ceph/wip-revert-make-check-qa-only

ceph-pull-requests: don't skip qa-only PRs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

ceph-pull-requests: don't skip qa-only PRs

This is a partial revert of commit 78849bdf7d0d ("ceph-pull-requests:
Skip qa-only PRs").  Unfortunately "make check" needs to be run even on
qa-only PRs at least once to get linter coverage for files under qa/.

Revert to running "make check" on qa-only PRs for ceph-pull-requests
but continue skipping it for ceph-pull-requests-arm64.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2545 from ceph/cb-small

ceph-build-pull-requests fixes

Merge pull request #2543 from ceph/arm64

ceph-dev-cron: Build main for arm64

ceph-build-pull-requests: setuptools for pkg_resources

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-build-pull-requests: Don't reuse previous venv

If we add a pkg to pkgs[@], it won't actually get installed in a reused venv if one exists

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-build-pull-requests: Run on small

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-dev-cron: Comment explaining cfg

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2544 from ceph/csd-noarch

ceph-source-dist: Run on any architecture

ceph-source-dist: Run on any architecture

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-cron: Build main for arm64

Signed-off-by: David Galloway <david.galloway@ibm.com>

scripts/build_utils: Update for Umbrella

Signed-off-by: Jamie Pryde <jamiepry@uk.ibm.com>

Merge pull request #2541 from ceph/cron-poll

ceph-dev-cron: Use straight cron/timed trigger

ceph-dev-cron: Use straight cron/timed trigger

PollSCM was polling ceph-build.git for changes instead of ceph.git.

Signed-off-by: David Galloway <david.galloway@ibm.com>

sign-rpms: Sign using SHA256 algo

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2538 from ceph/api

ceph-api: Use modern label. They've been running on Noble for weeks.

Merge pull request #2533 from ceph/cron-pipe

ceph-dev-cron: Convert to groovy, trigger ceph-dev-pipeline instead

ceph-api: Use modern label. They've been running on Noble for weeks.

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-dev-cron: Convert to groovy, trigger ceph-dev-pipeline instead

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2537 from ceph/pipx

builder.yml: Install pipx

builder.yml: Install pipx

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2536 from ceph/fix-uv-installs

setup_chacractl.sh: Use setup_uv.sh

setup_uv.sh: Do not fail needlessly on apt systems

Signed-off-by: Zack Cerza <zack@cerza.org>

setup_chacractl.sh: Use setup_uv.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2535 from rhcs-dashboard/qa-not-ignore

ceph-pr-api: don't ignore qa/ folder changes

ceph-pr-api: don't ignore qa/ folder changes

API tests are run from qa/ folder so let's not ignore those changes

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2534 from ceph/fix-rebuild

ceph-tag: Fix condition for RCs

ceph-tag: Fix condition for RCs

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2532 from ceph/trixie

Add Trixie

Add Trixie

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2531 from Matan-B/wip-matanb-debug-container-prune

quay-pruner/build/util.py: handle debug tag suffix

quay-pruner/build/util.py: handle debug tag suffix

Follow-up to: https://github.com/ceph/ceph/pull/67041

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2530 from rhcs-dashboard/sync-images-revert-fix

sync-images: fix build script

sync-images: fix build script

i accidentally merged the last PR but forgot to push my local changes to
the PR before doing it. I was testing things locally and thought I
pushed it. So adding the fix that were in my local.

I added GODEBUG=http2client=0  to fix the http client related timeouts
that were happening frequently. And also added a retry so that it won't
fail immediately.

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2524 from rhcs-dashboard/sync-image-fix

sync-images: failing due to ratelimiting in dockerhub

Merge pull request #2529 from ceph/libvirt-restart

scripts: Refactor restart_libvirt_services

scripts: Refactor restart_libvirt_services

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2528 from ceph/cephadm-ansible-py3.12

cephadm-ansible-prs: do not pin to pip 22.4.0

cephadm-ansible-prs: do not pin to pip 22.4.0

given that the builder is now using py3.12, we can't pin to 22.4.0 as it
depends on distutils that was dropped from py3.12

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2527 from ceph/cephadm2

cephadm-ansible-prs: Set !centos9 on both labels

cephadm-ansible-prs: Set !centos9 on both labels

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2526 from ceph/cephadm-prs-update

cephadm-ansible-prs: Don't run on centos9

cephadm-ansible-prs: Don't run on centos9

95% of the builders are Ubuntu Noble and most of them have vagrant & vagrant-libvirt installed

Signed-off-by: David Galloway <david.galloway@ibm.com>

sync-images: failing due to ratelimiting in dockerhub

So trying to authenticate which might increase the ratelimit

https://jenkins.ceph.com/job/sync-images/651/console
```
Copying image sha256:a99b7d9812dbb7527eab8161807e879c3e66951ddccf21c5302061a651289da6 (15/16)
time="2025-12-25T20:16:10Z" level=fatal msg="Error copying ref \"docker://nginx:1.27.4\": copying image 15/16 from manifest list: determining manifest MIME type for docker://nginx:1.27.4: reading manifest sha256:a99b7d9812dbb7527eab8161807e879c3e66951ddccf21c5302061a651289da6 in docker.io/library/nginx: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit"
Build step 'Execute shell' marked build as failure
```

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #2523 from ceph/migrate-cephadm-ansible-jenkins

migrate cephadm-ansible to the main jenkins

migrate cephadm-ansible to the main jenkins

This commit removes this file in order to make cephadm-ansible CI jobs
use the main jenkins.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #2520 from ceph/wip-kernel-ice

kernel: compile ice.ko module

kernel: compile ice.ko module

This driver is needed for

01:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)

on the new trial nodes.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2519 from ceph/no-centos-make-cehck

ceph-pull-request: Don't run make check on CentOS9

ceph-pull-request: Don't run make check on CentOS9

Not ideal but easiest for stabilization for now

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2518 from ceph/fix-withcrimson

ceph-dev-pipeline: fix withCrimson def

ceph-dev-pipeline: fix withCrimson def

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2516 from ceph/withcrimson

ceph-dev-pipeline: Still send with_crimson=true for main and wip

ceph-dev-pipeline: Still send with_crimson=true

The replacement of crimson builds with debug builds is still half baked.  ceph.spec and install-deps are still expecting with_crimson to be set if the crimson dependency packages should be installed.

See https://github.com/ceph/ceph/blame/main/ceph.spec.in#L367-L384 for the dependencies that will never get installed.

This is manifesting in:
```
+ rpmbuild --rebuild '-D_topdir /ceph/rpmbuild' --with=sccache --without=dwz --with=tcmalloc /ceph/ceph-20.3.0-4645.gcfa448da.el9.src.rpm
error: Failed build dependencies:
cryptopp-devel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
gcc-toolset-13-gcc-plugin-annobin is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
gcc-toolset-13-libasan-devel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
gcc-toolset-13-libubsan-devel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
gnutls-devel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
hwloc-devel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
libasan is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
libpciaccess-devel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
libubsan is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
lksctp-tools-devel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
ragel is needed by ceph-2:20.3.0-4645.gcfa448da.el9.x86_64
Installing /ceph/ceph-20.3.0-4645.gcfa448da.el9.src.rpm
2025-12-22 17:48:10,148: INFO: step done: rpm failed in 00:00:27
```

Nothing will ever tell build-with-container.py or install-deps.sh to pull those dependencies in because we're no longer setting WITH_CRIMSON due to this removal https://github.com/ceph/ceph-build/commit/0f0e4fd7dea0c06d855b93581e5b13cc0bf4c350#diff-d34216471695ce2f36f9cf1550524392c85b94d0566b3bc6d591383411b91f25R218-L381.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2513 from ceph/apt-mirror-cwi

ceph-windows: Update qcow2 image location. Bump VM spec.

Merge pull request #2514 from ceph/fix-ceph-api

ceph-pr-api: Fix parameter

ceph-pr-api: Fix parameter

It's actually impossible to manually trigger this job right now without this

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-windows: Update qcow2 image location. Bump VM spec.

The previous one was living on the LRC and served via reverse proxy.  The RDU lab was shut down earlier and I neglected to back the qcow2 image up.  Who knows when the old LRC will be brought back online so the image has been rebuilt.

Also bumping the VM resources.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2512 from ceph/locale

builder.yaml: Ensure locales are installed

Revert "ceph-build-pull-requests: set UTF-8 for ansible-galaxy"

This reverts commit 1f500f1d47dd1ddb463489b7061fab3576c913d6.

Ensure locales are installed. Set correct format.

I guess Ubuntu just tolerated the incorrect spelling.  Or used to?  Anyway..

```
root@toko07:~# grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
root@toko07:~# locale -a | grep -i en_us
en_US.utf8

[root@toko08 ~]# grep PRETTY_NAME /etc/os-release
PRETTY_NAME="CentOS Stream 9"
[root@toko08 ~]# locale -a | grep -i en_us
en_US
en_US.iso885915
en_US.utf8
```

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2497 from Matan-B/wip-matanb-crimson-cleanup

build flavors: Cleanup "crimson", introduce "debug"

Merge pull request #2508 from ceph/newvagrant

builder.yaml: CentOS version-specific chacra repo for vagrant

Merge pull request #2509 from ceph/cwpr

ceph-windows fixes

ceph-windows-PRs: Fix virt-install command

Behavior changed in newer virt-install.

`--import` says boot the prebuilt qcow2.  Older virt-install versions tolerated `--boot hd` without `--import` but now `--import` is required in this scenario and `--boot hd` is redundant.

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-windows-image-build: Typo

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2507 from ceph/no-chgrp

No chgrp

builder.yaml: CentOS version-specific chacra repo for vagrant

We know X version works on CentOS 7.  We need a newer version for CentOS 9.  We should have a per major version repo.

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2506 from dmick/remove-virtualenv

quay-pruner: virtualenv is no more, use python3 -mvenv

ceph-dev-new-setup/build: Introduce crimson-tentacle builds

See the following comment:
```
Tentacle is the last release that needs dedicated Crimson builds,
Later releases are able to use Crimson with the default build.
As the "Crimson flavor" is no longer available, we need a *temporary* way
to be able build Crimson for tentacle.
Note: This could be removed once Crimson we have Umbrella release builds.
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>