ceph-dev-pipeline: Consume SHA1 parameter

If SHA1 is passed, ensure it is reflected in the consumed artifact.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-source-dist: Consume SHA1 parameter

If SHA1 is passed, check out the commit directly as opposed to just the branch

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Pass SHA1 param to child job

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2408 from ceph/cdp-rocky

ceph-dev-pipeline: Don't reject rocky

ceph-dev-pipeline: Don't reject rocky

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2406 from dmick/setup-container-runtime

ceph-release-container: add setup_container_runtime.sh

ceph-release-container: add setup_container_runtime.sh

A ceph-release-container run suffered from

error running container: from /usr/bin/crun creating container for
[<sh command>]: sd-bus call: Interactive authentication required.:
Permission denied

That is thought to be because of the lack of 'linger' on the
jenkins-build user, such that the /run/user directory didn't
persist.  The exact mechanism of failure isn't fully understood,
but this script has fixed similar issues for ceph-dev-pipeline.

Also, add an echo of the node for convenience.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2404 from cbodley/wip-noble-gcc14

scripts/build_utils: default to gcc-14 for ubuntu noble

scripts/build_utils: default to gcc-14 for ubuntu noble

noble provides both gcc-13 and 14, but its gcc 13.2.0 suffers from an
LTO bug that causes ceph daemons to crash horribly (tracked in
https://tracker.ceph.com/issues/63867)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2391 from ceph/doc-trlr-fix

ceph-trigger-build: Clarify build job selection

Merge pull request #2402 from ceph/podman-linger-earlier

Enable linger earlier for podman

Enable linger earlier for podman

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2397 from ceph/cdp-cw

ceph-dev-pipeline: Clean up workspace in post

ceph-dev-pipeline: Clean up workspace in post

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2396 from ceph/loginctl-linger

Enable linger for the Jenkins user

Enable linger for the Jenkins user

This fixes errors like:
RunRoot is pointing to a path (/run/user/$ID/containers) which is not writable. Most likely podman will fail.
Error: creating events dirs: mkdir /run/user/$ID: permission denied

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Clarify build job selection

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2384 from ceph/wip-disable-cdnt

ceph-dev-new-trigger: Disable

Merge pull request #2390 from ceph/doc-trlr-fix

ceph-trigger-build: correct git trailer examples

ceph-trigger-build: correct git trailer examples

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2389 from ceph/trigger-pipeline

ceph-trigger-build: Build noble by default

ceph-trigger-build: Build noble by default

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2371 from ceph/pipeline-container

ceph-dev-pipeline: Set CEPH_BUILD_NORMALIZE_PATHS

Merge pull request #2381 from dmick/wip-fog-capture

sepia-fog-images: add image test to 'poll for deploy' loop

Merge pull request #2382 from ceph/wip-noble

Support for Ubuntu Noble

Support for Ubuntu Noble

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2388 from ceph/ctb-crimson

ceph-trigger-build: Correct crimson-only behavior

Merge pull request #2385 from ceph/wip-trailer-docs

ceph-trigger-build: Update docs

ceph-trigger-build: Correct crimson-only behavior

We were accidentally building default instead of crimson-debug

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Update docs

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-trigger-build: build reef on focal

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Remove extra line break

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Add missing def keywords

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-new-trigger: Disable

Replaced by ceph-trigger-build job

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-trigger-build: Fix newlines in job description

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Correct branch

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Remove preserve- from name

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2372 from ceph/ceph-trigger-build

Add ceph-trigger-build job

ceph-dev-pipeline: Matrix entries for noble, el10

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: support new crimson flavors

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: support new crimson flavors

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: potentially build multiple flavors

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-source-dist: drop unused env vars

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: support the crimson flavor

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: drop unused setup job params

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-source-dist: drop flavor param

Signed-off-by: Zack Cerza <zack@cerza.org>

Add ceph-trigger-build job

Signed-off-by: Zack Cerza <zack@cerza.org>

sepia-fog-images: add image test to 'poll for deploy' loop

When checking for scheduled deploy tasks, check not only for existing
deploy tasks but for deploy tasks *for the image we're proposing to
capture*.  There are often deploy tasks on a busy fog server, but we
only need to avoid affecting tasks for the image we're updating

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2380 from ceph/dangling-trigger

ceph-dev-new-trigger: Remove dangling job trigger

ceph-dev-new-trigger: Remove dangling job trigger

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2373 from cbodley/wip-rm-jaeger

remove 'jaeger' flavor now that it's on by default

Merge pull request #2379 from dmick/fix-builder-playbook

Fix up ssh client and host key code; add comments on how to run

builder.yml: fix jenkins ssh key and github.com host key code

Previous code was just wrong; 'first_found' can fail, and does
not return file contents; default(URL) just doesn't work

Signed-off-by: Dan Mick <dan.mick@redhat.com>

builder.yml: fix comments, note importance of invocation path

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2377 from ceph/wip-kernel-reduce-distros

kernel: stop building for trusty, xenial and bionic

Merge pull request #2376 from ceph/wip-kernel-dwarf-h

kernel: install elfutils-devel or libdw-dev(el)

kernel: stop building for trusty, xenial and bionic

These are too old and kernel packages are no longer needed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

kernel: install elfutils-devel or libdw-dev(el)

This is a required dependency for CONFIG_GENDWARFKSYMS since 6.15.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

remove 'jaeger' flavor now that it's on by default

jaeger was enabled by default in all ceph builds in
https://github.com/ceph/ceph/pull/44684 which was present in reef and
later releases. so no supported releases need a flavor to turn it on

ceph-dev-new-trigger hadn't been updated from 'DISTROS=centos8 focal'
so is probably not being used

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2375 from ceph/wip-fix-2369

scripts: Fix missing close case

Merge pull request #2374 from ceph/dev-setup-prj-permission-update

Added new project for copy artifact and updated build discarder

scripts: Fix missing close case

Signed-off-by: David Galloway <david.galloway@ibm.com>

Added new project for copy artifact and updated build discarder

Merge pull request #2369 from Matan-B/wip-matanb-crimson-debug-flavor

Seperate Crimson flavors to crimson-debug/release

ceph-dev-cron: Build both crimson flavors

When building the daily main build, use both crimson flavors.
This is critical for the weekly nightly runs.
Note: Usual builds would still only build a *single* crimson-flavor (debug).
      This commit only affects the dev-cron

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Seperate Crimson flavors to crimson-debug/release
The following change would allow us to:

* Keep testing "Debug" builds (crimson-debug) by default.
* When crimson-only is used, crimson-release would also be built.
* `crimson-release` builds would be used in the weekly nightlies and on
  selected PR gatings (when crimson-debug is not enough).

Note: Once crimson-release is stable enough we could switch crimson-debug
      to actaully be of: "RelWithDebInfo + !NDEBUG + asserts" instead of "Debug".
      For now, let's keep it as Debug since we know it's stable.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

ceph-dev-pipeline: Set CEPH_BUILD_NORMALIZE_PATHS

See https://github.com/ceph/ceph/commit/5478fb94a3daf0887689db137d6153d9b1914793

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2370 from clwluvw/vagrant-libvirt

vagrant-libvirt: install the latest version

vagrant-libvirt: install the latest version

The pinned version is already too old and is failing in CI due to:
```
Error message given during initialization: Unable to resolve dependency:
user requested 'vagrant-libvirt (= 0.12.2)'
```

Let's try the latest and see if the reason for pinning the old one
still exists.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #2368 from dmick/ceph-build-no-containers

ceph-build: remove container build step

ceph-build: remove container build step

This was removed from build_rpm, but then when a reorg happened
to separate the function into a separate script, the fact that
ceph-build doesn't do container builds anymore was missed.

Release container builds now happen after the manual signing
step using job ceph-release-containers

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2367 from Matan-B/wip-matanb-crimson-remove-debug

scripts/build_utils: remove Debug build from Crimson

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

scripts/build_utils: remove Debug build from Crimson

See: https://github.com/ceph/ceph-build/pull/2167/files
Builds should be set to Debug only when ending in "-debug".

https://github.com/ceph/ceph-build/pull/1805 set Crimson to always use Debug.
That made sense in earlier development phases.
However, today we should align with Classical builds and use Debug only when needed.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2365 from ceph/builder-ansible

FIx up builder.yml

examples/builder.yml: update comments with new usage info

Signed-off-by: Dan Mick <dan.mick@redhat.com>

slave.yml: remove old symlink

Signed-off-by: Dan Mick <dan.mick@redhat.com>

examples/builder.yml: set api token based on api_uri

This is really the ceph-jenkins ssh user's password, but they're
now stored in ceph-sepia-secrets so they don't have to be found
and entered on the cmdline here

This keeps 'one ansible-vault encrypt output' in one file, and
should be easier to edit/view/etc.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

examples/builder.yml: fix up subuid/subgid handling code

1) not necessary to make 'find highest uid' conditional (and the
   check was checking 0 instead of !0 anyway)
2) fix up choice of "calculated or default" using "| default()"

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2363 from ceph/pipeline-container

ceph-dev-pipeline: Build in containers

ceph-dev-pipeline: Fix extract/upload of cephadm

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: chown without sudo

And also at the very end of the job.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Store sccache logs

There is an issue where invocations of archiveArtifacts in parallel branches of
the pipeline overwrite each other.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build the builder less often

To build the builder image less often, we can:

1. When building, use two tags: branch.os.arch, and sha1.branch.os.arch
2. After building, push both tags
3. Before building, pull both tags

As a result, we can avoid building entirely if that exact image has already been
pushed, and avoid building at least some layers if the branch had been built
somewhat recently.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Split upload stage from build

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Re-narrow node label

smithis take far too long.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Add and use update_shaman.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build release RPM

Signed-off-by: Zack Cerza <zack@cerza.org>

setup_container_runtime.sh: Prune more images

Only leave one weeks' worth.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Fix CI_COMPILE

Setting an environment variable affects all matrix branches; this approach keeps
them isolated.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Copy even less artifacts

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use ceph-source-dist by default

But continue to offer ceph-dev-new-setup as an option.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2357 from ceph/ceph-source-dist

Add ceph-source-dist job

Add ceph-source-dist job

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2360 from cbodley/wip-doc-PendingReleaseNotes

scripts: docs_pr_only() considers PendingReleaseNotes

scripts: docs_pr_only() considers PendingReleaseNotes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-dev-pipeline: Fix syntax error

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2358 from ceph/fixapt

setup_container_runtime.sh: Add missing sudo

setup_container_runtime.sh: Add missing sudo

... to the apt update invocation.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2356 from ceph/setup-containers

Add scripts/setup_container_runtime.sh