ceph-trigger-build: Update docs

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-trigger-build: Fix newlines in job description

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Correct branch

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: Remove preserve- from name

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2372 from ceph/ceph-trigger-build

Add ceph-trigger-build job

ceph-dev-pipeline: Matrix entries for noble, el10

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-trigger-build: support new crimson flavors

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: support new crimson flavors

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: potentially build multiple flavors

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-source-dist: drop unused env vars

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: support the crimson flavor

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: drop unused setup job params

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-source-dist: drop flavor param

Signed-off-by: Zack Cerza <zack@cerza.org>

Add ceph-trigger-build job

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2380 from ceph/dangling-trigger

ceph-dev-new-trigger: Remove dangling job trigger

ceph-dev-new-trigger: Remove dangling job trigger

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2373 from cbodley/wip-rm-jaeger

remove 'jaeger' flavor now that it's on by default

Merge pull request #2379 from dmick/fix-builder-playbook

Fix up ssh client and host key code; add comments on how to run

builder.yml: fix jenkins ssh key and github.com host key code

Previous code was just wrong; 'first_found' can fail, and does
not return file contents; default(URL) just doesn't work

Signed-off-by: Dan Mick <dan.mick@redhat.com>

builder.yml: fix comments, note importance of invocation path

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2377 from ceph/wip-kernel-reduce-distros

kernel: stop building for trusty, xenial and bionic

Merge pull request #2376 from ceph/wip-kernel-dwarf-h

kernel: install elfutils-devel or libdw-dev(el)

kernel: stop building for trusty, xenial and bionic

These are too old and kernel packages are no longer needed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

kernel: install elfutils-devel or libdw-dev(el)

This is a required dependency for CONFIG_GENDWARFKSYMS since 6.15.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

remove 'jaeger' flavor now that it's on by default

jaeger was enabled by default in all ceph builds in
https://github.com/ceph/ceph/pull/44684 which was present in reef and
later releases. so no supported releases need a flavor to turn it on

ceph-dev-new-trigger hadn't been updated from 'DISTROS=centos8 focal'
so is probably not being used

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #2375 from ceph/wip-fix-2369

scripts: Fix missing close case

Merge pull request #2374 from ceph/dev-setup-prj-permission-update

Added new project for copy artifact and updated build discarder

scripts: Fix missing close case

Signed-off-by: David Galloway <david.galloway@ibm.com>

Added new project for copy artifact and updated build discarder

Merge pull request #2369 from Matan-B/wip-matanb-crimson-debug-flavor

Seperate Crimson flavors to crimson-debug/release

ceph-dev-cron: Build both crimson flavors

When building the daily main build, use both crimson flavors.
This is critical for the weekly nightly runs.
Note: Usual builds would still only build a *single* crimson-flavor (debug).
      This commit only affects the dev-cron

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Seperate Crimson flavors to crimson-debug/release
The following change would allow us to:

* Keep testing "Debug" builds (crimson-debug) by default.
* When crimson-only is used, crimson-release would also be built.
* `crimson-release` builds would be used in the weekly nightlies and on
  selected PR gatings (when crimson-debug is not enough).

Note: Once crimson-release is stable enough we could switch crimson-debug
      to actaully be of: "RelWithDebInfo + !NDEBUG + asserts" instead of "Debug".
      For now, let's keep it as Debug since we know it's stable.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2370 from clwluvw/vagrant-libvirt

vagrant-libvirt: install the latest version

vagrant-libvirt: install the latest version

The pinned version is already too old and is failing in CI due to:
```
Error message given during initialization: Unable to resolve dependency:
user requested 'vagrant-libvirt (= 0.12.2)'
```

Let's try the latest and see if the reason for pinning the old one
still exists.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #2368 from dmick/ceph-build-no-containers

ceph-build: remove container build step

ceph-build: remove container build step

This was removed from build_rpm, but then when a reorg happened
to separate the function into a separate script, the fact that
ceph-build doesn't do container builds anymore was missed.

Release container builds now happen after the manual signing
step using job ceph-release-containers

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2367 from Matan-B/wip-matanb-crimson-remove-debug

scripts/build_utils: remove Debug build from Crimson

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

scripts/build_utils: remove Debug build from Crimson

See: https://github.com/ceph/ceph-build/pull/2167/files
Builds should be set to Debug only when ending in "-debug".

https://github.com/ceph/ceph-build/pull/1805 set Crimson to always use Debug.
That made sense in earlier development phases.
However, today we should align with Classical builds and use Debug only when needed.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2365 from ceph/builder-ansible

FIx up builder.yml

examples/builder.yml: update comments with new usage info

Signed-off-by: Dan Mick <dan.mick@redhat.com>

slave.yml: remove old symlink

Signed-off-by: Dan Mick <dan.mick@redhat.com>

examples/builder.yml: set api token based on api_uri

This is really the ceph-jenkins ssh user's password, but they're
now stored in ceph-sepia-secrets so they don't have to be found
and entered on the cmdline here

This keeps 'one ansible-vault encrypt output' in one file, and
should be easier to edit/view/etc.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

examples/builder.yml: fix up subuid/subgid handling code

1) not necessary to make 'find highest uid' conditional (and the
   check was checking 0 instead of !0 anyway)
2) fix up choice of "calculated or default" using "| default()"

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2363 from ceph/pipeline-container

ceph-dev-pipeline: Build in containers

ceph-dev-pipeline: Fix extract/upload of cephadm

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: chown without sudo

And also at the very end of the job.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Store sccache logs

There is an issue where invocations of archiveArtifacts in parallel branches of
the pipeline overwrite each other.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build the builder less often

To build the builder image less often, we can:

1. When building, use two tags: branch.os.arch, and sha1.branch.os.arch
2. After building, push both tags
3. Before building, pull both tags

As a result, we can avoid building entirely if that exact image has already been
pushed, and avoid building at least some layers if the branch had been built
somewhat recently.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Split upload stage from build

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Re-narrow node label

smithis take far too long.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Add and use update_shaman.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build release RPM

Signed-off-by: Zack Cerza <zack@cerza.org>

setup_container_runtime.sh: Prune more images

Only leave one weeks' worth.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Fix CI_COMPILE

Setting an environment variable affects all matrix branches; this approach keeps
them isolated.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Copy even less artifacts

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use ceph-source-dist by default

But continue to offer ceph-dev-new-setup as an option.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2357 from ceph/ceph-source-dist

Add ceph-source-dist job

Add ceph-source-dist job

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2360 from cbodley/wip-doc-PendingReleaseNotes

scripts: docs_pr_only() considers PendingReleaseNotes

scripts: docs_pr_only() considers PendingReleaseNotes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-dev-pipeline: Fix syntax error

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2358 from ceph/fixapt

setup_container_runtime.sh: Add missing sudo

setup_container_runtime.sh: Add missing sudo

... to the apt update invocation.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2356 from ceph/setup-containers

Add scripts/setup_container_runtime.sh

ceph-dev-new-setup: Use setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Add scripts/setup_container_runtime.sh

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2355 from ceph/cdp-containerize

ceph-dev-pipeline: Build in containers

ceph-dev-pipeline: Use uv to set up chacractl

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Relax node labels

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline-trigger: CEPH_BUILD_BRANCH

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Add CEPH_REPO param

So that we can build release branches from ceph.git

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-new-setup: Only use podman if >= v4

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use centos9 hosts

We want to do containerized builds with recent podman versions.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Avoid docker.io rate limits

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Copy less artifacts

This is purely to save time.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Extract less from source dist

Only what we need to build the builder container.

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Build in containers

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2343 from ceph/new-pip

build_utils: Use pip 24.0 by default

build_container: Look for pipeline's source dir

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Use ceph-build branch param

I missed this when filing #2323

Signed-off-by: Zack Cerza <zack@cerza.org>

ceph-dev-pipeline: Fix indentation

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2348 from Matan-B/wip-matanb-cleanup-crimson-builds

ceph-dev*: Remove Crimson unsupported builds

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

ceph-dev*: Remove Crimson un-supported builds

Crimson should be built only for main and T.
Remove R and S builds.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #2346 from Matan-B/wip-matanb-clean-with-seastar

ceph-dev*: final transition to WITH_CRIMSON

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #2345 from dmick/ceph-release-containers-defaults

ceph-release-containers: tweak the parameter defaults

ceph-dev*: final transition to WITH_CRIMSON

Follow up to: https://github.com/ceph/ceph-build/pull/2329
Now that https://github.com/ceph/ceph/pull/62325 is merged.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

build_utils: Use pip 24.0 by default

distutils is removed from python 3.12 so pip 20.3.4 is incompatible with systems running python3.12.

Ubuntu Noble is our new target Jenkins builder OS so let's bump to 24.0 (the latest as of this PR)

https://stackoverflow.com/questions/69919970/no-module-named-distutils-util-but-distutils-is-installed

Signed-off-by: David Galloway <david.galloway@ibm.com>

ceph-release-containers: tweak the parameter defaults

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2344 from dmick/fix-include-raw

PR 2315 added back in an 'include-raw' which is deprecated

PR 2315 added 'include-raw' which is deprecated

should use include-raw-verbatim.  Also, {} must be doubled.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2342 from ceph/add-ceph-release-containers

Add ceph-release-containers job

Merge pull request #2315 from batrick/tentacle-kickoff

Tentacle kickoff!

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Add ceph-release-containers job

Builds prerelease versions of ceph container images: two arch-specific
images and one 'manifest-list' image.  Pushed all to quay.ceph.io/ceph,
in a private set of repos (prerelease-amd64, prerelease-arm64, and
prerelease) for testing and signoff.  Once moved, the script
container/make-manifest-list.py can be used with the --promote flag to
promote them from quay.ceph.io to quay.io for public release.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2341 from dmick/remove-container-build-from-ceph-build

ceph-build: remove container build

*: update relevant scripts and triggers for tentacle

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

scripts: add tentacle

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Merge pull request #2340 from ceph/sepia-fog-fixes

sepia-fog-images: Mark down systems we borrow instead of lock

ceph-build: remove container build

The container must be built from signed packages, and signing
is not part of the Jenkins build.  So we'll create two new jobs
to build the prerelease containers (so that builders don't have
to have access to both amd64 and arm64 build hosts).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

sepia-fog-images: Mark down systems we borrow instead of lock

Signed-off-by: David Galloway <david.galloway@ibm.com>