Jenkins Job Builder definitions for CBT performance regression checks on Ceph pull
requests.
+Layout
+------
+
+::
+
+ ceph-perf-pull-requests/
+ config/definitions/ceph-perf-pull-requests.yml # JJB job wiring only
+ build/
+ common.sh # shared wipe / process helpers
+ setup-pre.sh # packages, pre-wipe
+ prepare-workloads.py # classic vs crimson YAML + SeaStore device pick
+ prepare-workloads-*.sh # embed the .py into the agent workspace
+ setup-post.sh # venv, github check in_progress
+ run-cbt # one CBT archive run (env: SRC_DIR, WORKLOAD, ...)
+ compare # compare.py + github check completed
+ cleanup # postbuild wipe
+ reboot # failure/abort reboot
+ README.rst
+
+Shell steps are inlined into the Jenkins job at ``jenkins-jobs update`` time via
+``!include-raw-verbatim`` (same pattern as ``ceph-volume-cephadm-prs``). Template
+parameters are passed with EnvInject (``OSD_FLAVOR``, check app ids, etc.).
+
Jobs
----
The ``ceph-perf`` project generates two freestyle jobs:
-- ``ceph-perf-classic`` — same Crimson-inclusive build as ``ceph-pull-requests``,
- CBT run with ``run-cbt.sh --classical`` (classical ``ceph-osd``)
-- ``ceph-perf-crimson`` — same build, CBT run with default ``run-cbt.sh``
- (``crimson-osd`` via ``vstart.sh --crimson``)
+- ``ceph-perf-classic`` — classical ``ceph-osd`` via ``run-cbt.sh --classical``,
+ workload from ``qa/suites/perf-basic/workloads/radosbench_4K_write.yaml``
+- ``ceph-perf-crimson`` — ``crimson-osd`` via ``run-cbt.sh``, workloads from
+ ``src/test/crimson/cbt/radosbench_4K_{read,write}.yaml``. **Always SeaStore**
+ (never CyanStore). Prefers the **3 largest** unmounted NVMes when present
+ (pilot node ``o02``, see https://tracker.ceph.com/issues/78071 — so three
+ 7.3T drives, never OS ``sda`` / small mounted disks). If fewer than 3 spare
+ NVMes exist, creates three ~32GiB sparse images under
+ ``$WORKSPACE/seastore-imgs/`` and still runs SeaStore.
Both run on ``performance`` nodes, build ``ceph-main`` and the PR merge ref
(``WITH_CRIMSON=ON``, ``vstart-base`` + ``crimson-osd``; compiler selection via
-``run-make.sh`` / ``discover_compiler``, same idea as make-check), then execute
-**both** workloads from ``ceph-main``:
-
-- ``src/test/crimson/cbt/radosbench_4K_read.yaml`` — 4K random read
-- ``src/test/crimson/cbt/radosbench_4K_write.yaml`` — 4K write
+``run-make.sh`` / ``discover_compiler``, same idea as make-check). Only one perf
+job runs at a time (``concurrent: false`` + shared throttle category ``ceph-perf``)
+so classic and crimson cannot load the same machine together. The ``ceph-perf``
+throttle category must exist in Jenkins (Throttle Concurrent Builds plugin).
Results are compared with ``cbt/compare.py``. A GitHub check
-(``perf-test-{osd-flavor}``) gets a combined markdown report with separate
-sections for read and write. The check fails if **either** workload regresses.
+(``perf-test-{osd-flavor}``) gets a markdown report that includes node name,
+store backend, and SHAs. The check fails if any compared workload regresses.
Triggering
----------
What the job does
-----------------
-For each of ``ceph-main`` and the PR merge ref the job:
-
-1. Builds Ceph once (the second workload reuses ``build/``).
-2. Runs CBT for **read**, archiving under ``{basedir}/read/<short-sha>/``.
-3. Runs CBT for **write**, archiving under ``{basedir}/write/<short-sha>/``.
-
-Then ``compare.py`` runs twice (read archive pair, write archive pair) and the
-markdown outputs are merged into one GitHub check body. Example shape:
-
-**4K random read** — ``all 20 tests passed``
-
-============== ==================== ========== ======== ========
-run metric baseline result accepted
-============== ==================== ========== ======== ========
-prefill/host/0 bandwidth … …
-prefill/host/0 iops_avg … …
-… … … …
-rand/host/1 latency_avg … …
-============== ==================== ========== ======== ========
-
-**4K write** — ``all 10 tests passed``
-
-============== ==================== ========== ======== ========
-run metric baseline result accepted
-============== ==================== ========== ======== ========
-write/host/0 bandwidth … …
-write/host/0 iops_avg … …
-… … … …
-write/host/1 latency_avg … …
-============== ==================== ========== ======== ========
-
-Blank ``accepted`` means OK; ``:x:`` / ❌ means the metric failed the rule.
-
-Benchmark YAMLs are always taken from **ceph-main** (not the PR tree) so both
-sides use the same workload definition and acceptance rules.
+1. Prepares flavor-specific workload YAML(s) under ``$WORKSPACE/perf-workloads/``
+ (adds ``acceptable:`` with ~10% near-tolerance for compare.py). It also adds a
+ short-term **warm-up** ``prefill_time`` (>= the measured window, min 30s) so
+ the measurement does not run on a cold cluster: radosbench has no dedicated
+ warm-up phase, and a cold/absent prefill adds variance (and can leave a read
+ phase with no output). This applies to the crimson **read** workload and the
+ classic **perf-basic write** workload (crimson write is left as the checked-in
+ file). The prefill pass lands in ``prefill/`` and is not compared. The durable
+ fix belongs in CBT / the checked-in Ceph CBT YAML.
+2. For each of ``ceph-main`` and the PR merge ref: build once (reuse ``build/``
+ for a second workload), run CBT, archive under
+ ``{basedir}/{workload}/{store_tag}/<short-sha>/``.
+3. Compare archives and post the GitHub check.
+4. Archive reports + CBT results as Jenkins artifacts (kept ~60 days with the
+ build console), so “View more details” keep working after the fact.
+
+On failure/success the job drops partial CBT archives (so they are never reused),
+stops cluster processes, and wipes the job’s SeaStore NVMes (``wipefs`` + leading
+zeros; nvme-only, never if mounted). Compare regressions fail both the GitHub
+check **and** the Jenkins build.
+
+An archive is treated as complete (reusable, and accepted after a run) only if
+the **measured** phase produced ``json_output.*`` (``write/``, ``rand/`` or
+``seq/`` — ``prefill/`` alone does not count). This prevents reusing a run that
+was SIGKILLed mid-read, which otherwise makes ``compare.py`` fail with
+``FileNotFoundError`` on a missing ``json_output``.
+
+``store_tag`` is ``classic`` or ``seastore`` so backends never reuse each
+other’s baselines.
Developer guide: reading the results
------------------------------------
What question the check answers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-**Did this PR make the fixed 4K CBT workloads worse than current ``main`` on the
+**Did this PR make the fixed CBT workload(s) worse than current ``main`` on the
performance node, beyond the allowed tolerance?**
It is a regression smoke signal, not a full performance study and not a
Where things live
~~~~~~~~~~~~~~~~~
-================ ================================= ============================================
-Piece Location Role
-================ ================================= ============================================
-Workload + rules Ceph ``src/test/crimson/cbt/`` ``radosbench_4K_{read,write}.yaml``
-Job wiring This directory (JJB YAML) Build, run CBT, post GitHub check
-CBT runner ``https://github.com/ceph/cbt`` Runs ``rados bench``, collects metrics
-Compare ``cbt/compare.py`` PR archive vs main archive → table
-GitHub check ``perf-test-classic`` / Report on the pull request
- ``perf-test-crimson``
-================ ================================= ============================================
+================== =================================== ============================================
+Piece Location Role
+================== =================================== ============================================
+Classic workload Ceph ``qa/suites/perf-basic/`` ``radosbench_4K_write.yaml``
+Crimson workload Ceph ``src/test/crimson/cbt/`` ``radosbench_4K_read/write.yaml``
+Job wiring This directory (JJB YAML) Build, run CBT, post GitHub check
+CBT runner ``https://github.com/ceph/cbt`` Runs ``rados bench``, collects metrics
+Compare ``cbt/compare.py`` PR archive vs main archive → table
+GitHub check ``perf-test-classic`` / crimson Report on the pull request
+Jenkins artifacts Build artifacts + console Baseline/PR archives, report, meta
+================== =================================== ============================================
Useful links:
-- Read workload: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_read.yaml
-- Write workload: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_write.yaml
+- Classic write: https://github.com/ceph/ceph/blob/main/qa/suites/perf-basic/workloads/radosbench_4K_write.yaml
+- Crimson read: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_read.yaml
+- Crimson write: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_write.yaml
- Metric collectors: https://github.com/ceph/cbt/blob/main/benchmark/radosbench.py
- Acceptance evaluation: https://github.com/ceph/cbt/blob/main/benchmark/benchmark.py
- Compare / report: https://github.com/ceph/cbt/blob/main/compare.py
+- Perf pilot node (SeaStore NVMe): https://tracker.ceph.com/issues/78071
What each workload runs
~~~~~~~~~~~~~~~~~~~~~~~
-**4K random read** (``radosbench_4K_read.yaml``):
+**Classic — 4K write** (``qa/suites/perf-basic/.../radosbench_4K_write.yaml``):
-- Small local cluster (3 OSDs, replicated pool).
-- Prefill write (~3s) so there is data to read.
-- Random 4K read (~30s).
-- Two concurrent client processes (``concurrent_procs: 2``).
+- Classical ``ceph-osd`` (``run-cbt.sh --classical``).
+- Write-only 4K bench (``time: 300``, ``concurrent_ops: 4``, 256 PGs).
+- ``acceptable:`` is injected by the job (perf-basic upstream has none).
-**4K write** (``radosbench_4K_write.yaml``):
+**Crimson — 4K random read** (``radosbench_4K_read.yaml``):
-- Same cluster shape.
-- Write-only 4K bench (~3s); no separate prefill/read phase.
+- Prefill write (~3s) then random 4K read (~30s), ``concurrent_ops: 16``, 128 PGs.
- Two concurrent client processes.
-Why the row counts are 20 and 10
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Each compared cell is one **(phase × client process × metric)** tuple.
-
-Five metrics:
-
-=================== ================================================= ============
-Metric Meaning Better means
-=================== ================================================= ============
-bandwidth Throughput Higher
-iops_avg Average IOPS Higher
-iops_stddev IOPS variance / noise Lower
-latency_avg Average latency Lower
-cpu_cycles_per_op CPU cycles per op (from ``perf`` if collected) Lower
-=================== ================================================= ============
-
-- Read: phases ``prefill`` + ``rand``, 2 clients, 5 metrics → **20 rows**
- (names like ``prefill/toko02/0``, ``rand/toko02/1``).
-- Write: phase ``write`` only, 2 clients, 5 metrics → **10 rows**
- (names like ``write/toko02/0``).
-
-``cpu_cycles_per_op`` may show ``0.0`` when ``perf`` data was not collected; that
-usually passes as a no-op comparison.
+**Crimson — 4K write** (``radosbench_4K_write.yaml``):
-How to read the GitHub table columns
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+- Write-only 4K bench (~3s), same cluster shape.
-======== =========================================================
-Column Meaning
-======== =========================================================
-run Phase / machine / client process (``rand/host/0``)
-metric Which number
-baseline Value from **current ceph** ``main``
-result Value from **this PR**
-accepted Blank = OK; ``:x:`` / ❌ = failed the acceptance rule
-======== =========================================================
-
-Pass / fail rules (``acceptable:`` in the YAML)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-::
+Pass / fail rules
+~~~~~~~~~~~~~~~~~
- bandwidth: (or (greater) (near 0.05)) # ≥ baseline, or within ~5%
- iops_avg: (or (greater) (near 0.05))
- latency_avg: (or (less) (near 0.05)) # ≤ baseline, or within ~5%
- iops_stddev: (or (less) (near 2.00)) # ≤ baseline, or within ~2×
- cpu_cycles_per_op:(or (less) (near 0.05))
+Jenkins workspace copies use ~**10%** near-tolerance (wider than the 5% baked into
+the crimson CBT YAML upstream) to reduce single-shot false positives::
-Small regressions up to about 5% on throughput/latency are allowed;
-noise (``iops_stddev``) may grow up to about 2× baseline.
+ bandwidth: (or (greater) (near 0.10))
+ iops_avg: (or (greater) (near 0.10))
+ latency_avg: (or (less) (near 0.10))
+ iops_stddev: (or (less) (near 2.00))
+ cpu_cycles_per_op:(or (less) (near 0.10))
Jenkins console log order (easy to misread)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CBT verbose logs look like::
- rand/host/1: bandwidth: (or (greater) (near 0.05)):: 203.5/220.4 => rejected
+ rand/host/1: bandwidth: (or (greater) (near 0.10)):: 203.5/220.4 => rejected
The fraction is **``result/baseline``** (PR first, main second), **not**
-baseline/result. In the example above the PR is slower than main, so rejection
-is correct. Prefer the GitHub table headers when in doubt.
+baseline/result. Prefer the GitHub table headers when in doubt.
What the baseline is
~~~~~~~~~~~~~~~~~~~~
1. Checks out current ``origin/main`` into ``ceph-main``.
2. Builds and runs CBT; archives under
- ``$WORKSPACE/cbt-results/{read,write}/<main-short-sha>/``.
+ ``$WORKSPACE/cbt-results/{workload}/{store_tag}/<main-short-sha>/``.
3. Checks out the PR merge ref into ``ceph-pr`` and does the same under
- ``$WORKSPACE/ceph-pr/{
read,write}/<pr-short-sha>/``.
+ ``$WORKSPACE/ceph-pr/{
workload}/{store_tag}/<pr-short-sha>/``.
4. Compares PR archives to main archives.
-If a main short-SHA archive already exists and is non-empty on that node, the
-main CBT run for that workload is **reused** (not rebuilt). When ``main`` moves,
-you get a new baseline directory — numbers can look very different from an
-earlier run on the same PR. Node noise can also move results on the same SHA.
-
-How to add or change metrics
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-1. Edit the ``acceptable:`` block in the Ceph workload YAML(s) if the metric is
- already collected by CBT.
-2. To collect a **new** measurement, implement a getter in
- ``cbt/benchmark/radosbench.py`` and wire evaluation in
- ``cbt/benchmark/benchmark.py``.
-3. ``compare.py`` only reports; it does not define metrics.
+If a matching **baseline** archive already exists and is **complete** (its
+measured phase has ``json_output.*``) on that node, the main CBT run is
+**reused** (PR-side archives are rebuilt each run because the ``ceph-pr``
+checkout wipes its workspace). When ``main`` moves, or the store backend
+changes (new ``store_tag``), you get a new baseline directory.
Teuthology YAML translation
---------------------------
-Benchmark definitions under ``src/test/crimson/cbt/`` use teuthology's
-``tasks`` format. ``run-cbt.sh`` calls ``t2c.py`` to extract the ``cbt`` task
-and emit a CBT configuration. That translator (including ``yaml.safe_load`` for
-input parsing) lives in the Ceph repository with unit tests in
-``test_t2c.py``; the Jenkins job does not patch it at build time.
+Benchmark definitions use teuthology's ``tasks`` format. ``run-cbt.sh`` calls
+``t2c.py`` to extract the ``cbt`` task and emit a CBT configuration. That
+translator lives in the Ceph repository with unit tests in ``test_t2c.py``; the
+Jenkins job does not patch it at build time.
--- /dev/null
+#!/bin/bash
+set -euo pipefail
+: "${WORKSPACE:?}"
+set +e
+# Postbuild cleanup on SUCCESS/UNSTABLE/FAILURE/ABORTED.
+# Best-effort after the WORKSPACE guard (wipe/pkill must not fail the publisher).
+kill_cluster_procs
+wipe_seastore_devs
--- /dev/null
+#!/bin/bash
+# Shared helpers for ceph-perf jobs.
+# Included (concatenated) before setup / run-cbt / compare / cleanup via JJB.
+
+kill_cluster_procs() {
+ local p
+ for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados; do
+ sudo pkill -9 -x "$p" || true
+ done
+}
+
+# Wipe SeaStore targets listed in $WORKSPACE/seastore-devs.txt:
+# - /dev/nvme*: wipefs + leading zeros (never if mounted)
+# - $WORKSPACE/seastore-imgs/* sparse images: delete
+# Anything else is refused.
+wipe_seastore_devs() {
+ # Require WORKSPACE so we never operate on /seastore-imgs or similar.
+ : "${WORKSPACE:?}"
+ local devs_file="${WORKSPACE}/seastore-devs.txt"
+ local img_dir="${WORKSPACE}/seastore-imgs"
+ local dev mountpoints
+ if test ! -s "$devs_file"; then
+ rm -rf "$img_dir"
+ return 0
+ fi
+ while IFS= read -r dev || test -n "$dev"; do
+ test -n "$dev" || continue
+ case "$dev" in
+ /dev/nvme*)
+ if test ! -b "$dev"; then
+ echo "skip wipe, not a block device: $dev" >&2
+ continue
+ fi
+ if ! command -v lsblk >/dev/null 2>&1; then
+ echo "REFUSING to wipe (lsblk not found to verify mounts): $dev" >&2
+ continue
+ fi
+ # Fail closed: if lsblk cannot report mount state, do not wipe.
+ if ! mountpoints=$(lsblk -n -o MOUNTPOINT "$dev" 2>/dev/null); then
+ echo "REFUSING to wipe (lsblk failed to verify mounts): $dev" >&2
+ continue
+ fi
+ if printf '%s\n' "$mountpoints" | grep -q '[^[:space:]]'; then
+ echo "REFUSING to wipe mounted device: $dev" >&2
+ continue
+ fi
+ echo "Wiping SeaStore device $dev"
+ sudo wipefs -a "$dev" || true
+ sudo dd if=/dev/zero of="$dev" bs=1M count=100 status=none conv=fsync || true
+ ;;
+ "${WORKSPACE}/seastore-imgs"/*)
+ echo "Removing SeaStore sparse image $dev"
+ rm -f "$dev" || true
+ ;;
+ *)
+ echo "REFUSING to wipe unexpected SeaStore path: $dev" >&2
+ continue
+ ;;
+ esac
+ done < "$devs_file"
+ rm -rf "$img_dir"
+}
+
+cleanup_vstart() {
+ # Prefer stopping from the active tree if present.
+ local tree
+ for tree in "${SRC_DIR:-}" ceph-pr ceph-main; do
+ test -n "$tree" || continue
+ if test -f "${WORKSPACE}/${tree}/build/ceph.conf"; then
+ ( cd "${WORKSPACE}/${tree}/build" && ../src/stop.sh --crimson ) || true
+ ( cd "${WORKSPACE}/${tree}/build" && ../src/stop.sh ) || true
+ fi
+ done
+ kill_cluster_procs
+}
--- /dev/null
+#!/bin/bash
+set -euxo pipefail
+
+: "${WORKSPACE:?}"
+: "${OSD_FLAVOR:?}"
+: "${CHECK_APP_ID:?}"
+: "${CHECK_INSTALL_ID:?}"
+: "${CHECK_NAME:?}"
+: "${CHECK_REPO_OWNER:?}"
+: "${CHECK_REPO_NAME:?}"
+
+ARCHIVE_MAIN="${WORKSPACE}/cbt-results"
+ARCHIVE_PR="${WORKSPACE}/ceph-pr"
+
+cd "${WORKSPACE}/ceph-main"
+sha_main=$(git rev-parse --short HEAD)
+cd "${WORKSPACE}/ceph-pr"
+sha_pr=$(git rev-parse --short HEAD)
+store_tag=$(cat "${WORKSPACE}/perf-store-tag")
+
+# shellcheck disable=SC1091
+. "${WORKSPACE}/gh-venv/bin/activate"
+
+report="${WORKSPACE}/report.md"
+: > "$report"
+result=success
+
+{
+ echo "flavor: ${OSD_FLAVOR}"
+ echo "node: $(hostname -f 2>/dev/null || hostname)"
+ echo "store: $store_tag"
+ echo "baseline_sha: $sha_main"
+ echo "pr_sha: $sha_pr"
+ echo "baseline_archives: ${ARCHIVE_MAIN}/<workload>/$store_tag/$sha_main/"
+ echo "pr_archives: ${ARCHIVE_PR}/<workload>/$store_tag/$sha_pr/"
+ if test -f "${WORKSPACE}/perf-meta.txt"; then
+ cat "${WORKSPACE}/perf-meta.txt"
+ fi
+} >> "$report"
+echo >> "$report"
+
+compare_workload() {
+ local label=$1
+ local workload=$2
+ local archive_main="${ARCHIVE_MAIN}/${workload}/${store_tag}/${sha_main}"
+ local archive_pr="${ARCHIVE_PR}/${workload}/${store_tag}/${sha_pr}"
+ local partial="${WORKSPACE}/report-${workload}.md"
+ rm -f "$partial"
+ echo "## ${label}" >> "$report"
+ echo >> "$report"
+ if python3 "${WORKSPACE}/cbt/compare.py" -v \
+ -a "$archive_pr" \
+ -b "$archive_main" \
+ --output "$partial"; then
+ :
+ else
+ result=failure
+ if test ! -s "$partial"; then
+ echo "cbt/compare.py failed for ${label}; see Jenkins console." > "$partial"
+ fi
+ fi
+ cat "$partial" >> "$report"
+ echo >> "$report"
+}
+
+if test "${OSD_FLAVOR}" = "classic"; then
+ compare_workload "4K write" write
+else
+ compare_workload "4K random read" read
+ compare_workload "4K write" write
+fi
+
+github-check \
+ --owner "${CHECK_REPO_OWNER}" \
+ --repo "${CHECK_REPO_NAME}" \
+ --pkey-file "${GITHUB_CHECK_PKEY_PEM}" \
+ --app-id "${CHECK_APP_ID}" \
+ --install-id "${CHECK_INSTALL_ID}" \
+ --name "${CHECK_NAME}" \
+ --sha "${ghprbActualCommit}" \
+ --external-id "${BUILD_ID}" \
+ --details-url "${BUILD_URL}" \
+ --status completed --conclusion "${result}" \
+ --title "${CHECK_NAME}" \
+ --summary "${result}" \
+ --text "$report"
+
+if test "$result" != "success"; then
+ exit 1
+fi
--- /dev/null
+#!/bin/bash
+# Embed prepare-workloads.py into the workspace (ceph-build is not checked out on the agent).
+# JJB concatenates: this header + prepare-workloads.py + footer.
+cat > "${WORKSPACE}/prepare-workloads.py" <<'PY'
--- /dev/null
+#!/usr/bin/env python3
+"""Prepare flavor-specific CBT YAMLs and SeaStore device list.
+
+Env:
+ WORKSPACE Jenkins workspace
+ OSD_FLAVOR classic | crimson
+"""
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import yaml
+
+
+def inject_acceptable(doc: dict, acceptable: dict) -> None:
+ for task in doc.get("tasks") or []:
+ if not isinstance(task, dict) or "cbt" not in task:
+ continue
+ bench = task["cbt"].setdefault("benchmarks", {}).setdefault("radosbench", {})
+ bench["acceptable"] = dict(acceptable)
+ return
+ raise SystemExit("no cbt/radosbench task found in workload YAML")
+
+
+def warmup_read_prefill(doc: dict) -> str | None:
+ """Warm the system with writes before a read measurement.
+
+ radosbench has no dedicated warm-up phase: a read test does one short
+ prefill write pass (default prefill_time: 3) then measures. Cold caches and
+ a tiny working set add variance and can leave the read phase without output.
+ Short-term stand-in for a real warm-up (per perf maintainer): lengthen the
+ single prefill to at least the measured read window. Durable fix belongs in
+ CBT (a native warm-up/2nd-prefill) or the checked-in Ceph CBT YAML.
+ """
+ for task in doc.get("tasks") or []:
+ if not isinstance(task, dict) or "cbt" not in task:
+ continue
+ bench = task["cbt"].setdefault("benchmarks", {}).setdefault("radosbench", {})
+ is_read = (
+ bench.get("read_only")
+ or "readmode" in bench
+ or "prefill_time" in bench
+ or "prefill_objects" in bench
+ )
+ if not is_read:
+ return None # write-only test: no read prefill to warm
+ read_time = bench.get("read_time") or bench.get("time") or 30
+ current = bench.get("prefill_time") or 0
+ warmup = max(current, read_time, 30)
+ bench["prefill_time"] = warmup
+ return f"read_warmup_prefill_time={current}->{warmup}"
+ return None
+
+
+def warmup_write_prefill(doc: dict) -> str | None:
+ """Warm the system with writes before a write measurement.
+
+ radosbench has no dedicated warm-up phase, so a write test measures from a
+ cold cluster (cold caches/allocator, unallocated store) which adds variance.
+ Per perf maintainer, add a prefill write pass (>= the measured write window,
+ min 30s) before the measurement. radosbench runs prefill -> write for a
+ write_only test, so the measured 'write/' phase is unaffected while
+ 'prefill/' warms the system. Durable fix belongs in CBT / the Ceph CBT YAML.
+ """
+ for task in doc.get("tasks") or []:
+ if not isinstance(task, dict) or "cbt" not in task:
+ continue
+ bench = task["cbt"].setdefault("benchmarks", {}).setdefault("radosbench", {})
+ write_time = bench.get("write_time") or bench.get("time") or 30
+ current = bench.get("prefill_time") or 0
+ warmup = max(current, write_time, 30)
+ bench["prefill_time"] = warmup
+ return f"write_warmup_prefill_time={current}->{warmup}"
+ raise SystemExit("no cbt/radosbench task found in workload YAML")
+
+
+def device_has_mount(dev_path: str) -> bool:
+ try:
+ out = subprocess.check_output(
+ ["lsblk", "-n", "-o", "MOUNTPOINT", dev_path],
+ text=True,
+ stderr=subprocess.DEVNULL,
+ )
+ except (subprocess.CalledProcessError, FileNotFoundError):
+ return True # fail closed
+ return any(line.strip() for line in out.splitlines())
+
+
+def main() -> int:
+ ws = Path(os.environ["WORKSPACE"])
+ flavor = os.environ["OSD_FLAVOR"]
+ out = ws / "perf-workloads"
+ out.mkdir(parents=True, exist_ok=True)
+ main_src = ws / "ceph-main"
+ # 10% near-tolerance to reduce single-shot false positives vs upstream 5%.
+ acceptable = {
+ "bandwidth": "(or (greater) (near 0.10))",
+ "iops_avg": "(or (greater) (near 0.10))",
+ "iops_stddev": "(or (less) (near 2.00))",
+ "latency_avg": "(or (less) (near 0.10))",
+ "cpu_cycles_per_op": "(or (less) (near 0.10))",
+ }
+ meta: list[str] = []
+ devs_file = ws / "seastore-devs.txt"
+
+ if flavor == "classic":
+ src = main_src / "qa/suites/perf-basic/workloads/radosbench_4K_write.yaml"
+ doc = yaml.safe_load(src.read_text())
+ inject_acceptable(doc, acceptable)
+ # perf-basic has no prefill; add a warm-up write pass before measuring.
+ meta.append(f"radosbench_4K_write.yaml_{warmup_write_prefill(doc)}")
+ (out / "radosbench_4K_write.yaml").write_text(
+ yaml.safe_dump(doc, sort_keys=False)
+ )
+ meta.append(f"write_yaml_source={src}")
+ meta.append("workloads=write")
+ store_tag = "classic"
+ devs_file.write_text("")
+ else:
+ for name in ("radosbench_4K_read.yaml", "radosbench_4K_write.yaml"):
+ src = main_src / "src/test/crimson/cbt" / name
+ doc = yaml.safe_load(src.read_text())
+ inject_acceptable(doc, acceptable)
+ note = warmup_read_prefill(doc)
+ if note:
+ meta.append(f"{name}_{note}")
+ (out / name).write_text(yaml.safe_dump(doc, sort_keys=False))
+ meta.append(f"{name}_source={src}")
+ meta.append("workloads=read,write")
+ # Crimson always uses SeaStore (never CyanStore). Prefer 3 spare NVMes;
+ # otherwise create workspace sparse images so the job still runs SeaStore.
+ store_tag = "seastore"
+ nvmes: list[tuple[int, str]] = []
+ for line in os.popen(
+ "lsblk -dn -b -o NAME,TYPE,SIZE,MOUNTPOINT"
+ ).read().splitlines():
+ parts = line.split()
+ if len(parts) < 3:
+ continue
+ name, typ, size = parts[0], parts[1], parts[2]
+ mnt = parts[3] if len(parts) > 3 else ""
+ if typ != "disk" or not name.startswith("nvme") or mnt not in ("", "-"):
+ continue
+ path = f"/dev/{name}"
+ if device_has_mount(path):
+ meta.append(f"skip_mounted={path}")
+ continue
+ try:
+ nvmes.append((int(size), path))
+ except ValueError:
+ continue
+ nvmes.sort(key=lambda t: (-t[0], t[1]))
+ if len(nvmes) >= 3:
+ chosen = [path for _, path in nvmes[:3]]
+ meta.append("seastore_backend=nvme")
+ meta.append(
+ "seastore_sizes_bytes="
+ + ",".join(str(sz) for sz, _ in nvmes[:3])
+ )
+ else:
+ img_dir = ws / "seastore-imgs"
+ img_dir.mkdir(parents=True, exist_ok=True)
+ # ~32GiB sparse files (same order as upstream cyanstore memstore_device_bytes).
+ sparse_bytes = 34359738368
+ chosen = []
+ for i in range(3):
+ img = img_dir / f"osd-{i}.img"
+ # Create/resize sparse image without allocating full size.
+ with open(img, "wb") as fh:
+ fh.truncate(sparse_bytes)
+ chosen.append(str(img))
+ meta.append("seastore_backend=sparse-file")
+ meta.append(f"seastore_sparse_bytes={sparse_bytes}")
+ meta.append(
+ f"seastore_note=only {len(nvmes)} unmounted NVMe(s); "
+ "using workspace sparse images"
+ )
+ devs = ",".join(chosen)
+ src_sh = (main_src / "src/script/run-cbt.sh").read_text()
+ old = (
+ "MDS=0 MGR=1 OSD=3 MON=1 $source_dir/src/vstart.sh -n -X \\\n"
+ " --without-dashboard --cyanstore \\\n"
+ ' -o "memstore_device_bytes=34359738368" \\\n'
+ " --crimson --nodaemon --redirect-output \\\n"
+ ' --osd-args "--memory 4G"'
+ )
+ new = (
+ "MDS=0 MGR=1 OSD=3 MON=1 $source_dir/src/vstart.sh -n -X \\\n"
+ f" --without-dashboard --seastore --seastore-devs {devs} \\\n"
+ " --crimson --nodaemon --redirect-output \\\n"
+ ' --osd-args "--memory 4G"'
+ )
+ if old not in src_sh:
+ raise SystemExit(
+ "run-cbt.sh cyanstore vstart block not found; cannot enable SeaStore"
+ )
+ wrap = ws / "run-cbt-seastore.sh"
+ wrap.write_text(src_sh.replace(old, new, 1))
+ wrap.chmod(0o755)
+ devs_file.write_text("\n".join(chosen) + "\n")
+ meta.append(f"seastore_devs={devs}")
+
+ (ws / "perf-store-tag").write_text(store_tag + "\n")
+ meta_path = ws / "perf-meta.txt"
+ meta_path.write_text(
+ "\n".join(
+ [
+ f"flavor={flavor}",
+ f"node={os.uname().nodename}",
+ f"store={store_tag}",
+ *meta,
+ "acceptable_near=0.10",
+ ]
+ )
+ + "\n"
+ )
+ print(meta_path.read_text())
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
--- /dev/null
+#!/bin/bash
+# Last resort when the agent is still responsive after a hard fail.
+set +e
+for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados; do
+ sudo pkill -9 -x "$p" || true
+done
+sudo reboot
--- /dev/null
+#!/bin/bash
+set -euxo pipefail
+
+# Env (from EnvInject / prior steps):
+# OSD_FLAVOR, SRC_DIR, WORKLOAD, ARCHIVE_BASEDIR
+: "${WORKSPACE:?}"
+: "${OSD_FLAVOR:?}"
+: "${SRC_DIR:?}"
+: "${WORKLOAD:?}"
+: "${ARCHIVE_KIND:?}"
+
+ARCHIVE_BASEDIR="${WORKSPACE}/${ARCHIVE_KIND}"
+BENCHMARK_YAML="${WORKSPACE}/perf-workloads/radosbench_4K_${WORKLOAD}.yaml"
+
+# Classic uses perf-basic write only; skip the crimson read workload.
+if test "${OSD_FLAVOR}" = "classic" -a "${WORKLOAD}" = "read"; then
+ echo "Skipping read workload for classic (qa/suites/perf-basic is write-only)"
+ exit 0
+fi
+
+cd "${WORKSPACE}/${SRC_DIR}"
+git submodule update --init --recursive
+store_tag=$(cat "${WORKSPACE}/perf-store-tag")
+# Layout: $ARCHIVE_BASEDIR/$WORKLOAD/$store_tag/<short-sha>/
+archive_dir="${ARCHIVE_BASEDIR}/${WORKLOAD}/${store_tag}/$(git rev-parse --short HEAD)"
+
+# An archive counts as complete only if the *measured* phase produced JSON
+# output. radosbench writes json_output.<proc>.<host> under the phase dir
+# (write/, rand/, seq/); the prefill warm-up writes it under prefill/. A run
+# SIGKILLed mid-read (or an old/partial baseline) can leave prefill/ with no
+# measured output; reusing/accepting that makes compare.py die later with
+# FileNotFoundError on the missing json_output. Ignore prefill/ so such
+# archives are re-run, not trusted.
+archive_is_complete() {
+ test -n "$(find "$1" -type f -name 'json_output.*' \
+ -not -path '*/prefill/*' -print -quit 2>/dev/null)"
+}
+
+if test -d "$archive_dir" && archive_is_complete "$archive_dir"; then
+ echo "Reusing existing complete CBT archive at $archive_dir"
+ exit 0
+fi
+rm -rf "$archive_dir"
+mkdir -p "$archive_dir"
+
+export NPROC
+NPROC=$(nproc)
+export FOR_MAKE_CHECK=true
+export WITH_CRIMSON=true
+
+if test -x build/bin/crimson-osd; then
+ echo "Reusing existing build at $PWD/build for workload=${WORKLOAD}"
+else
+ cmake_args="-DCMAKE_BUILD_TYPE=Release -DWITH_CRIMSON=ON -DWITH_TESTS=OFF -DWITH_SPDK=OFF -DWITH_DPDK=OFF -DWITH_QATLIB=OFF -DWITH_QATZIP=OFF ${CEPH_PERF_EXTRA_CMAKE_ARGS:-}"
+ timeout 7200 src/script/run-make.sh \
+ --cmake-args "$cmake_args" \
+ vstart-base crimson-osd
+fi
+
+cleanup_vstart_local() {
+ if test -f build/ceph.conf; then
+ ( cd build && ../src/stop.sh --crimson ) || true
+ ( cd build && ../src/stop.sh ) || true
+ fi
+ kill_cluster_procs
+}
+
+fail_and_drop_archive() {
+ local rc=$1
+ echo "ERROR: CBT step failed (rc=$rc); dropping partial archive $archive_dir" >&2
+ rm -rf "$archive_dir"
+ exit "$rc"
+}
+
+cleanup_all() {
+ cleanup_vstart_local || true
+ wipe_seastore_devs || true
+}
+trap cleanup_all EXIT
+
+cleanup_vstart_local
+wipe_seastore_devs
+sleep 2
+
+# shellcheck disable=SC1091
+. "${WORKSPACE}/gh-venv/bin/activate"
+
+run_cbt_sh=src/script/run-cbt.sh
+if test "${OSD_FLAVOR}" != "classic" -a -f "${WORKSPACE}/run-cbt-seastore.sh"; then
+ run_cbt_sh="${WORKSPACE}/run-cbt-seastore.sh"
+ echo "Using SeaStore-enabled run-cbt wrapper: $run_cbt_sh"
+fi
+
+set +e
+if test "${OSD_FLAVOR}" = "classic"; then
+ timeout --foreground --signal=KILL 3600 \
+ env PATH="${WORKSPACE}/gh-venv/bin:$PATH" \
+ "$run_cbt_sh" --build-dir "$PWD/build" --source-dir "$PWD" --cbt "$WORKSPACE/cbt" \
+ -a "$archive_dir" "$BENCHMARK_YAML" --classical
+ cbt_rc=$?
+else
+ timeout --foreground --signal=KILL 3600 \
+ env PATH="${WORKSPACE}/gh-venv/bin:$PATH" \
+ "$run_cbt_sh" --build-dir "$PWD/build" --source-dir "$PWD" --cbt "$WORKSPACE/cbt" \
+ -a "$archive_dir" "$BENCHMARK_YAML"
+ cbt_rc=$?
+fi
+set -e
+
+if test "$cbt_rc" -eq 124 -o "$cbt_rc" -eq 137; then
+ echo "ERROR: run-cbt.sh timed out after 3600s (rc=$cbt_rc)" >&2
+ fail_and_drop_archive 1
+fi
+if test "$cbt_rc" -ne 0; then
+ echo "ERROR: run-cbt.sh failed with rc=$cbt_rc" >&2
+ fail_and_drop_archive "$cbt_rc"
+fi
+if ! archive_is_complete "$archive_dir"; then
+ echo "ERROR: run-cbt.sh exited 0 but produced no measured json_output" \
+ "(incomplete archive, only prefill/ or empty): $archive_dir" >&2
+ fail_and_drop_archive 1
+fi
--- /dev/null
+#!/bin/bash
+# Continues setup after prepare-workloads.py has been written into $WORKSPACE.
+set -euxo pipefail
+
+python3 "${WORKSPACE}/prepare-workloads.py"
+
+if python3 -m venv "${WORKSPACE}/gh-venv"; then
+ :
+else
+ virtualenv -q --python python3 "${WORKSPACE}/gh-venv"
+fi
+# shellcheck disable=SC1091
+. "${WORKSPACE}/gh-venv/bin/activate"
+pip install -U pip setuptools wheel
+pip install cython
+pip install -r "${WORKSPACE}/cbt/requirements.txt" mdutils
+python3 -c "import yaml, lxml, matplotlib, mdutils, cython, setuptools"
+pip install git+https://github.com/ceph/githubcheck.git
+pip install 'pyjwt<2.11'
+
+echo "please hold tight..." | github-check \
+ --owner "${CHECK_REPO_OWNER}" \
+ --repo "${CHECK_REPO_NAME}" \
+ --pkey-file "${GITHUB_CHECK_PKEY_PEM}" \
+ --app-id "${CHECK_APP_ID}" \
+ --install-id "${CHECK_INSTALL_ID}" \
+ --name "${CHECK_NAME}" \
+ --sha "${ghprbActualCommit}" \
+ --external-id "${BUILD_ID}" \
+ --details-url "${BUILD_URL}" \
+ --status in_progress \
+ --title "${CHECK_NAME}" \
+ --summary running
--- /dev/null
+#!/bin/bash
+set -euxo pipefail
+
+: "${WORKSPACE:?}"
+: "${OSD_FLAVOR:?}"
+: "${CHECK_APP_ID:?}"
+: "${CHECK_INSTALL_ID:?}"
+: "${CHECK_NAME:?}"
+: "${CHECK_REPO_OWNER:?}"
+: "${CHECK_REPO_NAME:?}"
+
+cd "${WORKSPACE}/cbt"
+. /etc/os-release || ID=ubuntu
+case $ID in
+debian|ubuntu)
+ sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y \
+ python3-yaml python3-lxml python3-prettytable python3-matplotlib cython3 git
+ ;;
+centos|rhel|rocky|almalinux)
+ sudo dnf copr remove tchaikov/llvm-toolset-10 || true
+ sudo dnf module enable -y llvm-toolset || true
+ sudo dnf install -y llvm-toolset || true
+ sudo dnf install -y dnf-plugins-core
+ sudo dnf config-manager --set-enabled crb 2>/dev/null || \
+ sudo dnf config-manager --set-enabled powertools 2>/dev/null || \
+ sudo dnf config-manager --set-enabled PowerTools 2>/dev/null || true
+ if ! rpm -q epel-release >/dev/null 2>&1; then
+ major=${VERSION_ID%%.*}
+ sudo dnf install -y epel-release || \
+ sudo dnf install -y "https://dl.fedoraproject.org/pub/epel/epel-release-latest-${major}.noarch.rpm"
+ fi
+ sudo dnf install -y python3-pyyaml python3-lxml python3-prettytable \
+ python3-matplotlib python3-cython git
+ sudo dnf update -y libarchive || true
+ gcc_toolset_ver=13
+ if test -d /opt/rh/gcc-toolset-${gcc_toolset_ver}/root/lib/gcc/x86_64-redhat-linux/${gcc_toolset_ver}; then
+ sudo ln -sf /opt/rh/gcc-toolset-${gcc_toolset_ver}/root/lib/gcc/x86_64-redhat-linux/${gcc_toolset_ver} \
+ /usr/lib/gcc/x86_64-redhat-linux/${gcc_toolset_ver}
+ fi
+ ;;
+fedora)
+ sudo yum install -y python3-pyyaml python3-lxml python3-prettytable \
+ python3-matplotlib python3-cython clang git
+ ;;
+*)
+ echo "unknown distro: $ID"
+ exit 1
+ ;;
+esac
+
+kill_cluster_procs
+# Previous build may have left SeaStore devices dirty.
+wipe_seastore_devs
+
+mkdir -p "${WORKSPACE}/perf-workloads"
+rm -rf "${WORKSPACE}/seastore-imgs"
+rm -f "${WORKSPACE}/perf-workloads"/*.yaml \
+ "${WORKSPACE}/run-cbt-seastore.sh" \
+ "${WORKSPACE}/seastore-devs.txt" \
+ "${WORKSPACE}/prepare-workloads.py"
+: > "${WORKSPACE}/seastore-devs.txt"
clean:
after: true
-- builder:
- name: run-cbt
- builders:
- - shell: |
- #!/bin/bash
- set -euxo pipefail
- cd {src-dir}
- git submodule update --init --recursive
- # Per-workload archives so read/write do not clobber each other, and so
- # a prior read-only cache is not mistaken for a completed write run.
- # Layout: {archive-basedir}/{workload}/<short-sha>/
- archive_dir={archive-basedir}/{workload}/$(git rev-parse --short HEAD)
- if test -d "$archive_dir" && test -n "$(find "$archive_dir" -mindepth 1 -print -quit 2>/dev/null)" ; then
- exit 0
- fi
- mkdir -p "$archive_dir"
- export NPROC=$(nproc)
- export FOR_MAKE_CHECK=true
- export WITH_CRIMSON=true
- # Same Crimson-inclusive build as ceph-pull-requests (WITH_CRIMSON=true).
- # Classic vs crimson perf is selected at CBT runtime via run-cbt.sh.
- # Compiler: run-make.sh installs clang-19 on Jenkins (src/script/run-make.sh
- # get_llvm) and selects it via discover_compiler — same as make-check nodes.
- # Reuse build/ when running a second workload (write after read) in this job.
- if test -x build/bin/crimson-osd ; then
- echo "Reusing existing build at $PWD/build for workload={workload}"
- else
- cmake_args="-DCMAKE_BUILD_TYPE=Release -DWITH_CRIMSON=ON -DWITH_TESTS=OFF -DWITH_SPDK=OFF -DWITH_DPDK=OFF -DWITH_QATLIB=OFF -DWITH_QATZIP=OFF ${{CEPH_PERF_EXTRA_CMAKE_ARGS:-}}"
- timeout 7200 src/script/run-make.sh \
- --cmake-args "$cmake_args" \
- vstart-base crimson-osd
- fi
- # stop.sh must run from the build dir (sets CEPH_BIN=bin, conf=$PWD/ceph.conf).
- # Calling it from the source tree yields: /ceph-conf: No such file or directory.
- # pkill -x avoids matching this script (a -f 'rados bench' pattern self-kills).
- # Double braces: JJB str.format() on macro params like {{src-dir}}.
- cleanup_vstart() {{
- if test -f build/ceph.conf ; then
- ( cd build && ../src/stop.sh --crimson ) || true
- ( cd build && ../src/stop.sh ) || true
- fi
- for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados ; do
- sudo pkill -9 -x "$p" || true
- done
- }}
- # Leftover daemons from a previous hung run can wedge radosbench forever.
- cleanup_vstart
- sleep 2
- . ${{WORKSPACE}}/gh-venv/bin/activate
- # CBT itself has no set -e / timeout; radosbench prefill has hung for days on
- # bath01. Bound the whole run-cbt.sh (vstart + bench + stop) to 1h.
- # SIGKILL means run-cbt.sh cannot run its own stop.sh, so we clean up after.
- set +e
- if test {osd-flavor} = "classic" ; then
- timeout --foreground --signal=KILL 3600 \
- env PATH="${{WORKSPACE}}/gh-venv/bin:$PATH" \
- src/script/run-cbt.sh --build-dir $PWD/build --source-dir $PWD --cbt $WORKSPACE/cbt -a $archive_dir {benchmark-yaml} --classical
- cbt_rc=$?
- else
- timeout --foreground --signal=KILL 3600 \
- env PATH="${{WORKSPACE}}/gh-venv/bin:$PATH" \
- src/script/run-cbt.sh --build-dir $PWD/build --source-dir $PWD --cbt $WORKSPACE/cbt -a $archive_dir {benchmark-yaml}
- cbt_rc=$?
- fi
- set -e
- cleanup_vstart
- if test "$cbt_rc" -eq 124 -o "$cbt_rc" -eq 137 ; then
- echo "ERROR: run-cbt.sh timed out after 3600s (rc=$cbt_rc); likely hung radosbench/Crimson on this node" >&2
- exit 1
- fi
- if test "$cbt_rc" -ne 0 ; then
- echo "ERROR: run-cbt.sh failed with rc=$cbt_rc" >&2
- exit "$cbt_rc"
- fi
- # run-cbt.sh is /bin/sh without set -e; a failed cbt.py can still exit 0.
- test -n "$(find "$archive_dir" -mindepth 1 -print -quit 2>/dev/null)"
-
-- builder:
- name: compare-cbt-results
- builders:
- - shell: |
- #!/bin/bash
- set -euxo pipefail
- cd ${{WORKSPACE}}/{src-dir-main}
- sha_main=$(git rev-parse --short HEAD)
- cd ${{WORKSPACE}}/{src-dir-pr}
- sha_pr=$(git rev-parse --short HEAD)
- . ${{WORKSPACE}}/gh-venv/bin/activate
- report=${{WORKSPACE}}/report.md
- : > "$report"
- result=success
-
- # compare.py has #!/usr/bin/python3; must invoke via venv python3.
- # Run once per workload and merge into one GitHub-check markdown report
- # with clear section headings (read vs write).
- compare_workload() {{
- local label=$1
- local workload=$2
- local archive_main={archive-main}/$workload/$sha_main
- local archive_pr={archive-pr}/$workload/$sha_pr
- local partial=${{WORKSPACE}}/report-$workload.md
- rm -f "$partial"
- echo "## ${{label}}" >> "$report"
- echo >> "$report"
- if python3 ${{WORKSPACE}}/cbt/compare.py -v \
- -a "$archive_pr" \
- -b "$archive_main" \
- --output "$partial"; then
- :
- else
- result=failure
- if test ! -s "$partial"; then
- echo "cbt/compare.py failed for ${{label}}; see Jenkins console." > "$partial"
- fi
- fi
- cat "$partial" >> "$report"
- echo >> "$report"
- }}
-
- compare_workload "4K random read" read
- compare_workload "4K write" write
-
- github-check \
- --owner {check-repo-owner} \
- --repo {check-repo-name} \
- --pkey-file ${{GITHUB_CHECK_PKEY_PEM}} \
- --app-id "{check-app-id}" \
- --install-id "{check-install-id}" \
- --name {check-name} \
- --sha ${{ghprbActualCommit}} \
- --external-id ${{BUILD_ID}} \
- --details-url ${{BUILD_URL}} \
- --status completed --conclusion ${{result}} \
- --title {check-name} \
- --summary ${{result}} \
- --text $report
-
- job-template:
name: 'ceph-perf-{osd-flavor}'
project-type: freestyle
defaults: global
- concurrent: true
+ # Serialize within this job; throttle category also serializes classic vs crimson.
+ concurrent: false
node: performance
display-name: 'ceph: {osd-flavor} perf test'
- description: 'CBT performance regression check ({osd-flavor} flavor): builds ceph-pr and ceph-main, runs the checked-in radosbench 4K read and 4K write workloads, and compares results via cbt/compare.py. The GitHub check fails when compare.py reports a statistically worse result, too much noise, or errors. Trigger with comment: jenkins test {osd-flavor} perf'
+ description: 'CBT performance regression check ({osd-flavor} flavor): builds ceph-pr and ceph-main, runs flavor-specific radosbench workloads, and compares results via cbt/compare.py. Classic uses qa/suites/perf-basic write; crimson always uses SeaStore (spare NVMes when available, else workspace sparse images). Trigger with comment: jenkins test {osd-flavor} perf'
quiet-period: 5
block-downstream: false
block-upstream: false
properties:
- build-discarder:
- days-to-keep: 15
+ days-to-keep: 60
num-to-keep: 300
- artifact-days-to-keep: -1
- artifact-num-to-keep: -1
+ artifact-days-to-keep: 60
+ artifact-num-to-keep: 300
- github:
url: https://github.com/ceph/ceph/
- rebuild:
auto-rebuild: true
+ - throttle:
+ option: category
+ max-total: 1
+ max-per-node: 1
+ categories:
+ - ceph-perf
parameters:
- string:
allow-whitelist-orgs-as-admins: true
org-list:
- ceph
- # Comment-only: reliable path. Auto-build on every push with the
- # performance label was confusing when the sole performance node was
- # wedged (builds queued forever and looked like "trigger never works").
- # Use an explicit comment for each flavor.
- # ghprb matches trigger-phrase as a Java regex (case-insensitive).
trigger-phrase: '(?i)jenkins\s+test\s+{osd-flavor}\s+perf\b'
skip-build-phrase: '(?i)^jenkins do not test.*'
only-trigger-phrase: true
- cbt
builders:
- - shell: |
- #!/bin/bash
- set -euxo pipefail
- cd ${{WORKSPACE}}/cbt
- . /etc/os-release || ID=ubuntu
- case $ID in
- debian|ubuntu)
- sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y python3-yaml python3-lxml python3-prettytable python3-matplotlib cython3 git
- ;;
- centos|rhel|rocky|almalinux)
- # llvm/gcc-toolset are EL8/9-oriented; ignore on Rocky 10+ where modules differ.
- sudo dnf copr remove tchaikov/llvm-toolset-10 || true
- sudo dnf module enable -y llvm-toolset || true
- sudo dnf install -y llvm-toolset || true
- # Repos needed on EL for CBT Python RPMs:
- # - CRB/PowerTools: python3-cython (EL9/10)
- # - EPEL: python3-matplotlib (EL10 AppStream does not ship it)
- # config-manager comes from dnf-plugins-core; without it the
- # enable steps below would no-op and python3-cython would fail.
- sudo dnf install -y dnf-plugins-core
- sudo dnf config-manager --set-enabled crb 2>/dev/null || \
- sudo dnf config-manager --set-enabled powertools 2>/dev/null || \
- sudo dnf config-manager --set-enabled PowerTools 2>/dev/null || true
- if ! rpm -q epel-release >/dev/null 2>&1; then
- major=${{VERSION_ID%%.*}}
- sudo dnf install -y epel-release || \
- sudo dnf install -y "https://dl.fedoraproject.org/pub/epel/epel-release-latest-${{major}}.noarch.rpm"
- fi
- # Prefer lowercase python3-cython (EL10+); capital-C still works via Provides
- # when CRB is enabled, but keep the modern name as the primary request.
- sudo dnf install -y python3-pyyaml python3-lxml python3-prettytable \
- python3-matplotlib python3-cython git
- sudo dnf update -y libarchive || true
- gcc_toolset_ver=13
- # so clang is able to find gcc-toolset-${{gcc_toolset_ver}} which is listed as a
- # BuildRequires in ceph.spec.in, and it is installed by `run-make.sh`.
- # clang searches for GCC in a bunch of well known places:
- # see https://github.com/llvm-mirror/clang/blob/main/lib/Driver/ToolChains/Gnu.cpp
- if test -d /opt/rh/gcc-toolset-${{gcc_toolset_ver}}/root/lib/gcc/x86_64-redhat-linux/${{gcc_toolset_ver}} ; then
- sudo ln -sf /opt/rh/gcc-toolset-${{gcc_toolset_ver}}/root/lib/gcc/x86_64-redhat-linux/${{gcc_toolset_ver}} \
- /usr/lib/gcc/x86_64-redhat-linux/${{gcc_toolset_ver}}
- fi
- ;;
- fedora)
- sudo yum install -y python3-pyyaml python3-lxml python3-prettytable \
- python3-matplotlib python3-cython clang git
- ;;
- *)
- echo "unknown distro: $ID"
- exit 1
- ;;
- esac
- # Clear leftover cluster processes from a previous hung/aborted run on
- # this performance node before we spend hours rebuilding.
- for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados ; do
- sudo pkill -9 -x "$p" || true
- done
- # Rocky/EL10 does not ship the virtualenv CLI; prefer stdlib venv and
- # fall back if ensurepip/venv support is missing (if-test avoids set -e).
- if python3 -m venv "${{WORKSPACE}}/gh-venv"; then
- :
- else
- virtualenv -q --python python3 "${{WORKSPACE}}/gh-venv"
- fi
- . ${{WORKSPACE}}/gh-venv/bin/activate
- pip install -U pip setuptools wheel
- pip install cython
- # cbt imports mdutils; not listed in ceph/cbt master requirements.txt
- pip install -r ${{WORKSPACE}}/cbt/requirements.txt mdutils
- python3 -c "import yaml, lxml, matplotlib, mdutils, cython, setuptools"
- pip install git+https://github.com/ceph/githubcheck.git
- # github3.py 4.0.1 passes app_id as int; PyJWT 2.11+ requires iss to be str.
- pip install 'pyjwt<2.11'
- echo "please hold tight..." | github-check \
- --owner {check-repo-owner} \
- --repo {check-repo-name} \
- --pkey-file ${{GITHUB_CHECK_PKEY_PEM}} \
- --app-id "{check-app-id}" \
- --install-id "{check-install-id}" \
- --name {check-name} \
- --sha ${{ghprbActualCommit}} \
- --external-id ${{BUILD_ID}} \
- --details-url ${{BUILD_URL}} \
- --status in_progress \
- --title {check-name} \
- --summary running
-
- - run-cbt:
- osd-flavor: '{osd-flavor}'
- src-dir: "ceph-main"
- workload: "read"
- benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_read.yaml"
- # ideally cbt-results should be persited across jobs, so the test result can be reused
- archive-basedir: "$WORKSPACE/cbt-results"
- - run-cbt:
- osd-flavor: '{osd-flavor}'
- src-dir: "ceph-main"
- workload: "write"
- benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_write.yaml"
- archive-basedir: "$WORKSPACE/cbt-results"
- - run-cbt:
- osd-flavor: '{osd-flavor}'
- src-dir: "ceph-pr"
- workload: "read"
- # use the same benchmark definition as ceph-main for apples-to-apples
- # comparison and to avoid PR-side benchmark drift.
- benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_read.yaml"
- # use the basedir of git checkout, so it can be wiped
- archive-basedir: "$WORKSPACE/ceph-pr"
- - run-cbt:
- osd-flavor: '{osd-flavor}'
- src-dir: "ceph-pr"
- workload: "write"
- benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_write.yaml"
- archive-basedir: "$WORKSPACE/ceph-pr"
- - compare-cbt-results:
- src-dir-main: "ceph-main"
- archive-main: "$WORKSPACE/cbt-results"
- src-dir-pr: "ceph-pr"
- archive-pr: "$WORKSPACE/ceph-pr"
- check-app-id: '{check-app-id}'
- check-install-id: '{check-install-id}'
- check-name: '{check-name}'
- check-repo-owner: '{check-repo-owner}'
- check-repo-name: '{check-repo-name}'
+ - inject:
+ properties-content: |
+ OSD_FLAVOR={osd-flavor}
+ CHECK_APP_ID={check-app-id}
+ CHECK_INSTALL_ID={check-install-id}
+ CHECK_NAME={check-name}
+ CHECK_REPO_OWNER={check-repo-owner}
+ CHECK_REPO_NAME={check-repo-name}
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/common.sh
+ - ../../build/setup-pre.sh
+ - ../../build/prepare-workloads-header.sh
+ - ../../build/prepare-workloads.py
+ - ../../build/prepare-workloads-footer.sh
+ - ../../build/setup-post.sh
+
+ # ideally cbt-results should be persisted across jobs so main baselines can be reused
+ - inject:
+ properties-content: |
+ SRC_DIR=ceph-main
+ WORKLOAD=read
+ ARCHIVE_KIND=cbt-results
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/common.sh
+ - ../../build/run-cbt
+ - inject:
+ properties-content: |
+ SRC_DIR=ceph-main
+ WORKLOAD=write
+ ARCHIVE_KIND=cbt-results
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/common.sh
+ - ../../build/run-cbt
+ - inject:
+ properties-content: |
+ SRC_DIR=ceph-pr
+ WORKLOAD=read
+ ARCHIVE_KIND=ceph-pr
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/common.sh
+ - ../../build/run-cbt
+ - inject:
+ properties-content: |
+ SRC_DIR=ceph-pr
+ WORKLOAD=write
+ ARCHIVE_KIND=ceph-pr
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/common.sh
+ - ../../build/run-cbt
+
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/common.sh
+ - ../../build/compare
publishers:
+ - archive:
+ artifacts: 'report.md,report-*.md,perf-meta.txt,perf-store-tag,seastore-devs.txt,perf-workloads/**,cbt-results/**/*,ceph-pr/read/**/*,ceph-pr/write/**/*'
+ allow-empty: true
+ latest-only: false
- postbuildscript:
builders:
+ - role: SLAVE
+ build-on:
+ - SUCCESS
+ - UNSTABLE
+ - FAILURE
+ - ABORTED
+ build-steps:
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/common.sh
+ - ../../build/cleanup
- role: SLAVE
build-on:
- FAILURE
- ABORTED
build-steps:
- - shell: |
- #!/bin/bash
- # Kill hung CBT/Ceph first so reboot is not the only recovery path
- # when the agent is still responsive enough to run this step.
- # Use -x so pkill does not match this script line.
- for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados ; do
- sudo pkill -9 -x "$p" || true
- done
- sudo reboot
+ - shell:
+ !include-raw-verbatim:
+ - ../../build/reboot
wrappers:
- timeout:
- # Two Crimson builds (≤2h each) + four CBT runs (read+write × main+PR,
- # ≤1h each) + setup/compare. Absolute ceiling so a wedged radosbench
- # cannot hold the node for days.
+ # Two Crimson builds (≤2h each) + CBT runs + setup/compare.
timeout: 480
timeout-var: 'BUILD_TIMEOUT'
fail: true
projects / ceph-build.git / commitdiff