From: Kautilya Tripathi Date: Tue, 28 Jul 2026 03:58:06 +0000 (+0530) Subject: ceph-perf: harden PR jobs and always run crimson on SeaStore X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=f7e198df45673648e7eec638f7bd0c91f5635fe6;p=ceph-build.git ceph-perf: harden PR jobs and always run crimson on SeaStore Split the freestyle job into build/ scripts included via JJB, keep 60-day logs/artifacts, and serialize classic vs crimson with concurrent:false plus the ceph-perf throttle. Classic uses perf-basic 4K write only. Crimson always uses SeaStore (3 largest spare NVMes, else workspace sparse images never CyanStore). Inject ~10% compare tolerance, fail Jenkins when compare.py rejects a regression, drop partial CBT archives, and wipe SeaStore targets safely on cleanup. Signed-off-by: Kautilya Tripathi --- diff --git a/ceph-perf-pull-requests/README.rst b/ceph-perf-pull-requests/README.rst index f28bc3e10..9f792b3c2 100644 --- a/ceph-perf-pull-requests/README.rst +++ b/ceph-perf-pull-requests/README.rst @@ -4,27 +4,54 @@ ceph-perf-pull-requests Jenkins Job Builder definitions for CBT performance regression checks on Ceph pull requests. +Layout +------ + +:: + + ceph-perf-pull-requests/ + config/definitions/ceph-perf-pull-requests.yml # JJB job wiring only + build/ + common.sh # shared wipe / process helpers + setup-pre.sh # packages, pre-wipe + prepare-workloads.py # classic vs crimson YAML + SeaStore device pick + prepare-workloads-*.sh # embed the .py into the agent workspace + setup-post.sh # venv, github check in_progress + run-cbt # one CBT archive run (env: SRC_DIR, WORKLOAD, ...) + compare # compare.py + github check completed + cleanup # postbuild wipe + reboot # failure/abort reboot + README.rst + +Shell steps are inlined into the Jenkins job at ``jenkins-jobs update`` time via +``!include-raw-verbatim`` (same pattern as ``ceph-volume-cephadm-prs``). Template +parameters are passed with EnvInject (``OSD_FLAVOR``, check app ids, etc.). + Jobs ---- The ``ceph-perf`` project generates two freestyle jobs: -- ``ceph-perf-classic`` — same Crimson-inclusive build as ``ceph-pull-requests``, - CBT run with ``run-cbt.sh --classical`` (classical ``ceph-osd``) -- ``ceph-perf-crimson`` — same build, CBT run with default ``run-cbt.sh`` - (``crimson-osd`` via ``vstart.sh --crimson``) +- ``ceph-perf-classic`` — classical ``ceph-osd`` via ``run-cbt.sh --classical``, + workload from ``qa/suites/perf-basic/workloads/radosbench_4K_write.yaml`` +- ``ceph-perf-crimson`` — ``crimson-osd`` via ``run-cbt.sh``, workloads from + ``src/test/crimson/cbt/radosbench_4K_{read,write}.yaml``. **Always SeaStore** + (never CyanStore). Prefers the **3 largest** unmounted NVMes when present + (pilot node ``o02``, see https://tracker.ceph.com/issues/78071 — so three + 7.3T drives, never OS ``sda`` / small mounted disks). If fewer than 3 spare + NVMes exist, creates three ~32GiB sparse images under + ``$WORKSPACE/seastore-imgs/`` and still runs SeaStore. Both run on ``performance`` nodes, build ``ceph-main`` and the PR merge ref (``WITH_CRIMSON=ON``, ``vstart-base`` + ``crimson-osd``; compiler selection via -``run-make.sh`` / ``discover_compiler``, same idea as make-check), then execute -**both** workloads from ``ceph-main``: - -- ``src/test/crimson/cbt/radosbench_4K_read.yaml`` — 4K random read -- ``src/test/crimson/cbt/radosbench_4K_write.yaml`` — 4K write +``run-make.sh`` / ``discover_compiler``, same idea as make-check). Only one perf +job runs at a time (``concurrent: false`` + shared throttle category ``ceph-perf``) +so classic and crimson cannot load the same machine together. The ``ceph-perf`` +throttle category must exist in Jenkins (Throttle Concurrent Builds plugin). Results are compared with ``cbt/compare.py``. A GitHub check -(``perf-test-{osd-flavor}``) gets a combined markdown report with separate -sections for read and write. The check fails if **either** workload regresses. +(``perf-test-{osd-flavor}``) gets a markdown report that includes node name, +store backend, and SHAs. The check fails if any compared workload regresses. Triggering ---------- @@ -45,41 +72,35 @@ how this job fits into CI. What the job does ----------------- -For each of ``ceph-main`` and the PR merge ref the job: - -1. Builds Ceph once (the second workload reuses ``build/``). -2. Runs CBT for **read**, archiving under ``{basedir}/read//``. -3. Runs CBT for **write**, archiving under ``{basedir}/write//``. - -Then ``compare.py`` runs twice (read archive pair, write archive pair) and the -markdown outputs are merged into one GitHub check body. Example shape: - -**4K random read** — ``all 20 tests passed`` - -============== ==================== ========== ======== ======== -run metric baseline result accepted -============== ==================== ========== ======== ======== -prefill/host/0 bandwidth … … -prefill/host/0 iops_avg … … -… … … … -rand/host/1 latency_avg … … -============== ==================== ========== ======== ======== - -**4K write** — ``all 10 tests passed`` - -============== ==================== ========== ======== ======== -run metric baseline result accepted -============== ==================== ========== ======== ======== -write/host/0 bandwidth … … -write/host/0 iops_avg … … -… … … … -write/host/1 latency_avg … … -============== ==================== ========== ======== ======== - -Blank ``accepted`` means OK; ``:x:`` / ❌ means the metric failed the rule. - -Benchmark YAMLs are always taken from **ceph-main** (not the PR tree) so both -sides use the same workload definition and acceptance rules. +1. Prepares flavor-specific workload YAML(s) under ``$WORKSPACE/perf-workloads/`` + (adds ``acceptable:`` with ~10% near-tolerance for compare.py). It also adds a + short-term **warm-up** ``prefill_time`` (>= the measured window, min 30s) so + the measurement does not run on a cold cluster: radosbench has no dedicated + warm-up phase, and a cold/absent prefill adds variance (and can leave a read + phase with no output). This applies to the crimson **read** workload and the + classic **perf-basic write** workload (crimson write is left as the checked-in + file). The prefill pass lands in ``prefill/`` and is not compared. The durable + fix belongs in CBT / the checked-in Ceph CBT YAML. +2. For each of ``ceph-main`` and the PR merge ref: build once (reuse ``build/`` + for a second workload), run CBT, archive under + ``{basedir}/{workload}/{store_tag}//``. +3. Compare archives and post the GitHub check. +4. Archive reports + CBT results as Jenkins artifacts (kept ~60 days with the + build console), so “View more details” keep working after the fact. + +On failure/success the job drops partial CBT archives (so they are never reused), +stops cluster processes, and wipes the job’s SeaStore NVMes (``wipefs`` + leading +zeros; nvme-only, never if mounted). Compare regressions fail both the GitHub +check **and** the Jenkins build. + +An archive is treated as complete (reusable, and accepted after a run) only if +the **measured** phase produced ``json_output.*`` (``write/``, ``rand/`` or +``seq/`` — ``prefill/`` alone does not count). This prevents reusing a run that +was SIGKILLed mid-read, which otherwise makes ``compare.py`` fail with +``FileNotFoundError`` on a missing ``json_output``. + +``store_tag`` is ``classic`` or ``seastore`` so backends never reuse each +other’s baselines. Developer guide: reading the results ------------------------------------ @@ -89,7 +110,7 @@ This section explains the GitHub check table in plain terms. What question the check answers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**Did this PR make the fixed 4K CBT workloads worse than current ``main`` on the +**Did this PR make the fixed CBT workload(s) worse than current ``main`` on the performance node, beyond the allowed tolerance?** It is a regression smoke signal, not a full performance study and not a @@ -98,103 +119,67 @@ functional correctness test. Where things live ~~~~~~~~~~~~~~~~~ -================ ================================= ============================================ -Piece Location Role -================ ================================= ============================================ -Workload + rules Ceph ``src/test/crimson/cbt/`` ``radosbench_4K_{read,write}.yaml`` -Job wiring This directory (JJB YAML) Build, run CBT, post GitHub check -CBT runner ``https://github.com/ceph/cbt`` Runs ``rados bench``, collects metrics -Compare ``cbt/compare.py`` PR archive vs main archive → table -GitHub check ``perf-test-classic`` / Report on the pull request - ``perf-test-crimson`` -================ ================================= ============================================ +================== =================================== ============================================ +Piece Location Role +================== =================================== ============================================ +Classic workload Ceph ``qa/suites/perf-basic/`` ``radosbench_4K_write.yaml`` +Crimson workload Ceph ``src/test/crimson/cbt/`` ``radosbench_4K_read/write.yaml`` +Job wiring This directory (JJB YAML) Build, run CBT, post GitHub check +CBT runner ``https://github.com/ceph/cbt`` Runs ``rados bench``, collects metrics +Compare ``cbt/compare.py`` PR archive vs main archive → table +GitHub check ``perf-test-classic`` / crimson Report on the pull request +Jenkins artifacts Build artifacts + console Baseline/PR archives, report, meta +================== =================================== ============================================ Useful links: -- Read workload: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_read.yaml -- Write workload: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_write.yaml +- Classic write: https://github.com/ceph/ceph/blob/main/qa/suites/perf-basic/workloads/radosbench_4K_write.yaml +- Crimson read: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_read.yaml +- Crimson write: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_write.yaml - Metric collectors: https://github.com/ceph/cbt/blob/main/benchmark/radosbench.py - Acceptance evaluation: https://github.com/ceph/cbt/blob/main/benchmark/benchmark.py - Compare / report: https://github.com/ceph/cbt/blob/main/compare.py +- Perf pilot node (SeaStore NVMe): https://tracker.ceph.com/issues/78071 What each workload runs ~~~~~~~~~~~~~~~~~~~~~~~ -**4K random read** (``radosbench_4K_read.yaml``): +**Classic — 4K write** (``qa/suites/perf-basic/.../radosbench_4K_write.yaml``): -- Small local cluster (3 OSDs, replicated pool). -- Prefill write (~3s) so there is data to read. -- Random 4K read (~30s). -- Two concurrent client processes (``concurrent_procs: 2``). +- Classical ``ceph-osd`` (``run-cbt.sh --classical``). +- Write-only 4K bench (``time: 300``, ``concurrent_ops: 4``, 256 PGs). +- ``acceptable:`` is injected by the job (perf-basic upstream has none). -**4K write** (``radosbench_4K_write.yaml``): +**Crimson — 4K random read** (``radosbench_4K_read.yaml``): -- Same cluster shape. -- Write-only 4K bench (~3s); no separate prefill/read phase. +- Prefill write (~3s) then random 4K read (~30s), ``concurrent_ops: 16``, 128 PGs. - Two concurrent client processes. -Why the row counts are 20 and 10 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Each compared cell is one **(phase × client process × metric)** tuple. - -Five metrics: - -=================== ================================================= ============ -Metric Meaning Better means -=================== ================================================= ============ -bandwidth Throughput Higher -iops_avg Average IOPS Higher -iops_stddev IOPS variance / noise Lower -latency_avg Average latency Lower -cpu_cycles_per_op CPU cycles per op (from ``perf`` if collected) Lower -=================== ================================================= ============ - -- Read: phases ``prefill`` + ``rand``, 2 clients, 5 metrics → **20 rows** - (names like ``prefill/toko02/0``, ``rand/toko02/1``). -- Write: phase ``write`` only, 2 clients, 5 metrics → **10 rows** - (names like ``write/toko02/0``). - -``cpu_cycles_per_op`` may show ``0.0`` when ``perf`` data was not collected; that -usually passes as a no-op comparison. +**Crimson — 4K write** (``radosbench_4K_write.yaml``): -How to read the GitHub table columns -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- Write-only 4K bench (~3s), same cluster shape. -======== ========================================================= -Column Meaning -======== ========================================================= -run Phase / machine / client process (``rand/host/0``) -metric Which number -baseline Value from **current ceph** ``main`` -result Value from **this PR** -accepted Blank = OK; ``:x:`` / ❌ = failed the acceptance rule -======== ========================================================= - -Pass / fail rules (``acceptable:`` in the YAML) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:: +Pass / fail rules +~~~~~~~~~~~~~~~~~ - bandwidth: (or (greater) (near 0.05)) # ≥ baseline, or within ~5% - iops_avg: (or (greater) (near 0.05)) - latency_avg: (or (less) (near 0.05)) # ≤ baseline, or within ~5% - iops_stddev: (or (less) (near 2.00)) # ≤ baseline, or within ~2× - cpu_cycles_per_op:(or (less) (near 0.05)) +Jenkins workspace copies use ~**10%** near-tolerance (wider than the 5% baked into +the crimson CBT YAML upstream) to reduce single-shot false positives:: -Small regressions up to about 5% on throughput/latency are allowed; -noise (``iops_stddev``) may grow up to about 2× baseline. + bandwidth: (or (greater) (near 0.10)) + iops_avg: (or (greater) (near 0.10)) + latency_avg: (or (less) (near 0.10)) + iops_stddev: (or (less) (near 2.00)) + cpu_cycles_per_op:(or (less) (near 0.10)) Jenkins console log order (easy to misread) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ CBT verbose logs look like:: - rand/host/1: bandwidth: (or (greater) (near 0.05)):: 203.5/220.4 => rejected + rand/host/1: bandwidth: (or (greater) (near 0.10)):: 203.5/220.4 => rejected The fraction is **``result/baseline``** (PR first, main second), **not** -baseline/result. In the example above the PR is slower than main, so rejection -is correct. Prefer the GitHub table headers when in doubt. +baseline/result. Prefer the GitHub table headers when in doubt. What the baseline is ~~~~~~~~~~~~~~~~~~~~ @@ -205,31 +190,21 @@ Each job: 1. Checks out current ``origin/main`` into ``ceph-main``. 2. Builds and runs CBT; archives under - ``$WORKSPACE/cbt-results/{read,write}//``. + ``$WORKSPACE/cbt-results/{workload}/{store_tag}//``. 3. Checks out the PR merge ref into ``ceph-pr`` and does the same under - ``$WORKSPACE/ceph-pr/{read,write}//``. + ``$WORKSPACE/ceph-pr/{workload}/{store_tag}//``. 4. Compares PR archives to main archives. -If a main short-SHA archive already exists and is non-empty on that node, the -main CBT run for that workload is **reused** (not rebuilt). When ``main`` moves, -you get a new baseline directory — numbers can look very different from an -earlier run on the same PR. Node noise can also move results on the same SHA. - -How to add or change metrics -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Edit the ``acceptable:`` block in the Ceph workload YAML(s) if the metric is - already collected by CBT. -2. To collect a **new** measurement, implement a getter in - ``cbt/benchmark/radosbench.py`` and wire evaluation in - ``cbt/benchmark/benchmark.py``. -3. ``compare.py`` only reports; it does not define metrics. +If a matching **baseline** archive already exists and is **complete** (its +measured phase has ``json_output.*``) on that node, the main CBT run is +**reused** (PR-side archives are rebuilt each run because the ``ceph-pr`` +checkout wipes its workspace). When ``main`` moves, or the store backend +changes (new ``store_tag``), you get a new baseline directory. Teuthology YAML translation --------------------------- -Benchmark definitions under ``src/test/crimson/cbt/`` use teuthology's -``tasks`` format. ``run-cbt.sh`` calls ``t2c.py`` to extract the ``cbt`` task -and emit a CBT configuration. That translator (including ``yaml.safe_load`` for -input parsing) lives in the Ceph repository with unit tests in -``test_t2c.py``; the Jenkins job does not patch it at build time. +Benchmark definitions use teuthology's ``tasks`` format. ``run-cbt.sh`` calls +``t2c.py`` to extract the ``cbt`` task and emit a CBT configuration. That +translator lives in the Ceph repository with unit tests in ``test_t2c.py``; the +Jenkins job does not patch it at build time. diff --git a/ceph-perf-pull-requests/build/cleanup b/ceph-perf-pull-requests/build/cleanup new file mode 100644 index 000000000..9734cce17 --- /dev/null +++ b/ceph-perf-pull-requests/build/cleanup @@ -0,0 +1,8 @@ +#!/bin/bash +set -euo pipefail +: "${WORKSPACE:?}" +set +e +# Postbuild cleanup on SUCCESS/UNSTABLE/FAILURE/ABORTED. +# Best-effort after the WORKSPACE guard (wipe/pkill must not fail the publisher). +kill_cluster_procs +wipe_seastore_devs diff --git a/ceph-perf-pull-requests/build/common.sh b/ceph-perf-pull-requests/build/common.sh new file mode 100644 index 000000000..78faf9f6c --- /dev/null +++ b/ceph-perf-pull-requests/build/common.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# Shared helpers for ceph-perf jobs. +# Included (concatenated) before setup / run-cbt / compare / cleanup via JJB. + +kill_cluster_procs() { + local p + for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados; do + sudo pkill -9 -x "$p" || true + done +} + +# Wipe SeaStore targets listed in $WORKSPACE/seastore-devs.txt: +# - /dev/nvme*: wipefs + leading zeros (never if mounted) +# - $WORKSPACE/seastore-imgs/* sparse images: delete +# Anything else is refused. +wipe_seastore_devs() { + # Require WORKSPACE so we never operate on /seastore-imgs or similar. + : "${WORKSPACE:?}" + local devs_file="${WORKSPACE}/seastore-devs.txt" + local img_dir="${WORKSPACE}/seastore-imgs" + local dev mountpoints + if test ! -s "$devs_file"; then + rm -rf "$img_dir" + return 0 + fi + while IFS= read -r dev || test -n "$dev"; do + test -n "$dev" || continue + case "$dev" in + /dev/nvme*) + if test ! -b "$dev"; then + echo "skip wipe, not a block device: $dev" >&2 + continue + fi + if ! command -v lsblk >/dev/null 2>&1; then + echo "REFUSING to wipe (lsblk not found to verify mounts): $dev" >&2 + continue + fi + # Fail closed: if lsblk cannot report mount state, do not wipe. + if ! mountpoints=$(lsblk -n -o MOUNTPOINT "$dev" 2>/dev/null); then + echo "REFUSING to wipe (lsblk failed to verify mounts): $dev" >&2 + continue + fi + if printf '%s\n' "$mountpoints" | grep -q '[^[:space:]]'; then + echo "REFUSING to wipe mounted device: $dev" >&2 + continue + fi + echo "Wiping SeaStore device $dev" + sudo wipefs -a "$dev" || true + sudo dd if=/dev/zero of="$dev" bs=1M count=100 status=none conv=fsync || true + ;; + "${WORKSPACE}/seastore-imgs"/*) + echo "Removing SeaStore sparse image $dev" + rm -f "$dev" || true + ;; + *) + echo "REFUSING to wipe unexpected SeaStore path: $dev" >&2 + continue + ;; + esac + done < "$devs_file" + rm -rf "$img_dir" +} + +cleanup_vstart() { + # Prefer stopping from the active tree if present. + local tree + for tree in "${SRC_DIR:-}" ceph-pr ceph-main; do + test -n "$tree" || continue + if test -f "${WORKSPACE}/${tree}/build/ceph.conf"; then + ( cd "${WORKSPACE}/${tree}/build" && ../src/stop.sh --crimson ) || true + ( cd "${WORKSPACE}/${tree}/build" && ../src/stop.sh ) || true + fi + done + kill_cluster_procs +} diff --git a/ceph-perf-pull-requests/build/compare b/ceph-perf-pull-requests/build/compare new file mode 100644 index 000000000..c713085fa --- /dev/null +++ b/ceph-perf-pull-requests/build/compare @@ -0,0 +1,90 @@ +#!/bin/bash +set -euxo pipefail + +: "${WORKSPACE:?}" +: "${OSD_FLAVOR:?}" +: "${CHECK_APP_ID:?}" +: "${CHECK_INSTALL_ID:?}" +: "${CHECK_NAME:?}" +: "${CHECK_REPO_OWNER:?}" +: "${CHECK_REPO_NAME:?}" + +ARCHIVE_MAIN="${WORKSPACE}/cbt-results" +ARCHIVE_PR="${WORKSPACE}/ceph-pr" + +cd "${WORKSPACE}/ceph-main" +sha_main=$(git rev-parse --short HEAD) +cd "${WORKSPACE}/ceph-pr" +sha_pr=$(git rev-parse --short HEAD) +store_tag=$(cat "${WORKSPACE}/perf-store-tag") + +# shellcheck disable=SC1091 +. "${WORKSPACE}/gh-venv/bin/activate" + +report="${WORKSPACE}/report.md" +: > "$report" +result=success + +{ + echo "flavor: ${OSD_FLAVOR}" + echo "node: $(hostname -f 2>/dev/null || hostname)" + echo "store: $store_tag" + echo "baseline_sha: $sha_main" + echo "pr_sha: $sha_pr" + echo "baseline_archives: ${ARCHIVE_MAIN}//$store_tag/$sha_main/" + echo "pr_archives: ${ARCHIVE_PR}//$store_tag/$sha_pr/" + if test -f "${WORKSPACE}/perf-meta.txt"; then + cat "${WORKSPACE}/perf-meta.txt" + fi +} >> "$report" +echo >> "$report" + +compare_workload() { + local label=$1 + local workload=$2 + local archive_main="${ARCHIVE_MAIN}/${workload}/${store_tag}/${sha_main}" + local archive_pr="${ARCHIVE_PR}/${workload}/${store_tag}/${sha_pr}" + local partial="${WORKSPACE}/report-${workload}.md" + rm -f "$partial" + echo "## ${label}" >> "$report" + echo >> "$report" + if python3 "${WORKSPACE}/cbt/compare.py" -v \ + -a "$archive_pr" \ + -b "$archive_main" \ + --output "$partial"; then + : + else + result=failure + if test ! -s "$partial"; then + echo "cbt/compare.py failed for ${label}; see Jenkins console." > "$partial" + fi + fi + cat "$partial" >> "$report" + echo >> "$report" +} + +if test "${OSD_FLAVOR}" = "classic"; then + compare_workload "4K write" write +else + compare_workload "4K random read" read + compare_workload "4K write" write +fi + +github-check \ + --owner "${CHECK_REPO_OWNER}" \ + --repo "${CHECK_REPO_NAME}" \ + --pkey-file "${GITHUB_CHECK_PKEY_PEM}" \ + --app-id "${CHECK_APP_ID}" \ + --install-id "${CHECK_INSTALL_ID}" \ + --name "${CHECK_NAME}" \ + --sha "${ghprbActualCommit}" \ + --external-id "${BUILD_ID}" \ + --details-url "${BUILD_URL}" \ + --status completed --conclusion "${result}" \ + --title "${CHECK_NAME}" \ + --summary "${result}" \ + --text "$report" + +if test "$result" != "success"; then + exit 1 +fi diff --git a/ceph-perf-pull-requests/build/prepare-workloads-footer.sh b/ceph-perf-pull-requests/build/prepare-workloads-footer.sh new file mode 100644 index 000000000..887de9264 --- /dev/null +++ b/ceph-perf-pull-requests/build/prepare-workloads-footer.sh @@ -0,0 +1 @@ +PY diff --git a/ceph-perf-pull-requests/build/prepare-workloads-header.sh b/ceph-perf-pull-requests/build/prepare-workloads-header.sh new file mode 100644 index 000000000..2d9d00c74 --- /dev/null +++ b/ceph-perf-pull-requests/build/prepare-workloads-header.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Embed prepare-workloads.py into the workspace (ceph-build is not checked out on the agent). +# JJB concatenates: this header + prepare-workloads.py + footer. +cat > "${WORKSPACE}/prepare-workloads.py" <<'PY' diff --git a/ceph-perf-pull-requests/build/prepare-workloads.py b/ceph-perf-pull-requests/build/prepare-workloads.py new file mode 100644 index 000000000..e86267e76 --- /dev/null +++ b/ceph-perf-pull-requests/build/prepare-workloads.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Prepare flavor-specific CBT YAMLs and SeaStore device list. + +Env: + WORKSPACE Jenkins workspace + OSD_FLAVOR classic | crimson +""" +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + +import yaml + + +def inject_acceptable(doc: dict, acceptable: dict) -> None: + for task in doc.get("tasks") or []: + if not isinstance(task, dict) or "cbt" not in task: + continue + bench = task["cbt"].setdefault("benchmarks", {}).setdefault("radosbench", {}) + bench["acceptable"] = dict(acceptable) + return + raise SystemExit("no cbt/radosbench task found in workload YAML") + + +def warmup_read_prefill(doc: dict) -> str | None: + """Warm the system with writes before a read measurement. + + radosbench has no dedicated warm-up phase: a read test does one short + prefill write pass (default prefill_time: 3) then measures. Cold caches and + a tiny working set add variance and can leave the read phase without output. + Short-term stand-in for a real warm-up (per perf maintainer): lengthen the + single prefill to at least the measured read window. Durable fix belongs in + CBT (a native warm-up/2nd-prefill) or the checked-in Ceph CBT YAML. + """ + for task in doc.get("tasks") or []: + if not isinstance(task, dict) or "cbt" not in task: + continue + bench = task["cbt"].setdefault("benchmarks", {}).setdefault("radosbench", {}) + is_read = ( + bench.get("read_only") + or "readmode" in bench + or "prefill_time" in bench + or "prefill_objects" in bench + ) + if not is_read: + return None # write-only test: no read prefill to warm + read_time = bench.get("read_time") or bench.get("time") or 30 + current = bench.get("prefill_time") or 0 + warmup = max(current, read_time, 30) + bench["prefill_time"] = warmup + return f"read_warmup_prefill_time={current}->{warmup}" + return None + + +def warmup_write_prefill(doc: dict) -> str | None: + """Warm the system with writes before a write measurement. + + radosbench has no dedicated warm-up phase, so a write test measures from a + cold cluster (cold caches/allocator, unallocated store) which adds variance. + Per perf maintainer, add a prefill write pass (>= the measured write window, + min 30s) before the measurement. radosbench runs prefill -> write for a + write_only test, so the measured 'write/' phase is unaffected while + 'prefill/' warms the system. Durable fix belongs in CBT / the Ceph CBT YAML. + """ + for task in doc.get("tasks") or []: + if not isinstance(task, dict) or "cbt" not in task: + continue + bench = task["cbt"].setdefault("benchmarks", {}).setdefault("radosbench", {}) + write_time = bench.get("write_time") or bench.get("time") or 30 + current = bench.get("prefill_time") or 0 + warmup = max(current, write_time, 30) + bench["prefill_time"] = warmup + return f"write_warmup_prefill_time={current}->{warmup}" + raise SystemExit("no cbt/radosbench task found in workload YAML") + + +def device_has_mount(dev_path: str) -> bool: + try: + out = subprocess.check_output( + ["lsblk", "-n", "-o", "MOUNTPOINT", dev_path], + text=True, + stderr=subprocess.DEVNULL, + ) + except (subprocess.CalledProcessError, FileNotFoundError): + return True # fail closed + return any(line.strip() for line in out.splitlines()) + + +def main() -> int: + ws = Path(os.environ["WORKSPACE"]) + flavor = os.environ["OSD_FLAVOR"] + out = ws / "perf-workloads" + out.mkdir(parents=True, exist_ok=True) + main_src = ws / "ceph-main" + # 10% near-tolerance to reduce single-shot false positives vs upstream 5%. + acceptable = { + "bandwidth": "(or (greater) (near 0.10))", + "iops_avg": "(or (greater) (near 0.10))", + "iops_stddev": "(or (less) (near 2.00))", + "latency_avg": "(or (less) (near 0.10))", + "cpu_cycles_per_op": "(or (less) (near 0.10))", + } + meta: list[str] = [] + devs_file = ws / "seastore-devs.txt" + + if flavor == "classic": + src = main_src / "qa/suites/perf-basic/workloads/radosbench_4K_write.yaml" + doc = yaml.safe_load(src.read_text()) + inject_acceptable(doc, acceptable) + # perf-basic has no prefill; add a warm-up write pass before measuring. + meta.append(f"radosbench_4K_write.yaml_{warmup_write_prefill(doc)}") + (out / "radosbench_4K_write.yaml").write_text( + yaml.safe_dump(doc, sort_keys=False) + ) + meta.append(f"write_yaml_source={src}") + meta.append("workloads=write") + store_tag = "classic" + devs_file.write_text("") + else: + for name in ("radosbench_4K_read.yaml", "radosbench_4K_write.yaml"): + src = main_src / "src/test/crimson/cbt" / name + doc = yaml.safe_load(src.read_text()) + inject_acceptable(doc, acceptable) + note = warmup_read_prefill(doc) + if note: + meta.append(f"{name}_{note}") + (out / name).write_text(yaml.safe_dump(doc, sort_keys=False)) + meta.append(f"{name}_source={src}") + meta.append("workloads=read,write") + # Crimson always uses SeaStore (never CyanStore). Prefer 3 spare NVMes; + # otherwise create workspace sparse images so the job still runs SeaStore. + store_tag = "seastore" + nvmes: list[tuple[int, str]] = [] + for line in os.popen( + "lsblk -dn -b -o NAME,TYPE,SIZE,MOUNTPOINT" + ).read().splitlines(): + parts = line.split() + if len(parts) < 3: + continue + name, typ, size = parts[0], parts[1], parts[2] + mnt = parts[3] if len(parts) > 3 else "" + if typ != "disk" or not name.startswith("nvme") or mnt not in ("", "-"): + continue + path = f"/dev/{name}" + if device_has_mount(path): + meta.append(f"skip_mounted={path}") + continue + try: + nvmes.append((int(size), path)) + except ValueError: + continue + nvmes.sort(key=lambda t: (-t[0], t[1])) + if len(nvmes) >= 3: + chosen = [path for _, path in nvmes[:3]] + meta.append("seastore_backend=nvme") + meta.append( + "seastore_sizes_bytes=" + + ",".join(str(sz) for sz, _ in nvmes[:3]) + ) + else: + img_dir = ws / "seastore-imgs" + img_dir.mkdir(parents=True, exist_ok=True) + # ~32GiB sparse files (same order as upstream cyanstore memstore_device_bytes). + sparse_bytes = 34359738368 + chosen = [] + for i in range(3): + img = img_dir / f"osd-{i}.img" + # Create/resize sparse image without allocating full size. + with open(img, "wb") as fh: + fh.truncate(sparse_bytes) + chosen.append(str(img)) + meta.append("seastore_backend=sparse-file") + meta.append(f"seastore_sparse_bytes={sparse_bytes}") + meta.append( + f"seastore_note=only {len(nvmes)} unmounted NVMe(s); " + "using workspace sparse images" + ) + devs = ",".join(chosen) + src_sh = (main_src / "src/script/run-cbt.sh").read_text() + old = ( + "MDS=0 MGR=1 OSD=3 MON=1 $source_dir/src/vstart.sh -n -X \\\n" + " --without-dashboard --cyanstore \\\n" + ' -o "memstore_device_bytes=34359738368" \\\n' + " --crimson --nodaemon --redirect-output \\\n" + ' --osd-args "--memory 4G"' + ) + new = ( + "MDS=0 MGR=1 OSD=3 MON=1 $source_dir/src/vstart.sh -n -X \\\n" + f" --without-dashboard --seastore --seastore-devs {devs} \\\n" + " --crimson --nodaemon --redirect-output \\\n" + ' --osd-args "--memory 4G"' + ) + if old not in src_sh: + raise SystemExit( + "run-cbt.sh cyanstore vstart block not found; cannot enable SeaStore" + ) + wrap = ws / "run-cbt-seastore.sh" + wrap.write_text(src_sh.replace(old, new, 1)) + wrap.chmod(0o755) + devs_file.write_text("\n".join(chosen) + "\n") + meta.append(f"seastore_devs={devs}") + + (ws / "perf-store-tag").write_text(store_tag + "\n") + meta_path = ws / "perf-meta.txt" + meta_path.write_text( + "\n".join( + [ + f"flavor={flavor}", + f"node={os.uname().nodename}", + f"store={store_tag}", + *meta, + "acceptable_near=0.10", + ] + ) + + "\n" + ) + print(meta_path.read_text()) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ceph-perf-pull-requests/build/reboot b/ceph-perf-pull-requests/build/reboot new file mode 100644 index 000000000..f95c3fbd0 --- /dev/null +++ b/ceph-perf-pull-requests/build/reboot @@ -0,0 +1,7 @@ +#!/bin/bash +# Last resort when the agent is still responsive after a hard fail. +set +e +for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados; do + sudo pkill -9 -x "$p" || true +done +sudo reboot diff --git a/ceph-perf-pull-requests/build/run-cbt b/ceph-perf-pull-requests/build/run-cbt new file mode 100644 index 000000000..dc2b77bac --- /dev/null +++ b/ceph-perf-pull-requests/build/run-cbt @@ -0,0 +1,122 @@ +#!/bin/bash +set -euxo pipefail + +# Env (from EnvInject / prior steps): +# OSD_FLAVOR, SRC_DIR, WORKLOAD, ARCHIVE_BASEDIR +: "${WORKSPACE:?}" +: "${OSD_FLAVOR:?}" +: "${SRC_DIR:?}" +: "${WORKLOAD:?}" +: "${ARCHIVE_KIND:?}" + +ARCHIVE_BASEDIR="${WORKSPACE}/${ARCHIVE_KIND}" +BENCHMARK_YAML="${WORKSPACE}/perf-workloads/radosbench_4K_${WORKLOAD}.yaml" + +# Classic uses perf-basic write only; skip the crimson read workload. +if test "${OSD_FLAVOR}" = "classic" -a "${WORKLOAD}" = "read"; then + echo "Skipping read workload for classic (qa/suites/perf-basic is write-only)" + exit 0 +fi + +cd "${WORKSPACE}/${SRC_DIR}" +git submodule update --init --recursive +store_tag=$(cat "${WORKSPACE}/perf-store-tag") +# Layout: $ARCHIVE_BASEDIR/$WORKLOAD/$store_tag// +archive_dir="${ARCHIVE_BASEDIR}/${WORKLOAD}/${store_tag}/$(git rev-parse --short HEAD)" + +# An archive counts as complete only if the *measured* phase produced JSON +# output. radosbench writes json_output.. under the phase dir +# (write/, rand/, seq/); the prefill warm-up writes it under prefill/. A run +# SIGKILLed mid-read (or an old/partial baseline) can leave prefill/ with no +# measured output; reusing/accepting that makes compare.py die later with +# FileNotFoundError on the missing json_output. Ignore prefill/ so such +# archives are re-run, not trusted. +archive_is_complete() { + test -n "$(find "$1" -type f -name 'json_output.*' \ + -not -path '*/prefill/*' -print -quit 2>/dev/null)" +} + +if test -d "$archive_dir" && archive_is_complete "$archive_dir"; then + echo "Reusing existing complete CBT archive at $archive_dir" + exit 0 +fi +rm -rf "$archive_dir" +mkdir -p "$archive_dir" + +export NPROC +NPROC=$(nproc) +export FOR_MAKE_CHECK=true +export WITH_CRIMSON=true + +if test -x build/bin/crimson-osd; then + echo "Reusing existing build at $PWD/build for workload=${WORKLOAD}" +else + cmake_args="-DCMAKE_BUILD_TYPE=Release -DWITH_CRIMSON=ON -DWITH_TESTS=OFF -DWITH_SPDK=OFF -DWITH_DPDK=OFF -DWITH_QATLIB=OFF -DWITH_QATZIP=OFF ${CEPH_PERF_EXTRA_CMAKE_ARGS:-}" + timeout 7200 src/script/run-make.sh \ + --cmake-args "$cmake_args" \ + vstart-base crimson-osd +fi + +cleanup_vstart_local() { + if test -f build/ceph.conf; then + ( cd build && ../src/stop.sh --crimson ) || true + ( cd build && ../src/stop.sh ) || true + fi + kill_cluster_procs +} + +fail_and_drop_archive() { + local rc=$1 + echo "ERROR: CBT step failed (rc=$rc); dropping partial archive $archive_dir" >&2 + rm -rf "$archive_dir" + exit "$rc" +} + +cleanup_all() { + cleanup_vstart_local || true + wipe_seastore_devs || true +} +trap cleanup_all EXIT + +cleanup_vstart_local +wipe_seastore_devs +sleep 2 + +# shellcheck disable=SC1091 +. "${WORKSPACE}/gh-venv/bin/activate" + +run_cbt_sh=src/script/run-cbt.sh +if test "${OSD_FLAVOR}" != "classic" -a -f "${WORKSPACE}/run-cbt-seastore.sh"; then + run_cbt_sh="${WORKSPACE}/run-cbt-seastore.sh" + echo "Using SeaStore-enabled run-cbt wrapper: $run_cbt_sh" +fi + +set +e +if test "${OSD_FLAVOR}" = "classic"; then + timeout --foreground --signal=KILL 3600 \ + env PATH="${WORKSPACE}/gh-venv/bin:$PATH" \ + "$run_cbt_sh" --build-dir "$PWD/build" --source-dir "$PWD" --cbt "$WORKSPACE/cbt" \ + -a "$archive_dir" "$BENCHMARK_YAML" --classical + cbt_rc=$? +else + timeout --foreground --signal=KILL 3600 \ + env PATH="${WORKSPACE}/gh-venv/bin:$PATH" \ + "$run_cbt_sh" --build-dir "$PWD/build" --source-dir "$PWD" --cbt "$WORKSPACE/cbt" \ + -a "$archive_dir" "$BENCHMARK_YAML" + cbt_rc=$? +fi +set -e + +if test "$cbt_rc" -eq 124 -o "$cbt_rc" -eq 137; then + echo "ERROR: run-cbt.sh timed out after 3600s (rc=$cbt_rc)" >&2 + fail_and_drop_archive 1 +fi +if test "$cbt_rc" -ne 0; then + echo "ERROR: run-cbt.sh failed with rc=$cbt_rc" >&2 + fail_and_drop_archive "$cbt_rc" +fi +if ! archive_is_complete "$archive_dir"; then + echo "ERROR: run-cbt.sh exited 0 but produced no measured json_output" \ + "(incomplete archive, only prefill/ or empty): $archive_dir" >&2 + fail_and_drop_archive 1 +fi diff --git a/ceph-perf-pull-requests/build/setup-post.sh b/ceph-perf-pull-requests/build/setup-post.sh new file mode 100644 index 000000000..48572f812 --- /dev/null +++ b/ceph-perf-pull-requests/build/setup-post.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Continues setup after prepare-workloads.py has been written into $WORKSPACE. +set -euxo pipefail + +python3 "${WORKSPACE}/prepare-workloads.py" + +if python3 -m venv "${WORKSPACE}/gh-venv"; then + : +else + virtualenv -q --python python3 "${WORKSPACE}/gh-venv" +fi +# shellcheck disable=SC1091 +. "${WORKSPACE}/gh-venv/bin/activate" +pip install -U pip setuptools wheel +pip install cython +pip install -r "${WORKSPACE}/cbt/requirements.txt" mdutils +python3 -c "import yaml, lxml, matplotlib, mdutils, cython, setuptools" +pip install git+https://github.com/ceph/githubcheck.git +pip install 'pyjwt<2.11' + +echo "please hold tight..." | github-check \ + --owner "${CHECK_REPO_OWNER}" \ + --repo "${CHECK_REPO_NAME}" \ + --pkey-file "${GITHUB_CHECK_PKEY_PEM}" \ + --app-id "${CHECK_APP_ID}" \ + --install-id "${CHECK_INSTALL_ID}" \ + --name "${CHECK_NAME}" \ + --sha "${ghprbActualCommit}" \ + --external-id "${BUILD_ID}" \ + --details-url "${BUILD_URL}" \ + --status in_progress \ + --title "${CHECK_NAME}" \ + --summary running diff --git a/ceph-perf-pull-requests/build/setup-pre.sh b/ceph-perf-pull-requests/build/setup-pre.sh new file mode 100644 index 000000000..ce5958424 --- /dev/null +++ b/ceph-perf-pull-requests/build/setup-pre.sh @@ -0,0 +1,61 @@ +#!/bin/bash +set -euxo pipefail + +: "${WORKSPACE:?}" +: "${OSD_FLAVOR:?}" +: "${CHECK_APP_ID:?}" +: "${CHECK_INSTALL_ID:?}" +: "${CHECK_NAME:?}" +: "${CHECK_REPO_OWNER:?}" +: "${CHECK_REPO_NAME:?}" + +cd "${WORKSPACE}/cbt" +. /etc/os-release || ID=ubuntu +case $ID in +debian|ubuntu) + sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y \ + python3-yaml python3-lxml python3-prettytable python3-matplotlib cython3 git + ;; +centos|rhel|rocky|almalinux) + sudo dnf copr remove tchaikov/llvm-toolset-10 || true + sudo dnf module enable -y llvm-toolset || true + sudo dnf install -y llvm-toolset || true + sudo dnf install -y dnf-plugins-core + sudo dnf config-manager --set-enabled crb 2>/dev/null || \ + sudo dnf config-manager --set-enabled powertools 2>/dev/null || \ + sudo dnf config-manager --set-enabled PowerTools 2>/dev/null || true + if ! rpm -q epel-release >/dev/null 2>&1; then + major=${VERSION_ID%%.*} + sudo dnf install -y epel-release || \ + sudo dnf install -y "https://dl.fedoraproject.org/pub/epel/epel-release-latest-${major}.noarch.rpm" + fi + sudo dnf install -y python3-pyyaml python3-lxml python3-prettytable \ + python3-matplotlib python3-cython git + sudo dnf update -y libarchive || true + gcc_toolset_ver=13 + if test -d /opt/rh/gcc-toolset-${gcc_toolset_ver}/root/lib/gcc/x86_64-redhat-linux/${gcc_toolset_ver}; then + sudo ln -sf /opt/rh/gcc-toolset-${gcc_toolset_ver}/root/lib/gcc/x86_64-redhat-linux/${gcc_toolset_ver} \ + /usr/lib/gcc/x86_64-redhat-linux/${gcc_toolset_ver} + fi + ;; +fedora) + sudo yum install -y python3-pyyaml python3-lxml python3-prettytable \ + python3-matplotlib python3-cython clang git + ;; +*) + echo "unknown distro: $ID" + exit 1 + ;; +esac + +kill_cluster_procs +# Previous build may have left SeaStore devices dirty. +wipe_seastore_devs + +mkdir -p "${WORKSPACE}/perf-workloads" +rm -rf "${WORKSPACE}/seastore-imgs" +rm -f "${WORKSPACE}/perf-workloads"/*.yaml \ + "${WORKSPACE}/run-cbt-seastore.sh" \ + "${WORKSPACE}/seastore-devs.txt" \ + "${WORKSPACE}/prepare-workloads.py" +: > "${WORKSPACE}/seastore-devs.txt" diff --git a/ceph-perf-pull-requests/config/definitions/ceph-perf-pull-requests.yml b/ceph-perf-pull-requests/config/definitions/ceph-perf-pull-requests.yml index 0b7a37090..6db906870 100644 --- a/ceph-perf-pull-requests/config/definitions/ceph-perf-pull-requests.yml +++ b/ceph-perf-pull-requests/config/definitions/ceph-perf-pull-requests.yml @@ -45,151 +45,15 @@ clean: after: true -- builder: - name: run-cbt - builders: - - shell: | - #!/bin/bash - set -euxo pipefail - cd {src-dir} - git submodule update --init --recursive - # Per-workload archives so read/write do not clobber each other, and so - # a prior read-only cache is not mistaken for a completed write run. - # Layout: {archive-basedir}/{workload}// - archive_dir={archive-basedir}/{workload}/$(git rev-parse --short HEAD) - if test -d "$archive_dir" && test -n "$(find "$archive_dir" -mindepth 1 -print -quit 2>/dev/null)" ; then - exit 0 - fi - mkdir -p "$archive_dir" - export NPROC=$(nproc) - export FOR_MAKE_CHECK=true - export WITH_CRIMSON=true - # Same Crimson-inclusive build as ceph-pull-requests (WITH_CRIMSON=true). - # Classic vs crimson perf is selected at CBT runtime via run-cbt.sh. - # Compiler: run-make.sh installs clang-19 on Jenkins (src/script/run-make.sh - # get_llvm) and selects it via discover_compiler — same as make-check nodes. - # Reuse build/ when running a second workload (write after read) in this job. - if test -x build/bin/crimson-osd ; then - echo "Reusing existing build at $PWD/build for workload={workload}" - else - cmake_args="-DCMAKE_BUILD_TYPE=Release -DWITH_CRIMSON=ON -DWITH_TESTS=OFF -DWITH_SPDK=OFF -DWITH_DPDK=OFF -DWITH_QATLIB=OFF -DWITH_QATZIP=OFF ${{CEPH_PERF_EXTRA_CMAKE_ARGS:-}}" - timeout 7200 src/script/run-make.sh \ - --cmake-args "$cmake_args" \ - vstart-base crimson-osd - fi - # stop.sh must run from the build dir (sets CEPH_BIN=bin, conf=$PWD/ceph.conf). - # Calling it from the source tree yields: /ceph-conf: No such file or directory. - # pkill -x avoids matching this script (a -f 'rados bench' pattern self-kills). - # Double braces: JJB str.format() on macro params like {{src-dir}}. - cleanup_vstart() {{ - if test -f build/ceph.conf ; then - ( cd build && ../src/stop.sh --crimson ) || true - ( cd build && ../src/stop.sh ) || true - fi - for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados ; do - sudo pkill -9 -x "$p" || true - done - }} - # Leftover daemons from a previous hung run can wedge radosbench forever. - cleanup_vstart - sleep 2 - . ${{WORKSPACE}}/gh-venv/bin/activate - # CBT itself has no set -e / timeout; radosbench prefill has hung for days on - # bath01. Bound the whole run-cbt.sh (vstart + bench + stop) to 1h. - # SIGKILL means run-cbt.sh cannot run its own stop.sh, so we clean up after. - set +e - if test {osd-flavor} = "classic" ; then - timeout --foreground --signal=KILL 3600 \ - env PATH="${{WORKSPACE}}/gh-venv/bin:$PATH" \ - src/script/run-cbt.sh --build-dir $PWD/build --source-dir $PWD --cbt $WORKSPACE/cbt -a $archive_dir {benchmark-yaml} --classical - cbt_rc=$? - else - timeout --foreground --signal=KILL 3600 \ - env PATH="${{WORKSPACE}}/gh-venv/bin:$PATH" \ - src/script/run-cbt.sh --build-dir $PWD/build --source-dir $PWD --cbt $WORKSPACE/cbt -a $archive_dir {benchmark-yaml} - cbt_rc=$? - fi - set -e - cleanup_vstart - if test "$cbt_rc" -eq 124 -o "$cbt_rc" -eq 137 ; then - echo "ERROR: run-cbt.sh timed out after 3600s (rc=$cbt_rc); likely hung radosbench/Crimson on this node" >&2 - exit 1 - fi - if test "$cbt_rc" -ne 0 ; then - echo "ERROR: run-cbt.sh failed with rc=$cbt_rc" >&2 - exit "$cbt_rc" - fi - # run-cbt.sh is /bin/sh without set -e; a failed cbt.py can still exit 0. - test -n "$(find "$archive_dir" -mindepth 1 -print -quit 2>/dev/null)" - -- builder: - name: compare-cbt-results - builders: - - shell: | - #!/bin/bash - set -euxo pipefail - cd ${{WORKSPACE}}/{src-dir-main} - sha_main=$(git rev-parse --short HEAD) - cd ${{WORKSPACE}}/{src-dir-pr} - sha_pr=$(git rev-parse --short HEAD) - . ${{WORKSPACE}}/gh-venv/bin/activate - report=${{WORKSPACE}}/report.md - : > "$report" - result=success - - # compare.py has #!/usr/bin/python3; must invoke via venv python3. - # Run once per workload and merge into one GitHub-check markdown report - # with clear section headings (read vs write). - compare_workload() {{ - local label=$1 - local workload=$2 - local archive_main={archive-main}/$workload/$sha_main - local archive_pr={archive-pr}/$workload/$sha_pr - local partial=${{WORKSPACE}}/report-$workload.md - rm -f "$partial" - echo "## ${{label}}" >> "$report" - echo >> "$report" - if python3 ${{WORKSPACE}}/cbt/compare.py -v \ - -a "$archive_pr" \ - -b "$archive_main" \ - --output "$partial"; then - : - else - result=failure - if test ! -s "$partial"; then - echo "cbt/compare.py failed for ${{label}}; see Jenkins console." > "$partial" - fi - fi - cat "$partial" >> "$report" - echo >> "$report" - }} - - compare_workload "4K random read" read - compare_workload "4K write" write - - github-check \ - --owner {check-repo-owner} \ - --repo {check-repo-name} \ - --pkey-file ${{GITHUB_CHECK_PKEY_PEM}} \ - --app-id "{check-app-id}" \ - --install-id "{check-install-id}" \ - --name {check-name} \ - --sha ${{ghprbActualCommit}} \ - --external-id ${{BUILD_ID}} \ - --details-url ${{BUILD_URL}} \ - --status completed --conclusion ${{result}} \ - --title {check-name} \ - --summary ${{result}} \ - --text $report - - job-template: name: 'ceph-perf-{osd-flavor}' project-type: freestyle defaults: global - concurrent: true + # Serialize within this job; throttle category also serializes classic vs crimson. + concurrent: false node: performance display-name: 'ceph: {osd-flavor} perf test' - description: 'CBT performance regression check ({osd-flavor} flavor): builds ceph-pr and ceph-main, runs the checked-in radosbench 4K read and 4K write workloads, and compares results via cbt/compare.py. The GitHub check fails when compare.py reports a statistically worse result, too much noise, or errors. Trigger with comment: jenkins test {osd-flavor} perf' + description: 'CBT performance regression check ({osd-flavor} flavor): builds ceph-pr and ceph-main, runs flavor-specific radosbench workloads, and compares results via cbt/compare.py. Classic uses qa/suites/perf-basic write; crimson always uses SeaStore (spare NVMes when available, else workspace sparse images). Trigger with comment: jenkins test {osd-flavor} perf' quiet-period: 5 block-downstream: false block-upstream: false @@ -203,14 +67,20 @@ properties: - build-discarder: - days-to-keep: 15 + days-to-keep: 60 num-to-keep: 300 - artifact-days-to-keep: -1 - artifact-num-to-keep: -1 + artifact-days-to-keep: 60 + artifact-num-to-keep: 300 - github: url: https://github.com/ceph/ceph/ - rebuild: auto-rebuild: true + - throttle: + option: category + max-total: 1 + max-per-node: 1 + categories: + - ceph-perf parameters: - string: @@ -226,11 +96,6 @@ allow-whitelist-orgs-as-admins: true org-list: - ceph - # Comment-only: reliable path. Auto-build on every push with the - # performance label was confusing when the sole performance node was - # wedged (builds queued forever and looked like "trigger never works"). - # Use an explicit comment for each flavor. - # ghprb matches trigger-phrase as a Java regex (case-insensitive). trigger-phrase: '(?i)jenkins\s+test\s+{osd-flavor}\s+perf\b' skip-build-phrase: '(?i)^jenkins do not test.*' only-trigger-phrase: true @@ -249,155 +114,96 @@ - cbt builders: - - shell: | - #!/bin/bash - set -euxo pipefail - cd ${{WORKSPACE}}/cbt - . /etc/os-release || ID=ubuntu - case $ID in - debian|ubuntu) - sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y python3-yaml python3-lxml python3-prettytable python3-matplotlib cython3 git - ;; - centos|rhel|rocky|almalinux) - # llvm/gcc-toolset are EL8/9-oriented; ignore on Rocky 10+ where modules differ. - sudo dnf copr remove tchaikov/llvm-toolset-10 || true - sudo dnf module enable -y llvm-toolset || true - sudo dnf install -y llvm-toolset || true - # Repos needed on EL for CBT Python RPMs: - # - CRB/PowerTools: python3-cython (EL9/10) - # - EPEL: python3-matplotlib (EL10 AppStream does not ship it) - # config-manager comes from dnf-plugins-core; without it the - # enable steps below would no-op and python3-cython would fail. - sudo dnf install -y dnf-plugins-core - sudo dnf config-manager --set-enabled crb 2>/dev/null || \ - sudo dnf config-manager --set-enabled powertools 2>/dev/null || \ - sudo dnf config-manager --set-enabled PowerTools 2>/dev/null || true - if ! rpm -q epel-release >/dev/null 2>&1; then - major=${{VERSION_ID%%.*}} - sudo dnf install -y epel-release || \ - sudo dnf install -y "https://dl.fedoraproject.org/pub/epel/epel-release-latest-${{major}}.noarch.rpm" - fi - # Prefer lowercase python3-cython (EL10+); capital-C still works via Provides - # when CRB is enabled, but keep the modern name as the primary request. - sudo dnf install -y python3-pyyaml python3-lxml python3-prettytable \ - python3-matplotlib python3-cython git - sudo dnf update -y libarchive || true - gcc_toolset_ver=13 - # so clang is able to find gcc-toolset-${{gcc_toolset_ver}} which is listed as a - # BuildRequires in ceph.spec.in, and it is installed by `run-make.sh`. - # clang searches for GCC in a bunch of well known places: - # see https://github.com/llvm-mirror/clang/blob/main/lib/Driver/ToolChains/Gnu.cpp - if test -d /opt/rh/gcc-toolset-${{gcc_toolset_ver}}/root/lib/gcc/x86_64-redhat-linux/${{gcc_toolset_ver}} ; then - sudo ln -sf /opt/rh/gcc-toolset-${{gcc_toolset_ver}}/root/lib/gcc/x86_64-redhat-linux/${{gcc_toolset_ver}} \ - /usr/lib/gcc/x86_64-redhat-linux/${{gcc_toolset_ver}} - fi - ;; - fedora) - sudo yum install -y python3-pyyaml python3-lxml python3-prettytable \ - python3-matplotlib python3-cython clang git - ;; - *) - echo "unknown distro: $ID" - exit 1 - ;; - esac - # Clear leftover cluster processes from a previous hung/aborted run on - # this performance node before we spend hours rebuilding. - for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados ; do - sudo pkill -9 -x "$p" || true - done - # Rocky/EL10 does not ship the virtualenv CLI; prefer stdlib venv and - # fall back if ensurepip/venv support is missing (if-test avoids set -e). - if python3 -m venv "${{WORKSPACE}}/gh-venv"; then - : - else - virtualenv -q --python python3 "${{WORKSPACE}}/gh-venv" - fi - . ${{WORKSPACE}}/gh-venv/bin/activate - pip install -U pip setuptools wheel - pip install cython - # cbt imports mdutils; not listed in ceph/cbt master requirements.txt - pip install -r ${{WORKSPACE}}/cbt/requirements.txt mdutils - python3 -c "import yaml, lxml, matplotlib, mdutils, cython, setuptools" - pip install git+https://github.com/ceph/githubcheck.git - # github3.py 4.0.1 passes app_id as int; PyJWT 2.11+ requires iss to be str. - pip install 'pyjwt<2.11' - echo "please hold tight..." | github-check \ - --owner {check-repo-owner} \ - --repo {check-repo-name} \ - --pkey-file ${{GITHUB_CHECK_PKEY_PEM}} \ - --app-id "{check-app-id}" \ - --install-id "{check-install-id}" \ - --name {check-name} \ - --sha ${{ghprbActualCommit}} \ - --external-id ${{BUILD_ID}} \ - --details-url ${{BUILD_URL}} \ - --status in_progress \ - --title {check-name} \ - --summary running - - - run-cbt: - osd-flavor: '{osd-flavor}' - src-dir: "ceph-main" - workload: "read" - benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_read.yaml" - # ideally cbt-results should be persited across jobs, so the test result can be reused - archive-basedir: "$WORKSPACE/cbt-results" - - run-cbt: - osd-flavor: '{osd-flavor}' - src-dir: "ceph-main" - workload: "write" - benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_write.yaml" - archive-basedir: "$WORKSPACE/cbt-results" - - run-cbt: - osd-flavor: '{osd-flavor}' - src-dir: "ceph-pr" - workload: "read" - # use the same benchmark definition as ceph-main for apples-to-apples - # comparison and to avoid PR-side benchmark drift. - benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_read.yaml" - # use the basedir of git checkout, so it can be wiped - archive-basedir: "$WORKSPACE/ceph-pr" - - run-cbt: - osd-flavor: '{osd-flavor}' - src-dir: "ceph-pr" - workload: "write" - benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_write.yaml" - archive-basedir: "$WORKSPACE/ceph-pr" - - compare-cbt-results: - src-dir-main: "ceph-main" - archive-main: "$WORKSPACE/cbt-results" - src-dir-pr: "ceph-pr" - archive-pr: "$WORKSPACE/ceph-pr" - check-app-id: '{check-app-id}' - check-install-id: '{check-install-id}' - check-name: '{check-name}' - check-repo-owner: '{check-repo-owner}' - check-repo-name: '{check-repo-name}' + - inject: + properties-content: | + OSD_FLAVOR={osd-flavor} + CHECK_APP_ID={check-app-id} + CHECK_INSTALL_ID={check-install-id} + CHECK_NAME={check-name} + CHECK_REPO_OWNER={check-repo-owner} + CHECK_REPO_NAME={check-repo-name} + - shell: + !include-raw-verbatim: + - ../../build/common.sh + - ../../build/setup-pre.sh + - ../../build/prepare-workloads-header.sh + - ../../build/prepare-workloads.py + - ../../build/prepare-workloads-footer.sh + - ../../build/setup-post.sh + + # ideally cbt-results should be persisted across jobs so main baselines can be reused + - inject: + properties-content: | + SRC_DIR=ceph-main + WORKLOAD=read + ARCHIVE_KIND=cbt-results + - shell: + !include-raw-verbatim: + - ../../build/common.sh + - ../../build/run-cbt + - inject: + properties-content: | + SRC_DIR=ceph-main + WORKLOAD=write + ARCHIVE_KIND=cbt-results + - shell: + !include-raw-verbatim: + - ../../build/common.sh + - ../../build/run-cbt + - inject: + properties-content: | + SRC_DIR=ceph-pr + WORKLOAD=read + ARCHIVE_KIND=ceph-pr + - shell: + !include-raw-verbatim: + - ../../build/common.sh + - ../../build/run-cbt + - inject: + properties-content: | + SRC_DIR=ceph-pr + WORKLOAD=write + ARCHIVE_KIND=ceph-pr + - shell: + !include-raw-verbatim: + - ../../build/common.sh + - ../../build/run-cbt + + - shell: + !include-raw-verbatim: + - ../../build/common.sh + - ../../build/compare publishers: + - archive: + artifacts: 'report.md,report-*.md,perf-meta.txt,perf-store-tag,seastore-devs.txt,perf-workloads/**,cbt-results/**/*,ceph-pr/read/**/*,ceph-pr/write/**/*' + allow-empty: true + latest-only: false - postbuildscript: builders: + - role: SLAVE + build-on: + - SUCCESS + - UNSTABLE + - FAILURE + - ABORTED + build-steps: + - shell: + !include-raw-verbatim: + - ../../build/common.sh + - ../../build/cleanup - role: SLAVE build-on: - FAILURE - ABORTED build-steps: - - shell: | - #!/bin/bash - # Kill hung CBT/Ceph first so reboot is not the only recovery path - # when the agent is still responsive enough to run this step. - # Use -x so pkill does not match this script line. - for p in crimson-osd ceph-osd ceph-mon ceph-mgr rados ; do - sudo pkill -9 -x "$p" || true - done - sudo reboot + - shell: + !include-raw-verbatim: + - ../../build/reboot wrappers: - timeout: - # Two Crimson builds (≤2h each) + four CBT runs (read+write × main+PR, - # ≤1h each) + setup/compare. Absolute ceiling so a wedged radosbench - # cannot hold the node for days. + # Two Crimson builds (≤2h each) + CBT runs + setup/compare. timeout: 480 timeout-var: 'BUILD_TIMEOUT' fail: true