(``crimson-osd`` via ``vstart.sh --crimson``)
Both run on ``performance`` nodes, build ``ceph-main`` and the PR merge ref
-(``WITH_CRIMSON=ON``, ``vstart-base`` + ``crimson-osd``; clang-19 via ``run-make.sh`` on Jenkins,
-same as make-check),
-execute the ``radosbench_4K_read.yaml`` workload from ``ceph-main``, and compare
-results with ``cbt/compare.py``. A GitHub check (``perf-test-{osd-flavor}``) is
-updated with the comparison report.
+(``WITH_CRIMSON=ON``, ``vstart-base`` + ``crimson-osd``; compiler selection via
+``run-make.sh`` / ``discover_compiler``, same idea as make-check), then execute
+**both** workloads from ``ceph-main``:
+
+- ``src/test/crimson/cbt/radosbench_4K_read.yaml`` — 4K random read
+- ``src/test/crimson/cbt/radosbench_4K_write.yaml`` — 4K write
+
+Results are compared with ``cbt/compare.py``. A GitHub check
+(``perf-test-{osd-flavor}``) gets a combined markdown report with separate
+sections for read and write. The check fails if **either** workload regresses.
Triggering
----------
See ``doc/dev/continuous-integration.rst`` in the Ceph tree for an overview of
how this job fits into CI.
+What the job does
+-----------------
+
+For each of ``ceph-main`` and the PR merge ref the job:
+
+1. Builds Ceph once (the second workload reuses ``build/``).
+2. Runs CBT for **read**, archiving under ``{basedir}/read/<short-sha>/``.
+3. Runs CBT for **write**, archiving under ``{basedir}/write/<short-sha>/``.
+
+Then ``compare.py`` runs twice (read archive pair, write archive pair) and the
+markdown outputs are merged into one GitHub check body. Example shape:
+
+**4K random read** — ``all 20 tests passed``
+
+============== ==================== ========== ======== ========
+run metric baseline result accepted
+============== ==================== ========== ======== ========
+prefill/host/0 bandwidth … …
+prefill/host/0 iops_avg … …
+… … … …
+rand/host/1 latency_avg … …
+============== ==================== ========== ======== ========
+
+**4K write** — ``all 10 tests passed``
+
+============== ==================== ========== ======== ========
+run metric baseline result accepted
+============== ==================== ========== ======== ========
+write/host/0 bandwidth … …
+write/host/0 iops_avg … …
+… … … …
+write/host/1 latency_avg … …
+============== ==================== ========== ======== ========
+
+Blank ``accepted`` means OK; ``:x:`` / ❌ means the metric failed the rule.
+
+Benchmark YAMLs are always taken from **ceph-main** (not the PR tree) so both
+sides use the same workload definition and acceptance rules.
+
+Developer guide: reading the results
+------------------------------------
+
+This section explains the GitHub check table in plain terms.
+
+What question the check answers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Did this PR make the fixed 4K CBT workloads worse than current ``main`` on the
+performance node, beyond the allowed tolerance?**
+
+It is a regression smoke signal, not a full performance study and not a
+functional correctness test.
+
+Where things live
+~~~~~~~~~~~~~~~~~
+
+================ ================================= ============================================
+Piece Location Role
+================ ================================= ============================================
+Workload + rules Ceph ``src/test/crimson/cbt/`` ``radosbench_4K_{read,write}.yaml``
+Job wiring This directory (JJB YAML) Build, run CBT, post GitHub check
+CBT runner ``https://github.com/ceph/cbt`` Runs ``rados bench``, collects metrics
+Compare ``cbt/compare.py`` PR archive vs main archive → table
+GitHub check ``perf-test-classic`` / Report on the pull request
+ ``perf-test-crimson``
+================ ================================= ============================================
+
+Useful links:
+
+- Read workload: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_read.yaml
+- Write workload: https://github.com/ceph/ceph/blob/main/src/test/crimson/cbt/radosbench_4K_write.yaml
+- Metric collectors: https://github.com/ceph/cbt/blob/main/benchmark/radosbench.py
+- Acceptance evaluation: https://github.com/ceph/cbt/blob/main/benchmark/benchmark.py
+- Compare / report: https://github.com/ceph/cbt/blob/main/compare.py
+
+What each workload runs
+~~~~~~~~~~~~~~~~~~~~~~~
+
+**4K random read** (``radosbench_4K_read.yaml``):
+
+- Small local cluster (3 OSDs, replicated pool).
+- Prefill write (~3s) so there is data to read.
+- Random 4K read (~30s).
+- Two concurrent client processes (``concurrent_procs: 2``).
+
+**4K write** (``radosbench_4K_write.yaml``):
+
+- Same cluster shape.
+- Write-only 4K bench (~3s); no separate prefill/read phase.
+- Two concurrent client processes.
+
+Why the row counts are 20 and 10
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each compared cell is one **(phase × client process × metric)** tuple.
+
+Five metrics:
+
+=================== ================================================= ============
+Metric Meaning Better means
+=================== ================================================= ============
+bandwidth Throughput Higher
+iops_avg Average IOPS Higher
+iops_stddev IOPS variance / noise Lower
+latency_avg Average latency Lower
+cpu_cycles_per_op CPU cycles per op (from ``perf`` if collected) Lower
+=================== ================================================= ============
+
+- Read: phases ``prefill`` + ``rand``, 2 clients, 5 metrics → **20 rows**
+ (names like ``prefill/toko02/0``, ``rand/toko02/1``).
+- Write: phase ``write`` only, 2 clients, 5 metrics → **10 rows**
+ (names like ``write/toko02/0``).
+
+``cpu_cycles_per_op`` may show ``0.0`` when ``perf`` data was not collected; that
+usually passes as a no-op comparison.
+
+How to read the GitHub table columns
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+======== =========================================================
+Column Meaning
+======== =========================================================
+run Phase / machine / client process (``rand/host/0``)
+metric Which number
+baseline Value from **current ceph** ``main``
+result Value from **this PR**
+accepted Blank = OK; ``:x:`` / ❌ = failed the acceptance rule
+======== =========================================================
+
+Pass / fail rules (``acceptable:`` in the YAML)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ bandwidth: (or (greater) (near 0.05)) # ≥ baseline, or within ~5%
+ iops_avg: (or (greater) (near 0.05))
+ latency_avg: (or (less) (near 0.05)) # ≤ baseline, or within ~5%
+ iops_stddev: (or (less) (near 2.00)) # ≤ baseline, or within ~2×
+ cpu_cycles_per_op:(or (less) (near 0.05))
+
+Small regressions up to about 5% on throughput/latency are allowed;
+noise (``iops_stddev``) may grow up to about 2× baseline.
+
+Jenkins console log order (easy to misread)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CBT verbose logs look like::
+
+ rand/host/1: bandwidth: (or (greater) (near 0.05)):: 203.5/220.4 => rejected
+
+The fraction is **``result/baseline``** (PR first, main second), **not**
+baseline/result. In the example above the PR is slower than main, so rejection
+is correct. Prefer the GitHub table headers when in doubt.
+
+What the baseline is
+~~~~~~~~~~~~~~~~~~~~
+
+The baseline is **not** “the commit before the PR.”
+
+Each job:
+
+1. Checks out current ``origin/main`` into ``ceph-main``.
+2. Builds and runs CBT; archives under
+ ``$WORKSPACE/cbt-results/{read,write}/<main-short-sha>/``.
+3. Checks out the PR merge ref into ``ceph-pr`` and does the same under
+ ``$WORKSPACE/ceph-pr/{read,write}/<pr-short-sha>/``.
+4. Compares PR archives to main archives.
+
+If a main short-SHA archive already exists and is non-empty on that node, the
+main CBT run for that workload is **reused** (not rebuilt). When ``main`` moves,
+you get a new baseline directory — numbers can look very different from an
+earlier run on the same PR. Node noise can also move results on the same SHA.
+
+How to add or change metrics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Edit the ``acceptable:`` block in the Ceph workload YAML(s) if the metric is
+ already collected by CBT.
+2. To collect a **new** measurement, implement a getter in
+ ``cbt/benchmark/radosbench.py`` and wire evaluation in
+ ``cbt/benchmark/benchmark.py``.
+3. ``compare.py`` only reports; it does not define metrics.
+
Teuthology YAML translation
---------------------------
set -euxo pipefail
cd {src-dir}
git submodule update --init --recursive
- archive_dir={archive-basedir}/$(git rev-parse --short HEAD)
+ # Per-workload archives so read/write do not clobber each other, and so
+ # a prior read-only cache is not mistaken for a completed write run.
+ # Layout: {archive-basedir}/{workload}/<short-sha>/
+ archive_dir={archive-basedir}/{workload}/$(git rev-parse --short HEAD)
if test -d "$archive_dir" && test -n "$(find "$archive_dir" -mindepth 1 -print -quit 2>/dev/null)" ; then
exit 0
fi
# Classic vs crimson perf is selected at CBT runtime via run-cbt.sh.
# Compiler: run-make.sh installs clang-19 on Jenkins (src/script/run-make.sh
# get_llvm) and selects it via discover_compiler — same as make-check nodes.
- cmake_args="-DCMAKE_BUILD_TYPE=Release -DWITH_CRIMSON=ON -DWITH_TESTS=OFF -DWITH_SPDK=OFF -DWITH_DPDK=OFF -DWITH_QATLIB=OFF -DWITH_QATZIP=OFF ${{CEPH_PERF_EXTRA_CMAKE_ARGS:-}}"
- timeout 7200 src/script/run-make.sh \
- --cmake-args "$cmake_args" \
- vstart-base crimson-osd
+ # Reuse build/ when running a second workload (write after read) in this job.
+ if test -x build/bin/crimson-osd ; then
+ echo "Reusing existing build at $PWD/build for workload={workload}"
+ else
+ cmake_args="-DCMAKE_BUILD_TYPE=Release -DWITH_CRIMSON=ON -DWITH_TESTS=OFF -DWITH_SPDK=OFF -DWITH_DPDK=OFF -DWITH_QATLIB=OFF -DWITH_QATZIP=OFF ${{CEPH_PERF_EXTRA_CMAKE_ARGS:-}}"
+ timeout 7200 src/script/run-make.sh \
+ --cmake-args "$cmake_args" \
+ vstart-base crimson-osd
+ fi
# stop.sh must run from the build dir (sets CEPH_BIN=bin, conf=$PWD/ceph.conf).
# Calling it from the source tree yields: /ceph-conf: No such file or directory.
# pkill -x avoids matching this script (a -f 'rados bench' pattern self-kills).
#!/bin/bash
set -euxo pipefail
cd ${{WORKSPACE}}/{src-dir-main}
- archive_dir_main={archive-main}/$(git rev-parse --short HEAD)
+ sha_main=$(git rev-parse --short HEAD)
cd ${{WORKSPACE}}/{src-dir-pr}
- archive_dir_pr={archive-pr}/$(git rev-parse --short HEAD)
+ sha_pr=$(git rev-parse --short HEAD)
. ${{WORKSPACE}}/gh-venv/bin/activate
report=${{WORKSPACE}}/report.md
- # compare.py has #!/usr/bin/python3; must invoke via venv python3
- if python3 ${{WORKSPACE}}/cbt/compare.py -v \
- -a $archive_dir_pr \
- -b $archive_dir_main \
- --output $report; then
- result=success
- else
- result=failure
- if test ! -f $report; then
- echo "cbt/compare.py failed; see Jenkins console for details." > $report
+ : > "$report"
+ result=success
+
+ # compare.py has #!/usr/bin/python3; must invoke via venv python3.
+ # Run once per workload and merge into one GitHub-check markdown report
+ # with clear section headings (read vs write).
+ compare_workload() {{
+ local label=$1
+ local workload=$2
+ local archive_main={archive-main}/$workload/$sha_main
+ local archive_pr={archive-pr}/$workload/$sha_pr
+ local partial=${{WORKSPACE}}/report-$workload.md
+ rm -f "$partial"
+ echo "## ${{label}}" >> "$report"
+ echo >> "$report"
+ if python3 ${{WORKSPACE}}/cbt/compare.py -v \
+ -a "$archive_pr" \
+ -b "$archive_main" \
+ --output "$partial"; then
+ :
+ else
+ result=failure
+ if test ! -s "$partial"; then
+ echo "cbt/compare.py failed for ${{label}}; see Jenkins console." > "$partial"
+ fi
fi
- fi
+ cat "$partial" >> "$report"
+ echo >> "$report"
+ }}
+
+ compare_workload "4K random read" read
+ compare_workload "4K write" write
+
github-check \
--owner {check-repo-owner} \
--repo {check-repo-name} \
concurrent: true
node: performance
display-name: 'ceph: {osd-flavor} perf test'
- description: 'CBT performance regression check ({osd-flavor} flavor): builds ceph-pr and ceph-main, runs the checked-in radosbench 4K read workload, and compares results via cbt/compare.py. The GitHub check fails when compare.py reports a statistically worse result, too much noise, or errors. Trigger with comment: jenkins test {osd-flavor} perf'
+ description: 'CBT performance regression check ({osd-flavor} flavor): builds ceph-pr and ceph-main, runs the checked-in radosbench 4K read and 4K write workloads, and compares results via cbt/compare.py. The GitHub check fails when compare.py reports a statistically worse result, too much noise, or errors. Trigger with comment: jenkins test {osd-flavor} perf'
quiet-period: 5
block-downstream: false
block-upstream: false
- run-cbt:
osd-flavor: '{osd-flavor}'
src-dir: "ceph-main"
+ workload: "read"
benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_read.yaml"
# ideally cbt-results should be persited across jobs, so the test result can be reused
archive-basedir: "$WORKSPACE/cbt-results"
+ - run-cbt:
+ osd-flavor: '{osd-flavor}'
+ src-dir: "ceph-main"
+ workload: "write"
+ benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_write.yaml"
+ archive-basedir: "$WORKSPACE/cbt-results"
- run-cbt:
osd-flavor: '{osd-flavor}'
src-dir: "ceph-pr"
+ workload: "read"
# use the same benchmark definition as ceph-main for apples-to-apples
# comparison and to avoid PR-side benchmark drift.
benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_read.yaml"
# use the basedir of git checkout, so it can be wiped
archive-basedir: "$WORKSPACE/ceph-pr"
+ - run-cbt:
+ osd-flavor: '{osd-flavor}'
+ src-dir: "ceph-pr"
+ workload: "write"
+ benchmark-yaml: "$WORKSPACE/ceph-main/src/test/crimson/cbt/radosbench_4K_write.yaml"
+ archive-basedir: "$WORKSPACE/ceph-pr"
- compare-cbt-results:
src-dir-main: "ceph-main"
archive-main: "$WORKSPACE/cbt-results"
wrappers:
- timeout:
- # Two Crimson builds (≤2h each) + two CBT runs (≤1h each) + setup/compare.
- # Absolute ceiling so a wedged radosbench cannot hold the node for days.
+ # Two Crimson builds (≤2h each) + four CBT runs (read+write × main+PR,
+ # ≤1h each) + setup/compare. Absolute ceiling so a wedged radosbench
+ # cannot hold the node for days.
timeout: 480
timeout-var: 'BUILD_TIMEOUT'
fail: true
projects / ceph-build.git / commitdiff