Add CI performance benchmark suite#223
Open
Max042004 wants to merge 2 commits into
Open
Conversation
Performance results need a stable workload so source-data changes do not look like runtime regressions. Add a small committed project split across core, filesystem, and network modules for repeatable Git, ripgrep, zstd, and Make measurements. Keep the corpus separate because its contents define the benchmark and should change only when baselines are deliberately regenerated.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Max042004
force-pushed
the
issue-195-recovered
branch
2 times, most recently
from
July 18, 2026 16:11
c606264 to
899d081
Compare
jserv
reviewed
Jul 18, 2026
Max042004
force-pushed
the
issue-195-recovered
branch
from
July 18, 2026 17:18
899d081 to
3a0517a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Max042004
force-pushed
the
issue-195-recovered
branch
from
July 19, 2026 09:36
3a0517a to
653c5fd
Compare
Contributor
|
Once the benchmark is complete, upload the results to GitHub Pages, as rv32emu-bench does. |
Max042004
force-pushed
the
issue-195-recovered
branch
from
July 19, 2026 11:59
653c5fd to
fb40607
Compare
jserv
reviewed
Jul 19, 2026
jserv
reviewed
Jul 19, 2026
jserv
reviewed
Jul 19, 2026
Functional tests do not reveal changes in syscall, process-startup, or application-level performance. Add a two-tier suite so pull requests produce repeatable measurements and expose regressions before merge. Run pinned lmbench microbenchmarks and representative developer workloads with warmups and median reporting. Store raw samples in JSON, compare them with captured elfuse, QEMU Linux, and OrbStack baselines, and treat missing baseline metrics as failures. Run the suite exclusively on the self-hosted Apple Silicon machine to avoid interference from sibling jobs. Keep comparison report-only until runner variance is characterized, while retaining artifacts for review and baseline refreshes. Closes sysprog21#195
Max042004
force-pushed
the
issue-195-recovered
branch
from
July 19, 2026 13:40
fb40607 to
c5495eb
Compare
henrybear327
approved these changes
Jul 19, 2026
henrybear327
left a comment
henrybear327
left a comment
Collaborator
There was a problem hiding this comment.
In general, LGTM. Just some nits, as the text / comment is bloating a bit I feel.
Thanks.
| github.repository == 'sysprog21/elfuse' && | ||
| (github.event_name == 'push' || github.event_name == 'pull_request') | ||
| runs-on: [self-hosted, macOS, arm64] | ||
| # Sanitizer builds run several times slower than the release build, so the |
Collaborator
There was a problem hiding this comment.
nit: rewrite the entire comment to make it clearer
| run_matrix: false | ||
| check_target: check-sanitizer | ||
| brew_pkgs: binutils | ||
| # Benchmark leg (issue #195): release build, two-tier suite |
| green `make check` covers BusyBox validation. Use `make test-busybox` to | ||
| iterate on a single applet failure without rerunning the unit suite. | ||
|
|
||
| ## Performance Benchmarks |
Collaborator
There was a problem hiding this comment.
nit: make the text concise
I think the current testing.md has grown to the size that's already too big and too complicated to read
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a reproducible two-tier performance benchmark suite for detecting
regressions in pull requests.
filesystem latency.
against a deterministic committed corpus.
and OrbStack.
CI integration
Add a benchmark leg to the self-hosted Apple Silicon runtime matrix.
The leg runs exclusively on the host to avoid interference from sibling
jobs, compares results in report-only mode, and uploads the JSON artifact.
Missing baseline metrics are treated as failures, preventing broken or
timed-out benchmarks from silently passing. Benchmark preparation and
cleanup commands are also bounded by timeouts.
The checked-in baseline has not yet been captured on the designated
self-hosted CI machine. It will be refreshed in a follow-up after the
benchmark leg is deployed and runner variance has been characterized.
Comparison therefore remains report-only for now.
Validation
Closes #195
Summary by cubic
Adds a two-tier performance benchmark suite to catch syscall, process-startup, and app-level regressions. Runs on the self-hosted Apple Silicon runner with an exclusive host lock and posts a sanitized PR comment with benchmark deltas. (Issue #195)
New Features
tests/bench-suite.sh: Tier 1lmbench(lat_syscall/lat_proc/lat_fs) and Tier 2 app workloads (git status,rg,make,zstd) with warmups; writes medians+samples tobuild/bench-results*.json. Guest-side timing forqemu-aarch64viatests/bench-timeit.c.tests/bench-corpusfor stable Tier 2 inputs; excluded from formatting to preserve baselines. Fixtures fetch app tools and cross-compile pinnedlmbench.tests/bench-baseline.jsonforelfuse-aarch64,qemu-aarch64, andorbstack.scripts/bench-compare.pyflags regressions with a tunable threshold and treats missing metrics as NEW (not gating).scripts/bench-promote.pyrefreshes baselines.make bench/make bench-ci;BENCH_ENVselectselfuse-aarch64(default) |qemu-aarch64|orbstack|native. Benchmark leg runs under an exclusive host lock viascripts/ci-host-lock.sh. A new workflow (.github/workflows/bench-report.yml) usesscripts/bench-report.pyto publish a sanitized advisory PR comment from the uploaded report artifact. Docs indocs/testing.md.Migration
make bench(orBENCH_ENV=qemu-aarch64 make bench-ci).scripts/bench-compare.py --report-only(tune viaBENCH_REGRESSION_THRESHOLD).make bench-ciper env, thenscripts/bench-promote.py; commit the updatedtests/bench-baseline.json.Written for commit c5495eb. Summary will update on new commits.