Make benchmark CI guard robust against run-to-run noise - #196
Merged
Conversation
Adopt observ's benchmark strategy. Single benchmark runs on hosted runners are too noisy to gate on: process-level effects (hash randomization, memory layout, runner load) shift individual timings by 20-40% on identical code, so the old single-run 5% mean gate flagged code paths a diff never touched. Instead, interleave 3 runs each of master and PR code with a fixed PYTHONHASHSEED, then compare via bench/compare_runs.py: a regression is only reported when the fastest PR run is slower than the slowest master run by more than the threshold. The 25% threshold sits above the observed run-to-run noise floor, making this a tripwire for gross accidental regressions rather than a precision instrument. Also add a concurrency group to cancel superseded in-flight PR runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merged
berendkleinhaneveld
added a commit
that referenced
this pull request
Jul 21, 2026
Features: - Add pure-Python view API as alternative to cgx templates (#193) - Support text elements for PySide widgets that display text (#191) Fixes & internals: - Fragment parenting overhaul (#162) - Fix PyInstaller hook for CGX files inside packages (#184) - Write compiled AST to temp file when CGX_DEBUG is set (#175) Performance: - Speed up mount path: cheap arity check, reuse first(), leaner emit (#186) - Cache Fragment._component_parent() lookups (#187) - Avoid redundant anchor lookups in Fragment.anchor() and unkeyed v-for (#188) Documentation: - Add MkDocs documentation with GitHub Pages deployment (#176) - Add internals architecture documentation page (#194) - Add docs badge and links to README (#192) Tooling & CI: - Add benchmark suite and per-PR benchmark CI workflow (#185) - Make benchmark CI guard robust against run-to-run noise (#196) - Update GitHub actions from Node 20 to Node 24 (#190) - Migrate from pre-commit to prek (#195) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adopts observ's updated benchmark strategy for the CI benchmark guard. The old guard ran the suite once per side and failed on a 5% mean delta — far below the noise floor of hosted runners, so it regularly flagged code paths a diff never touched.
The new approach:
PYTHONHASHSEED=0so dict/set timings are comparable across processes.bench/compare_runs.py(ported from observ): a regression is only reported when the fastest PR run is slower than the slowest master run by more than the threshold — a whole-process outlier on either side can't produce a false positive.The comparison also writes a markdown table to
$GITHUB_STEP_SUMMARY.Changes
.github/workflows/benchmark.yml— interleaved runs, fixed hash seed,compare_runs.pygate, concurrency group. Keeps collagraph's existingsetup-uv+setup-python(python-version-file) convention and artifact upload.bench/compare_runs.py— new comparison script (generic; reads pytest-benchmark JSON).pyproject.toml— ruff per-file-ignoreT201for the reporting script.Validation
Ran the reduced flow locally end-to-end: the
master$i/branch$isave names produce files the workflow globs (*_master?.json/*_branch?.json) match, andcompare_runs.pyparses collagraph's benchmark JSON and reports correctly (exit 0 on no regression).ruff check/formatclean.🤖 Generated with Claude Code