Performance regression tests in CI

[simontaurus]

Add performance regression tests to CI. Track serialization/query times across commits.

Acceptance Criteria

• Performance benchmarks in CI
• Tracking of serialization times across commits
• Tracking of query times across commits
• Alerts on significant regressions

[raederan]

CI Benchmark (for instance #22 (comment)):

• compare version PR source and targent
• auto comment in PR

[raederan]

What is the initial benchmark setup?

existing benchmarking

oold-python/pytest.benchmark.ini

Line 11 in 9efac03

--benchmark-timer=time.perf_counter

see example
#95 (comment)

• Comparison "feature branch" vs "main" within PR functional?
• Negative example implementation, e.g., using "sleep" to test benchmarking
• Optimization of Benchmarking in general (if possible and optional)

[raederan]

@simontaurus see perspective below for comparison and sota tools for optional optimization

Perspective on continuous benchmarking

Tool	History	Alerting	Noise handling	Rewrite	External account	Cost
github-action-benchmark	JSON + charts on gh-pages	Commit comment, @mention, alert/fail thresholds	None, wall clock on shared runners	None, native pytest-benchmark adapter	No, GITHUB_TOKEN	Free
CodSpeed	Hosted dashboard, per commit	PR check + comment	Very high, Cachegrind instruction counting	Low, API-compatible	Yes, CODSPEED_TOKEN	Free for public repos
Bencher.dev	Hosted or self-hosted	PR comment, statistical threshold models	Statistical boundary models	None, ingests pytest-benchmark JSON	Yes, BENCHER_API_TOKEN	Free for public repos
asv	Strong, incl. bisection	Weak, manual	Assumes dedicated machine	Full rewrite	No	Free

Trade-offs in one line each: github-action-benchmark is lowest friction but has no noise handling; CodSpeed and Bencher solve noise (instruction counting vs. statistics) at the cost of an external account/token; asv is most feature-rich but needs a rewrite and a dedicated machine.

1. Feature branch vs. main within a PR

Aspect	Same-run worktree	Stored main baseline
Mechanism	PR checks out main into a worktree, re-runs, compares	Each merge to main records a result; PR downloads last entry
Noise	Best, same runner/minute	Worse, different runners, 10-30% drift
CI cost	~2x	~1x
Fragility	Silent skip if main lacks a benchmark	Needs history store to exist first
Usable threshold	Tight, ~1.2x	Loose, 1.5x+

Functional either way; the trade is CI cost vs. threshold tightness.

2. Negative example (sleep) to validate benchmarking

Temporarily insert an artificial slowdown into a benchmarked function, confirm the alert/fail threshold fires, then revert. One-off validation, not a permanent CI step; applies to any tool above.

3. What CodSpeed / Bencher win us, and whether it's worth it

• Noise immunity: CodSpeed via instruction counting, Bencher via statistical boundaries, both let thresholds go tight (CodSpeed ~1.05x) or adaptive (Bencher, no fixed ratio).
• Solves the PR-baseline trade-off directly: stored-main baseline becomes safe at ~1x cost, the same-run vs. stored-baseline noise penalty disappears.
• Setup stays drop-in at the benchmark-code level: pytest-codspeed is API-compatible, Bencher ingests existing pytest-benchmark JSON.
• Dashboard hosted (Bencher also self-hostable), unaffected by the gh-pages/deploy-pages conflict since nothing needs to be served from gh-pages.
• Cost: free for public repos, both. Trade-off: external account and token required.

4. Template repo fit

• github-action-benchmark: zero secrets, zero account setup, works the moment it's copied into a new repo, best default for a template used across many downstream repos.
• CodSpeed / Bencher: need a per-repo account and secret before the workflow runs, breaks "clone and go." Better as an opt-in upgrade documented in the template's README than as the wired-in default.

5. Is github-action-benchmark a drop-in swap for CodSpeed or Bencher later?

• Benchmark code: yes, unchanged, all three consume pytest-benchmark JSON.
• CI workflow: no, different action, inputs, and a secret/account needed.
• History: no, existing gh-pages JSON isn't importable, history restarts on switch.
• Branch-comparison strategy: carries over as-is, only the threshold and noise model change, since a template built on the stored-main-baseline pattern doesn't need to re-architect that part when upgrading.

[raederan]

go with Bencher.dev, for template opt in