Benchmarks

Performance benchmarks for qcr-core are built with Criterion.rs. They cover every hot path in the system: SSE parsing, session serialization, glob search, i18n lookup, skill loading, agent loop latency, tool execution overhead, config loading, and provider pipeline throughput.

Quick Start

# Compile all bench targets (no measurement — fast feedback)
cargo bench --no-run -p qcr-core

# Run every benchmark (~10–15 min on a laptop)
cargo bench -p qcr-core

# Run one target
cargo bench -p qcr-core --bench sse_parsing

# Run with short sample size for a quick smoke-test
cargo bench -p qcr-core --bench sse_parsing -- --sample-size 10 --warm-up-time 1 --measurement-time 2

# Filter to a specific group or scenario
cargo bench -p qcr-core --bench session_serialization -- session_roundtrip

Criterion writes HTML reports to target/criterion/report/index.html and JSON estimates to target/criterion/<group>/<bench>/new/estimates.json.

Benchmark Targets

All bench files live in crates/qcr-core/benches/.

File	What it measures
`sse_parsing.rs`	SSE stream parse throughput: token count scaling, chunk fragmentation, Anthropic vs OpenAI format, unicode split boundaries, time-to-first-event
`session_serialization.rs`	JSON serialize / deserialize / file roundtrip for sessions; scales from 10 to 1000 messages; tool-heavy variants; `list_sessions` directory scan
`glob_search.rs`	Gitignore-aware glob walk on synthetic trees (small / medium / large / 10 000 files); pattern complexity; `glob` crate vs `ignore` crate comparison
`skill_loading.rs`	Skill file parse from disk; 200-skill corpus; Unicode filenames; lookup-by-name throughput
`i18n_lookup.rs`	Locale key lookup; bulk 10 000 iterations; locale switching; interpolation; missing-key path
`agent_loop.rs`	End-to-end agent loop latency using `MockGenerator`; text-only, single tool, multi-turn, parallel tools; memory allocation tracking
`tool_execution.rs`	Tool registry cold/warm build; per-call overhead; file I/O tool benchmarks; memory per execution
`config_loading.rs`	Default config construction; env override application; full file load from disk
`provider_pipeline.rs`	Provider selection, failover path; mock streaming pipeline throughput

Shared Infrastructure

`benches/common/`

All bench files share a helper module at benches/common/mod.rs. Include it with:

#[path = "common/mod.rs"]
mod common;

Key helpers:

Function	Purpose
`common::rt()`	Single-threaded `tokio::Runtime` — create once per group, outside `b.iter`
`common::openai_sse_payload(n)`	Build an OpenAI-format SSE payload for `n` tokens
`common::anthropic_sse_payload(n)`	Anthropic-format SSE payload for `n` events
`common::split_into_chunks(payload, size)`	Slice a payload into fixed-size `Bytes` chunks
`common::split_at_utf8_boundaries(payload, size)`	Same but always splits at valid UTF-8 boundaries
`common::create_file_tree(root, n)`	Create `n` files in a temp directory
`common::create_nested_gitignore_tree(root, n)`	Nested dirs each with a `.gitignore`
`common::create_skill_files(dir, n)`	Write `n` skill YAML files
`common::write_minimal_config(path)	Minimal `.qcr/settings.json` for config-load benches ` for config-load benches
`common::with_tempdir(setup)`	Run a setup closure in a `TempDir`, return `(dir, T)`

`MockGenerator`

benches/common/mock_generator.rs implements the ContentGenerator trait with deterministic, in-memory StreamEvent sequences. Use it to isolate agent-loop overhead from real LLM network latency.

use common::mock_generator::MockGenerator;

// Produces: 20 × TextDelta + Usage + Done
let gen = MockGenerator::text_only(20);

// Produces: ToolCallStart + ToolCallDelta + ToolCallEnd + Usage + Done
let gen = MockGenerator::with_tool_call("read_file", serde_json::json!({"path": "x.txt"}));

// Multi-turn: text + tool call, repeated `turns` times, then Done
let gen = MockGenerator::multi_turn(3);

// N parallel tool calls in a single response
let gen = MockGenerator::parallel_tool_calls(4);

Each MockGenerator is Clone and Send + Sync — safe to use inside iter_batched setups.

`CountingAllocator`

benches/common/allocator.rs wraps the system allocator and tracks peak / current heap bytes. Use it to measure allocations for a single bench iteration without a profiler:

use common::allocator::CountingAllocator;

#[global_allocator]
static ALLOC: CountingAllocator = CountingAllocator::new();

// In bench group setup:
CountingAllocator::reset();
// ... run the operation ...
let peak = CountingAllocator::peak_bytes();

Only one bench file should install CountingAllocator as the global allocator at a time (currently agent_loop.rs). Other targets measure via alloc_bytes / dealloc_bytes helpers that route through the counting allocator when it is active.

Writing a New Benchmark

Create crates/qcr-core/benches/<name>.rs.
Add a [[bench]] entry to crates/qcr-core/Cargo.toml:
```
[[bench]]
name = "<name>"
harness = false
```

Follow the Criterion group pattern:

use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput};

#[path = "common/mod.rs"]
mod common;

fn bench_my_thing(c: &mut Criterion) {
    let mut group = c.benchmark_group("my_group");

    for size in [10usize, 100, 1000] {
        // Build fixture once, outside iter
        let data = build_fixture(size);
        group.throughput(Throughput::Elements(size as u64));
        group.bench_with_input(BenchmarkId::from_parameter(size), &data, |b, d| {
            b.iter(|| {
                // measure only this
                process(d)
            });
        });
    }

    group.finish();
}

criterion_group!(benches, bench_my_thing);
criterion_main!(benches);

For async code, create a runtime once outside the loop:

fn bench_async_thing(c: &mut Criterion) {
    let rt = common::rt();
    let mut group = c.benchmark_group("async_group");

    group.bench_function("my_async_op", |b| {
        b.iter(|| rt.block_on(async { my_async_op().await }));
    });

    group.finish();
}

Use iter_batched when each iteration needs fresh state (e.g. a file that gets consumed):

use criterion::BatchSize;

b.iter_batched(
    || build_fresh_state(),     // setup — not measured
    |state| use_state(state),   // measured
    BatchSize::SmallInput,
);

Run cargo bench --no-run -p qcr-core to verify it compiles before measuring.

Regression Gating

How It Works

The CI pipeline (.github/workflows/bench.yml) runs benchmarks on every push to main and every pull request.

Push to main: benchmarks run and the Criterion estimates.json files are uploaded as the bench-baseline artifact (retained 90 days).
Pull request: benchmarks run, then scripts/bench_compare.py compares the PR results against the stored baseline. If any benchmark's mean increases by more than 10% the PR check fails and a comparison table is posted as a sticky comment.

bench_compare.py

# Compare two local criterion output directories
python3 scripts/bench_compare.py bench-baseline bench-current

# Custom threshold (5%) and inconclusive CV (3%)
python3 scripts/bench_compare.py bench-baseline bench-current --threshold 5 --inconclusive-cv 3

The script:

Recursively finds all estimates.json files under each directory.
Computes (current_mean - baseline_mean) / baseline_mean * 100 for each benchmark.
Classifies results as ok / regressed / improved / inconclusive / new.
Prints a Markdown table and summary to stdout.
Exits 1 if any benchmark is regressed.

A result is marked inconclusive (and not counted as a regression) when the current run's coefficient of variation exceeds 5%. This handles noisy CI environments.

Overriding the Gate

Set the PERF_EXCEPTION repository secret to 1 to skip the threshold check for a PR. The script still prints the comparison but exits 0. Use sparingly — it bypasses the gate for the entire run, not just one benchmark.

Saving a New Baseline Locally

After a significant intentional performance improvement, update the stored baseline:

# Run benches and collect estimates
cargo bench -p qcr-core
mkdir -p bench-baseline
find target/criterion -name "estimates.json" | while read src; do
  rel="${src#target/criterion/}"
  dst="bench-baseline/$rel"
  mkdir -p "$(dirname "$dst")"
  cp "$src" "$dst"
done

# Verify the new baseline is self-consistent (should show all "ok")
python3 scripts/bench_compare.py bench-baseline bench-baseline

Then commit bench-baseline/ to a dedicated baseline branch or push to main to trigger the CI artifact upload.

Benchmark Results (2026-03-08)

Results from a local run with --sample-size 10 --warm-up-time 1 --measurement-time 2. These serve as the initial baseline. Numbers vary by machine; the relative ratios are what matter.

SSE Parsing

Benchmark	Mean	Notes
`sse_event/event_is_done_check`	4.2 ns	branch on `[DONE]` sentinel
`sse_event/event_data_only`	16.8 ns	parse single `data:` line
`sse_event/event_with_event_field`	48.8 ns	parse `event:` + `data:`
`sse_event/event_json_parse`	295 ns	parse + deserialize JSON delta
`sse_small/openai_10_tokens_no_fragmentation`	2.49 µs	10 tokens, one chunk
`sse_small/openai_10_tokens_64b_chunks`	3.13 µs	10 tokens, 64-byte chunks
`sse_large/openai_tokens_full_payload/100`	34.5 µs	100 tokens, full payload
`sse_large/openai_tokens_32b_chunks/100`	45.1 µs	100 tokens, 32-byte chunks
`sse_large/openai_tokens_full_payload/500`	784 µs
`sse_large/openai_tokens_full_payload/1000`	8.91 ms
`sse_large/openai_tokens_32b_chunks/1000`	459 µs	fragmented is faster: less alloc pressure
`sse_anthropic/anthropic_events_full_payload/200`	172 µs	multi-field format ~1.7× slower than OpenAI
`sse_10000_chunks/openai_10000_64b_chunks`	3.39 ms
`sse_10000_chunks/openai_10000_full_payload`	310 ms
`sse_10000_chunks/anthropic_10000_full_payload`	521 ms
`sse_split_boundary/unicode_content_split_at_utf8_boundary`	51.1 µs	correct path
`sse_split_boundary/unicode_content_byte_by_byte`	486 µs	9.5× slower — never use
`sse_time_to_first_event/single_token_one_chunk`	516 ns	TTFE for trivial stream
`sse_time_to_first_event/first_of_1000_tokens_32b_chunks`	89.5 µs	TTFE through fragmented buffer

Session Serialization

Benchmark	Mean
`message_construction/user_message_short`	17.2 ns
`message_construction/assistant_message_with_code`	20.6 ns
`session_serialize/to_json/10`	2.08 µs
`session_serialize/to_json/100`	15.7 µs
`session_serialize/to_json/1000`	156 µs
`session_deserialize/from_json/10`	2.64 µs
`session_deserialize/from_json/100`	20.5 µs
`session_deserialize/from_json/1000`	198 µs
`session_roundtrip/serialize_then_deserialize/500`	179 µs
`session_file_roundtrip/save_and_load/100`	124 µs
`session_large/file_save_and_load/500`	375 µs
`session_tool_heavy/serialize/500`	133 µs
`session_tool_heavy/deserialize/500`	169 µs
`list_sessions/read_dir_and_parse_all/10`	168 µs
`list_sessions/read_dir_and_parse_all/50`	796 µs
`list_sessions/read_dir_and_parse_all/200`	3.20 ms

Serialization scales linearly — no quadratic allocations.

Glob Search

Benchmark	Mean
`glob_small_tree/specific_file`	28.5 µs
`glob_small_tree/all_json_files`	143 µs
`glob_medium_tree/all_rs_files/300`	478 µs
`glob_pattern_complexity/dir_prefix_wildcard`	124 µs
`glob_pattern_complexity/double_star_all`	429 µs
`glob_large_tree/all_json_files/500`	2.06 ms
`glob_10000/double_star_all`	10.6 ms
`glob_10000/all_json_full_walk`	11.2 ms
`gitignore_eval/match_1000_paths`	126 µs
`grep_compile_vs_scan/regex_compile_only`	182 µs
`grep_compile_vs_scan/regex_scan_precompiled`	57.0 µs

Pre-compiling a Regex is 3.2× faster than compiling on every call.

Design Notes

Why Criterion? Statistical sampling, outlier detection, HTML flame charts, and machine-readable JSON estimates in one crate. The JSON output powers the regression gate.

Why a single tokio runtime per group? Runtime startup (~50–200 µs) would dwarf short async operations. Creating rt() outside b.iter ensures we measure only the async work.

Why iter_batched for file I/O benchmarks? Some operations consume their input (e.g. a File handle, a TempDir). iter_batched runs the setup closure before each iteration so the measurement is clean.

Why MockGenerator instead of a recorded HTTP trace? A recorded trace adds I/O jitter and makes benchmarks network-dependent. MockGenerator emits pre-built StreamEvent values synchronously, giving sub-microsecond variance between runs and full control over the scenario.

Why CountingAllocator? Platform memory profilers (Instruments, Valgrind) require manual invocation and produce results outside the Criterion loop. CountingAllocator lets us assert peak_bytes() < threshold inline and surface regressions in the same CI pass.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmarks

Quick Start

Benchmark Targets

Shared Infrastructure

`benches/common/`

`MockGenerator`

`CountingAllocator`

Writing a New Benchmark

Regression Gating

How It Works

bench_compare.py

Overriding the Gate

Saving a New Baseline Locally

Benchmark Results (2026-03-08)

SSE Parsing

Session Serialization

Glob Search

Design Notes

FilesExpand file tree

benchmarks.md

Latest commit

History

benchmarks.md

File metadata and controls

Benchmarks

Quick Start

Benchmark Targets

Shared Infrastructure

benches/common/

MockGenerator

CountingAllocator

Writing a New Benchmark

Regression Gating

How It Works

bench_compare.py

Overriding the Gate

Saving a New Baseline Locally

Benchmark Results (2026-03-08)

SSE Parsing

Session Serialization

Glob Search

Design Notes

`benches/common/`

`MockGenerator`

`CountingAllocator`