feat(THU-749): eval production parity, search-necessity gate, and CI

[ital0]

Closes THU-749. Locks in the THU-748 search-policy rewrite with a regression gate, brings the eval runner to production parity, and wires everything into CI.

Production parity

• Eval turns now route through createBuiltInAdapter — the exact production seam — instead of calling aiFetchStreamingResponse directly. Opus 4.8 and DeepSeek V4 Flash run on the Pi harness, GLM 5.2 on the legacy/Tinfoil pipeline, decided by the adapter's own routing (isPiModelCandidate, now exported).
• The model matrix is derived from defaultModels in shared/defaults/models.ts (no hardcoded list); scenario ids gain an engine segment (model/engine/mode/ID) and an EVAL_ENGINES filter. A snapshot-style test fails loudly when a new default model lacks an eval slug.
• Per-turn webToolBudget mirrors chat-instance.ts exactly (createWebToolBudget(resolveWebToolIntent(...))).
• Harness fixes: EVAL_timeout → EVAL_TIMEOUT (documented name never worked), timeouts now abort the in-flight request and stream parsing, README drift corrected (single shared DB, not per-scenario).

Search-necessity gate

• 96 prompts across 9 semantic categories (never_search, answer_then_offer, single_search, research, unknown_entity, false_premise, adversarial_no_search, multi_turn_reuse, and search_wont_help behind EVAL_NECESSITY_OPTIONAL), each with a quarterly reviewBy date surfaced by the report when overdue.
• Deterministic web-tool-call counting (search/fetch_content only — Pi coding tools never count) is the primary metric: new minToolCalls, web-scoped maxToolCalls, noDuplicateToolCalls.
• LLM judge (streaming, strict JSON, temperature 0) covers only the semantic assertions: search offer, premise rebuttal, verification disclaimer, and answer correctness. Flash judges Opus; Opus judges Flash and GLM — never self-judging, never GLM.
• k=3 samples with modal outcomes, Wilson 95% intervals, per-category gates (never_search ≥95% … false_premise ≥75%), and headline gates: unnecessary-search rate ≤5%, missed-search rate ≤5%.
• Existing chat suite reconciled with the 4-bucket policy: 2 stale prompts replaced with genuinely fresh equivalents, 6 relaxed to output-only (they asserted citations for answers the policy now serves from knowledge).
• Deviation from the original ticket spec, documented in the README: the ticket's "3–12 calls" for the research category predates THU-748's enforced auto budget (2 executed calls in chat), so the gate asserts the decision to search; depth stays covered by the /research suite.

CI (.github/workflows/evals.yml)

• PR smoke: path-filtered (src/ai/**, src/acp/**, shared/agent-core/**, shared/defaults/**), deterministic EVAL_SMOKE=1 subset at k=1, backend booted with pglite + anonymous Better Auth session (no auth bypass), sticky PR comment (marker-upserted) showing gates, headline metrics, and per-category deltas vs checked-in baselines with Wilson-bound significance.
• Scheduled full suite: nightly + workflow_dispatch, full matrix at k=3, uploads report/metrics artifacts, and opens/updates a draft evals/baseline-refresh PR when baselines drift — PRs never need a before/after double run; the "before" always comes from baselines produced on main.
• Metrics JSON (schema v2) covers core and necessity scenarios, keyed by model/engine cell.

Validation

Run live 6× against a CI-mirrored local backend (pglite + anonymous auth + real provider keys). The battery caught and fixed five real defects: a localStorage crash at startup (no DOM under plain bun run), judge 500s (backend proxy is streaming-only), a GLM hang (happy-dom's fetch shadowed Bun natives and deadlocked the encrypted Tinfoil stream), a research-category criteria bug, and core scenarios missing from the metrics JSON. Post-fix: flash/pi and glm/legacy run end-to-end; deterministic categories are stable across runs; observed GLM intermittency traced to transient search-provider (Exa) socket failures — the Tinfoil reliability canary working as intended.

Before merging

• Repo secrets required: ANTHROPIC_API_KEY, FIREWORKS_API_KEY, TINFOIL_API_KEY, EXA_API_KEY.
• Anthropic-dependent paths (opus cell + all judge calls) still need live validation — the local key available during validation was invalid upstream.
• First scheduled run (or workflow_dispatch → full) generates the initial baselines; the PR comment degrades gracefully ("no baseline yet") until then.

[github-actions]

Semgrep Security Scan

No security issues found.

[github-actions]

Preview environment deployed 🚀

Service	URL
Marketing / blog / docs	https://thunderbolt-pr-1191.preview.thunderbolt.io
App	https://app-pr-1191.preview.thunderbolt.io
API	https://api-pr-1191.preview.thunderbolt.io
Keycloak	https://auth-pr-1191.preview.thunderbolt.io
PowerSync	https://powersync-pr-1191.preview.thunderbolt.io

Stack: preview-pr-1191 · Commit: b53bfffb1b48695d0e06aece31a7802c2911af4b

Auto-destroys on PR close/merge. Login via the bundled Keycloak realm — demo@thunderbolt.io / demo by default.

[github-actions]

AI Eval Smoke

No baseline yet — first scheduled run will create one.

opus/pi

Gates: failed

Core suite: 3/3 passed · no baseline

Headline metric	Current	Delta vs baseline	Significance	Gate
Unnecessary search	0.0%	no baseline	no baseline	passed
Missed search	50.0%	no baseline	no baseline	failed
Mean web calls, no-search expected	0.000	no baseline	not applicable	none

Category	Current	Delta vs baseline	Significance	Gate
never_search	1/1 (100.0%)	no baseline	no baseline	passed
answer_then_offer	1/1 (100.0%)	no baseline	no baseline	passed
research	1/1 (100.0%)	no baseline	no baseline	passed
single_search	1/1 (100.0%)	no baseline	no baseline	passed
unknown_entity	0/1 (0.0%)	no baseline	no baseline	failed
false_premise	0/1 (0.0%)	no baseline	no baseline	failed
adversarial_no_search	1/1 (100.0%)	no baseline	no baseline	passed
multi_turn_reuse	1/1 (100.0%)	no baseline	no baseline	passed

Failed scenarios (2)

• unknown-entity-01: Too few web tool calls: 0 (min: 1)
• false-premise-01: Too few web tool calls: 0 (min: 1)

glm/legacy

Gates: failed

Core suite: 3/3 passed · no baseline

Headline metric	Current	Delta vs baseline	Significance	Gate
Unnecessary search	0.0%	no baseline	no baseline	passed
Missed search	25.0%	no baseline	no baseline	failed
Mean web calls, no-search expected	0.000	no baseline	not applicable	none

Category	Current	Delta vs baseline	Significance	Gate
never_search	1/1 (100.0%)	no baseline	no baseline	passed
answer_then_offer	1/1 (100.0%)	no baseline	no baseline	passed
single_search	1/1 (100.0%)	no baseline	no baseline	passed
false_premise	0/1 (0.0%)	no baseline	no baseline	failed
unknown_entity	1/1 (100.0%)	no baseline	no baseline	passed
adversarial_no_search	1/1 (100.0%)	no baseline	no baseline	passed
multi_turn_reuse	1/1 (100.0%)	no baseline	no baseline	passed
research	1/1 (100.0%)	no baseline	no baseline	passed

Failed scenarios (1)

• false-premise-01: Too few web tool calls: 0 (min: 1)

flash/pi

Gates: failed

Core suite: 3/3 passed · no baseline

Headline metric	Current	Delta vs baseline	Significance	Gate
Unnecessary search	0.0%	no baseline	no baseline	passed
Missed search	0.0%	no baseline	no baseline	passed
Mean web calls, no-search expected	0.000	no baseline	not applicable	none

Category	Current	Delta vs baseline	Significance	Gate
never_search	1/1 (100.0%)	no baseline	no baseline	passed
answer_then_offer	1/1 (100.0%)	no baseline	no baseline	passed
single_search	0/1 (0.0%)	no baseline	no baseline	failed
unknown_entity	0/1 (0.0%)	no baseline	no baseline	failed
research	1/1 (100.0%)	no baseline	no baseline	passed
adversarial_no_search	1/1 (100.0%)	no baseline	no baseline	passed
false_premise	1/1 (100.0%)	no baseline	no baseline	passed
multi_turn_reuse	1/1 (100.0%)	no baseline	no baseline	passed

Failed scenarios (2)

• single-search-01: Too many web tool calls: 3 (max: 2)
• unknown-entity-01: Too many web tool calls: 3 (max: 2)

Full eval report artifact · Generated 2026-08-05T20:17:00.766Z

[github-actions]

PR Metrics

Metric	Value
Lines changed (prod code)	+2546 / -149
JS bundle size (gzipped)	🟢 589.2 KB → 591.2 KB (+1.9 KB, +0.3%)
Test coverage	🟢 81.28% → 81.43% (+0.2%)
Performance (preview)	Preview not ready — Render deploy may have timed out
Accessibility	—
Best Practices	—
SEO	—

Updated Wed, 05 Aug 2026 20:18:51 GMT · run #2604