Epic 11: Eval economics - prove the tiering with RTIA as testbed

[augustineuzokwe]

Goal

Turn a session's worth of eval-design theory into provable experiments, using RTIA itself as the testbed. Each experiment produces a number or artifact that doubles as evidence for a blog follow-up (Lessons 7-9 of the "testing AI features" series).

The core insight being proved

The LLM response cache (ADR-0013, keyed on prompt_hash + resolved input messages) is not just a cost optimisation. Its on/off state changes which class of failure you can detect:

Cache state	Catches	Blind to	Tier
ON	your prompt regressions (cheap, only the changed slice re-runs)	model drift	per-PR
OFF	model drift (everything runs live)	nothing extra	nightly
uncacheable	probabilistic safety failures (N runs, pass-rate)	—	nightly/batch

Turning the cache off isn't a cost decision; it switches the bug you're hunting. And a content-addressed cache is effectively a free, call-level regression-test-selection engine that follows the data-flow blast radius of a change.

Child issues

1. Cache = test selection (headline proof) - change one prompt, measure paid calls vs cache hits; show only the changed agent + downstream re-run.
2. Free local canary - deliberately break a prompt; show qwen2.5:3b on the NAS catches the model-agnostic regression for $0.
3. Stochastic calibration - reproduce the N=10/95% false-fail; show N=20 (or threshold 0.90) holds. Lesson: N and threshold must be statistically compatible.

Not in scope

• Building a per-PR Gemini gate (we decided RTIA doesn't need one; structural tests + manual-before-prompt-PRs suffice for a solo learning project).
• Shipping RTIA on a local model (the deep path is too slow on CPU; SPLIT path only).

Reference

Full findings: learning/2026-06-07-local-llm-and-eval-economics.md (gitignored).
Parent context: Epic #353 (Local LLM on NAS).