bench: scenario_bench — session-level typechecker cost/benefit (REPL/SDK/one-shot/write-heavy) - #88
Open
Felipe705x wants to merge 2 commits into
Open
bench: scenario_bench — session-level typechecker cost/benefit (REPL/SDK/one-shot/write-heavy)#88Felipe705x wants to merge 2 commits into
Felipe705x wants to merge 2 commits into
Conversation
…cking "We cannot improve what we cannot measure": the internal bench measures per-case pipeline cost and pattern_typecheck measures the checker in a hot loop; neither models a SESSION — amortization horizon, shape novelty, human think-time (hardware-cache eviction between queries), error mix, DML interleaving. This bin replays named session profiles end-to-end, tc-on vs tc-off, fresh store+caches per session, AB/BA config interleave, an LLC-sized eviction sweep as think-time, and the amortization/memory ledger (open/warm time, RSS) as first-class output. Profiles and first results (LDBC SF0.1, 3 reps, idle machine): - repl_explorer (typos/drift/DML — where mistakes happen naturally): tc-on 84–97 s vs tc-off 117–133 s per session (~30% saved) and ~50 MiB lower peak RSS; 6 mistakes rejected in microseconds. - sdk_app (reviewed templates × 300 requests — errors impossible by construction; measures the TAX of leaving typechecking on): ~10 us/request; session totals statistically identical. Answer: it's free, leave it on. - one_shot_reject_lazy (cost-to-first-verdict with the setup each path actually needs): tc-on 0.095 ms at 395 MiB vs tc-off 920 ms at 537 MiB — the runtime must build its ~140 MiB index to learn what the schema already knew. - one_shot_reject_eager: the "runtime rejects in 11 us" number from hot-loop benches requires ~2.2 s of session setup the typechecker never needed; per-session the two verdicts cost the same order. - write_heavy (DML→query cycles, lazy runtime): tc-on 8–10 ms vs tc-off 4.6–4.7 s (~500×) — each mutation invalidates the runtime's index (~0.9 s rebuild) while the typechecker's schema-side caches rebuild lazily in microseconds. Documented limitations: think-time eviction cannot model frequency scaling/predictor decay of real pauses; repl_explorer's drift/error mix is synthetic (no organic usage traces exist yet) and its knobs are meant to be swept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The legacy `bench` case set is patterns-only by design — it mirrors the Python reference checker, which supported only path patterns (right shape, not the queries users type). New `full` category exercises RETURN typing, ORDER BY (column and alias), aggregates, and the multi-MATCH collapse path; the legacy set stays intact for its comparison purpose. Verdicts all correct; full queries typecheck at 3.4–13 µs on the LDBC schema (1.1–4.1× parse cost). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #87. The measurement layer the whole series was missing: sessions, not isolated calls.
Neither existing instrument models what users experience — the internal bench measures per-case pipeline cost (its question: net guardrail value),
pattern_typecheckmeasures the checker in a hot loop (its question: component attribution, patterns-only by design for the Python comparison). This bin replays named session profiles end-to-end in two configs (tc-on = REPL behavior, tc-off = unchecked counterfactual), with fresh store+caches per session, AB/BA interleave against machine drift, an LLC-sized eviction sweep standing in for human think-time, and setup time + RSS as first-class columns — the amortization/memory ledger.First results (LDBC SF0.1, 3 reps, idle):
Bonus finding: the "runtime rejects bad queries in 11 µs" number from hot-loop measurements requires ~2.2 s of session setup (open + index warm) that the typechecker's verdict never needed — the runtime's fast rejection is exactly as regime-dependent as the typechecker's was before this series.
Documented limitations in the module doc: eviction sweep ≠ real seconds-long pauses; repl_explorer's drift/error knobs are synthetic (no organic traces exist yet) and meant to be swept.
🤖 Generated with Claude Code