feat(run-case): calibrate the regression gate against a null (bank + same-call, v2 measurement setup) - #34
Merged
Merged
Conversation
…ce model The two mean guardrails compared point estimates whose round-to-round spread was four to eight times the difference they acted on. Splitting an archive of 21 same-baseline rounds against itself — a comparison whose true answer is 'no difference' — they fired on 47% of draws. Swapping the statistic does not help: a class mean times the round count is exactly the net discordant-cell count, so the two are one quantity differing only in how many observations it is collapsed onto. What changes the answer is measuring the thresholds instead of deriving them. Three rules, each with its null measurement: - Only new-arm-only regressions count. Of eight rows the old gate once called confirmed false kills, seven were already red on main. - Confirmation is a majority of rounds, not a fixed two. At a fixed two the null's 95th percentile climbed from 3 rows at three rounds to 11 at eight, so the gate loosened as evidence accumulated. - A row margin replaces the class means, which are still printed but no longer decide. A single flaky row moves it by at most one, and the improving side subtracts the corpus's own flakiness back out. Injecting known damage into the null measures what survives: two protected rows broken consistently block 100% of the time at six rounds, twelve rows broken thinly block 78%, and nothing broken blocks 5%. Three rounds can clear a change but not condemn one, so a three-round breach reports INCONCLUSIVE rather than NO-SHIP. Also moves the aggregation policy out of the graded criteria section. It was being fed to the grader, telling it how its own scores would be pooled, and it made every gate edit invalidate criteria_sha256.
Operator-facing prose only. The 判分標準 section is extracted verbatim and sent to the grader as its rubric, so it stays byte-identical — criteria_sha256 is unchanged at 3876e6b1. Two lines were stale rather than verbose and are corrected, not shortened: the archiving section still named the retired 誤殺 0 / no-baseline-regression targets, and the dual-run step still said to repeat each configuration 2-3 times, which the null calibration since showed can clear a change but not block one.
Reverts both edits this branch made to the template: the round-budget paragraph citing humanizer-zh's null calibration, and the compression pass that followed it. The template is back to main byte-for-byte. The finding those edits carried is real but belongs somewhere a scaffold does not reach for it: three rounds clear a change and cannot condemn one. It stays recorded in skills/humanizer-zh/evals/regression-protocol.md and in the root backlog.
…nd same-call null Baseline arm answered the same prompt every round yet got regenerated every time — the rule blob and all 85 case prompts resent in full for text nobody was actually testing. Three changes, one measurement setup: - evals/run-case.json: chunks 6 -> 3, halving how many times the blob gets resent per round (partition re-verified: 85 ids, no gap, no overlap). - dispatch.py: codex runner reasoning effort xhigh -> high. - tools/run_case/bank.py (new): --build-bank dispatches N independent baseline generations once and persists them under evals/baseline-bank/<blob-sha[:12]>/; any round comparing against that same blob reads one back instead of re-dispatching. Reuse is verified against runner family/model/effort and each chunk's prompt sha256 — any mismatch is a hard error with a rebuild instruction, never a silent live-dispatch fallback. Same mechanism also closes a standing gap in the gate's calibration: the null it was built from paired archived rounds *across* separate grader calls, which is noisier than the real gate's single-call comparison, so its thresholds ran loose (documented at calibration.py:20-25, regression-protocol.md:90-92). --null-run scores two bank rounds against each other inside one grader call, with zero runner dispatch; calibrate_same_call pools those results directly instead of simulating the comparison by splitting rounds. aggregate.py's IDENTITY_FIELDS gained runner_effort, grader_effort and baseline_source so rounds measured under different setups (old 6-chunk/xhigh/cross-round vs new 3-chunk/high/bank) hard-error rather than silently pooling. Estimated new-arm-only dispatch drops from ~500k runner input tokens/round to ~135k; full null-calibration moves from 108-216 runner+grader jobs (dispatch --baseline HEAD per pool round) to 18 one-time runner jobs plus 45 grader-only --null-run jobs for a 6-round/15-pair pool. Supersedes the within-round-null plan this branch started from (same goal, reached via the bank instead of repeated --baseline HEAD rounds); the old plan file and project memory now carry a banner pointing here. skills/humanizer-zh/evals/null-r-series-SUPERSEDED.md flags the same thing in-repo, next to the r1-r3 probe rounds dispatched under the superseded approach (left as-is, not part of this commit). Calibration.json regeneration and the baseline bank itself still need a real dispatch pass (--build-bank, then --null-run across the pool) before this branch's gate numbers are trustworthy again.
Builds a 6-round baseline bank, runs all C(6,2)=15 --null-run pairings (same-call blind pairs, zero runner dispatch), and regenerates calibration.json from the pool (method: same-call, 3-round ceilings: 7 protection rows / 8 hit rows). Also commits the 3-round --baseline HEAD ship-check that verifies the new setup end to end (bank fetch + live new-arm dispatch + aggregate): SHIP, zero breaches, both arms the same blob. Keeps the three old-setup (6-chunk, xhigh, cross-round) probe rounds results-2026-08-04-null-r1~r3 as a reference range only — never pooled with the new-setup data; IDENTITY_FIELDS enforces the split.
design-notes.md: record the v2 measurement setup's completion — bank built, 15 null-run pairs, calibration.json regenerated, 3-round SHIP verified. Delete null-r-series-SUPERSEDED.md, its job (steering the other in-flight session off the old r-series) is done. backlog.md: open item — chunk 6→3 cuts token cost but not wall-clock; a single 34-case chunk still runs as one serial call, so round latency is bounded by the largest chunk, not the case count. Observed rounds up to 4h+, including two apparent stalls (all three chunks frozen 40min+, cleared by kill and retry) sharing the same failure signature as the recurring transient `claude exited 1` grader dispatch error.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the run-case regression gate's variance-model thresholds with a null-calibrated table, then rebuilds that calibration cheaply (bank + same-call same-text pairing) after the original cross-round pool went unreproducible and stale.
CONFIRM_ATfixed-2 → majority (n//2+1); mean guardrails →row_margin(net-worse minus net-better rows).tools/run_case/bank.py: baseline-arm outputs are generated once (--build-bank) and reused across rounds sharing the same baseline text, instead of regenerating every round.--null-run A,B: pairs two bank rounds inside one grader call (zero runner dispatch) — this is the gate's real pairing structure, not an approximation of it, closing the "cross-round null is noisier than the real comparison" gap the old calibration carried.evals/run-case.jsonchunks 6→3, codex runner effortxhigh→high— cuts the rule-blob resend volume ~75% per round.Why
The shipped
calibration.jsonnamed 21 source rounds that no longer exist on disk anywhere, all under a stalecriteria_sha256—--calibratecould not reproduce the shipped table. Recalibration was mandatory, not optional cleanup.Decisions
regression-protocol.mdalready discloses the existing numbers as old-setup/directional-only. Left as a follow-up.Assumptions
results-2026-08-04-null-r1~r3, 6-chunk/xhigh) are kept committed as a reference range only, never pooled with new-setup data — enforced byaggregate.py'sIDENTITY_FIELDS.Verification
--dry-run: 3-chunk partition, runner-prompt-new-arm-only, bank-missing hard error — all correct.chunk_prompt_sha256) → hard error on next use, confirmed then reverted.--aggregateon--null-runoutput → hard-rejected ("measures the noise floor, not a change"), confirming the old/new mixing guard.--calibrateregeneratedcalibration.jsonfrom the 15 null-run pairs (method: same-call).--baseline HEADship-check (new arm live-dispatched, base arm from bank, same underlying blob): SHIP, zero breaches.Known issue, not blocking
Recurring transient
claude exited 1(empty stderr) on grader dispatch — hit 11/15 times during--null-runpairing and twice during the ship-check rounds. Always cleared on retry; root cause not identified (not the earlierANTHROPIC_API_KEYcredit issue, confirmed via directclaude -psmoke test). Also observed: two apparent full stalls (all three parallel chunks frozen 40min+ with zero output growth), cleared by kill + retry.Separately, chunk 6→3 cuts token cost but not wall-clock — a single 34-case chunk is one serial generation call, so round latency is bounded by the largest chunk rather than total case count. Ship-check rounds ran up to 4h+. Recorded as an open item in root
backlog.mdfor whoever next touches chunk sizing or dispatch concurrency.Test plan
tools/run-case humanizer-zh --dry-runtools/run-case humanizer-zh --calibrate <15 null-run files>tools/run-case --aggregatemixing-guard rejection on null-run input--baseline HEADship-check → SHIP, zero breaches