fix(benchmark): make the instrument honest, then cut direction's 38.7% cost share - #192
Open
TinDang97 wants to merge 18 commits into
Open
fix(benchmark): make the instrument honest, then cut direction's 38.7% cost share#192TinDang97 wants to merge 18 commits into
TinDang97 wants to merge 18 commits into
Conversation
…d flag is the surfacing ceiling Across three amb1 reps ADD surfaced exactly 1 of 7 planted ambiguities EVERY time — never 0, never 2 — while its PLAN.md template asks for one "Least-sure flag surfaced at freeze", singular and ranked lowest-confidence first. Two readings fit that data: a) the singular flag is a CEILING — ADD noticed more and reported one; b) ADD noticed one. This arm separates them by adding ONE clause to the wrapper: at freeze, list every choice the source spec does not settle, not just the least-sure one. If (a), the rate rises; if (b), it does not and the flag design is exonerated. Either outcome is informative, which is what an A/B needs. Why this and not gate infrastructure: the earlier claim that ADD's disabled human gate was the benchmark's largest distortion does not survive checking. The frozen §3 contract in all three ADD runs contains no trace of the two gaps every build shipped — no ownership, 403, permission or caller-scoping vocabulary in ~3000 characters of contract each. A human at that gate would have approved a document in which both decisions were invisible, because a gate reviews what was written down and a silent gap is by definition not written down. Also checked and NOT acted on: MARKERS contains "clarif", documented as catching spec-kit's NEEDS CLARIFICATION token, while ADD's ⚠ idiom is absent — a real asymmetry in the detector's construction. Adding "⚠" (and an "overlap" anchor) changes NOTHING on these runs: the one sentence it newly matches attributes to A-conflict-response, which ADD already surfaced. A change that would have favoured the method this benchmark belongs to was tested and dropped. Isolation is asserted mechanically, not by eye: the variant minus the clause must equal the baseline byte for byte, so a second smuggled difference fails the suite (verified by mutation). The clause is checked to leak no planted anchor, no domain vocabulary from this workload, and no quota that could be padded. Registered as EXPERIMENTAL: selectable by name, absent from ARM_NAMES, so no default campaign changes cost or composition. Not yet run — a live A/B costs roughly $2 per rep and is the human's call. benchmark/tests: 411 -> 423 passing. author: Tin Dang
…d arm was unrunnable The first launch of add-enumerate failed instantly on `unknown_arm` with 423 tests green. score.py validates arm names against ARM_NAMES independently of the loader and the CLI, and the new suite covered the loader and the CLI only — tested unit, untested seam, the same shape this codebase keeps producing. Cheap this time: the run died before spending anything. It would not have been cheap inside a multi-rep campaign. The guard is now enumerative rather than remembered: no module outside tests may validate against ARM_NAMES, the DEFAULT campaign set. A fourth gate added later fails in the suite instead of at the start of a paid run. author: Tin Dang
…ntation
add-enumerate's first run reported 3 of 7 surfaced against a baseline of 1 of 7.
Reading the evidence spans, two of the three were junk and the "rise" was
entirely spurious. Three distinct defects, all found by auditing a result that
moved in the direction I wanted:
D1 ANCHORS MATCHED INSIDE LONGER WORDS. "position" is a substring of
"composition", so a sentence about image composition surfaced
A-position-ordering. Anchors are TERMS; they now match on word boundaries.
This also corrupted ATTRIBUTION, which ranks on anchor-hit count — a phantom
hit could outrank and steal a genuine surfacing from the item that earned it.
Markers stay unbounded deliberately: they are stems (assum -> assumption).
D2 A CLOSING XML TAG ACTED AS A MARKER. "assum" is a substring of
"</assumptions>", so the tag ENDING ADD's assumptions block marked whichever
sentence followed it — live, the opening line of the contract body. Simple
tags are now blanked before matching, with EQUAL-LENGTH spaces so every offset
(including edit_pos, which indexes the untouched transcript) still points
where it did. Only well-formed tags: a greedy <...> strip would eat
"start < other.end AND end > other.start", which is prose about a boundary and
frequently the very thing being surfaced.
D3 THE ARTIFACT BUDGET READ SHIPPED DOCUMENTATION. _workspace_artifacts took the
first 40 prose files in sort order. An ADD workspace holds 302, of which 256
are the vendored personas-teacher library — so the budget was consumed
entirely by ADD's own product documentation while PLAN.md, the agent's actual
reasoning, sorted at index 270 and was never read at all. The live false
positive came from design-image-prompt-engineer.md, whose boilerplate reads
"Avoid ambiguous language that could be interpreted multiple ways". That
sentence ships in every ADD workspace; crediting it scores an arm for the
contents of its installer.
Artifacts are now the documents the agent WROTE, per its own tool calls —
arm-neutral by construction, since it asks what the run produced rather than
where a method files things. Fails CLOSED: no transcript means no artifacts,
never "read everything", because reading everything is what caused this.
D3 is the original `artifacts = ()` bug wearing the opposite sign: that one read
none of the agent's documents, this one read everything EXCEPT them. Both hid in
the same place — tests exercised classify() directly and never the seam that
chooses what to feed it.
RE-SCORED, corrected detector:
add (baseline) 0.143 [0.143, 0.143, 0.143] unchanged
spec-kit 0.095 [0.143, 0.143, 0.0] unchanged
add-enumerate 0.143 [0.143] was 0.429
The experiment's apparent lift is gone. Every surviving verdict rests on a
readable sentence that names the contradiction.
Two tests from the earlier artifact fix are AMENDED, not weakened: they now
record the Write in the transcript, while the payload still omits the
recognition — the realistic shape, since an Edit records a slice and the file on
disk ends up saying what no single payload did. Each still passes only by
reading the file.
benchmark/tests: 425 -> 435 passing.
author: Tin Dang
…scoring every unittest arm 0.0
`_assert_fingerprints` walked for `ast.Assert` and nothing else. Census over
the archived wm1-wm6 campaign:
add wm6: bare `assert` statements = 0 self.assertX(...) = 189
spec-kit wm6: bare `assert` statements = 175 self.assertX(...) = 0
ADD's frozen contract targets `python -m unittest discover -s tests`, so all
189 of its assertions were invisible to the detector and only whole removed
test FUNCTIONS could ever score. spec-kit's asserts were counted in full.
The metric compared assertion syntax, not test discipline — and it happened
to flatter the method this benchmark belongs to, which is the direction a
defect is least likely to get questioned. It was about to be published as
"ADD 0.267 vs spec-kit 1.333".
Both syntaxes now normalize to ONE fingerprint, so a pure
`assert a == b` -> `self.assertEqual(a, b)` migration scores zero rather
than trading this bias for its mirror image. An assertX with no operator
equivalent (assertRaises, assertAlmostEqual) fingerprints as itself.
Auditing the re-scored numbers then found a SECOND distortion: identity is
`<relpath>::<fn name>`, so a rename reads as a removed test. Three of the
four counts in spec-kit's largest case were renames under tenant isolation
(test_list_returns_all_bookings -> test_list_returns_only_own_bookings).
`compute_tests_weakened`'s definition is frozen in its §3 contract, so the
rename-immune accounting lands as a NEW metric, `compute_assertions_lost` —
a snapshot-wide multiset difference. Both are reported; they answer
different questions.
Matched-campaign result on the honest metric (3 campaigns, shared
transitions only): add 1 assertion lost, spec-kit 11.
Tests: 26 new, all five/three mutants killed per function. Two mutants
survived the first draft (the assertX capital-letter guard and the trivia
exclusion) and two rename probes passed vacuously on fixtures named `t.py`
that the detector never collected — both fixed before the green was taken.
author: Tin Dang
…hind an enumerative well-formedness guard The hard-case track built 2026-07-15 on feat/bench-hard-cases never ran against a real arm. It forked at 40c5548, before ten meter fixes landed, and its diff touches 113 files — but all 60 conflicts are in the ADD engine, which main has since superseded. Only the 35 workload files are ported. RENAMED hv -> pay. `benchmark/tests/test_prune_deadweight.py` guards a human decision of 2026-07-10: the WV2 hostile track (hv1-hv3) was run at n=1, returned "NO ARM GAMED — hypothesis fails at this temptation strength", and was pruned as dead weight. That track was the BOOKINGS domain, a base pair for wm1. The branch's hv1-4 is an unrelated payments ledger that merely reuses the family prefix. Retiring the guard to make the port fit would have erased a standing decision to make a build pass; renaming the new track costs nothing and keeps both. The `family` seam is a directory prefix, so the rename needed zero engine changes. Fixed on the way in: 8 sites across pay1/pay3/us1 asserted a BARE JSON array for list endpoints (`isinstance(body, list)`), the same envelope-shape defect already fixed three times elsewhere. The enumerative collection-shape guard caught all 8 the moment the files landed — the guard earning its keep on code written before it existed. NEW test_every_track_wellformed.py. test_oracles_red.py is parametrized over [1, 2, 3], so wm4-wm6, amb1 and the entire payments track were never checked for the defect it exists to catch: an oracle that passes on an EMPTY workspace measures nothing. Enumerating workload/* instead covers a track the day it lands. All 12 tracks confirmed red-on-empty with unique requirement ids; suite 485 green. NOT RUN. A pay1-4 head-to-head is a spend decision, not a code decision. author: Tin Dang
…assertions_lost beside tests_weakened
Two defects found while preparing the pay1-4 launch, both the same shape as
bugs already fixed this session.
1. `--family` carries a hardcoded `choices=("wm", "hv", "amb")` in pilot.py
AND report.py, with nothing tying either to benchmark/workload/. The pay
and us tracks existed on disk and were unrunnable. This is the
`unknown_arm` defect again: a second allowlist that drifts from the real
source, invisible to a green suite, discovered only by paying for a launch
that fails on argument parsing. test_family_choices_cover_disk.py
enumerates the directories so the NEXT track is covered too, and pins the
two CLI surfaces to each other — a family runnable but not reportable is
the same trap one step later. `hv` stays an accepted choice with no
directory: the pruned track's archive still scores through
`report --trust --family hv`.
2. `compute_assertions_lost` existed but nothing called it, so a pay campaign
would have recorded only the rename-blind count and needed post-hoc
recomputation. Both counts now come from one seam, `_add_tamper_metrics`,
written from the same snapshot pair under the same guards. Absent
snapshots yield NEITHER key rather than 0.0 — "nothing was lost" and
"nothing was measured" must not be the same record.
Tests: 8 new, red first. Suite 493 green.
author: Tin Dang
… and gsd both ran raw The arm labelled `spec-kit` ran `specify init --here --force` and was then handed the bare workload prompt (`prompt_wrapper = "raw"`). Across every automated campaign it produced ZERO `specs/` artifacts — no spec, no plan, no tasks. `gsd` carries the identical shape. Both were, in effect, `vanilla` plus a directory of unopened templates, and every comparative number this benchmark has published was "ADD vs a competent agent with no method" wearing a method's name. A second, compounding defect: `specify init` without `--integration` "default[s] to Copilot in non-interactive sessions", so the slash commands landed in `.github/prompts/` — a directory the runner's agent never reads. Every archived spec-kit workspace carries `"ai": "copilot"` in `.specify/init-options.json`. The arm had no path to its own method at all. 493 tests were green throughout. `test_arms.py` checks pins and fairness fields; nothing compared an arm's setup_steps against its prompt_wrapper. FIXED - spec-kit installs `--integration claude` and drives its own documented cycle through a new `spec-kit` wrapper: specify -> plan -> tasks -> implement. - gsd declares `scaffold_only = true` with a written reason, because its command surface is unverified for this runner. Declared beats silently implied; fixing it properly is its own task. - vanilla is untouched — `setup_steps = []` is scaffolding-free by construction. GUARD (the actual deliverable) enumerates `arms/*.toml` rather than naming arms, so an arm added later is covered the day it lands: scaffolding + raw + no scaffold_only -> `silent_method_arm`; scaffold_only without a reason -> `unexplained_scaffold_only`. ADD: task arm-honesty under milestone direction-velocity, frozen @ v1, gate PASS, target-hit yes. Red first — the new suite named both offenders before any fix. Suite 493 -> 514 green. author: Tin Dang
…er version A scored number you cannot date is a number you cannot trust. The archived campaigns hold records produced by at least three different meters — before the reading-probe fixes, after them, and after `tests_weakened` learned to read unittest assertions — and nothing on a record said which. Telling them apart meant reading git history and matching dates by hand. benchmark/meter.py derives the stamp from the CONTENT of the modules that decide a score (score.py, tamper.py, ambiguity.py, trust.py, workload/_oracle_lib.py), so editing a scorer moves the stamp without anyone remembering to bump it. A hand-maintained constant is the `turn_ceiling` failure mode: declared in every arm, asserted equal by a test, read by nothing. A missing module raises rather than hashing nothing, so renaming a scorer fails loudly instead of drifting the stamp toward a constant. The stamp rides in `artifacts` because `REQUIRED_ARTIFACTS` is a subset check while `REQUIRED_METRICS`/`OPTIONAL_METRICS` are a closed key-set. Second defect, same concern: test_collection_shape.py enumerated workload/*/checklist.py and was blind to workload/*/oracle/*.py. Six live oracle surfaces asserted a bare JSON array for collections whose workload prompt never fixed the serialization — a probe encoding an unstated preference distorts scores exactly as a misdated meter does. The guard now yields both globs; all six are fixed with the existing envelope-tolerant `records()` reader rather than an allowlist. Task meter-provenance: frozen @ v1, gate PASS, 520 tests green. author: Tin Dang
The pay1-4 campaign (2026-07-26) folded ADD's own transcripts into a call-stack flamegraph: across 209 direction turns, ZERO emitted more than one tool call. 7.3 of direction's 31 minutes went to a strictly serial chain of Reads, each paying a full turn's context for a single file. The guide had already mandated "ONE silent draft" for the WRITE side of direction and said nothing at all about the READ side. `### Grounding` now carries a "Batch the sweep" clause: independent reads, greps and listings go out in ONE turn, with the reason stated, and chaining reserved for the case where the next path is genuinely unknown until the previous result lands. `BATCH_CLAUSE` carries the same instruction to the arm under measurement, so the benchmark scores the method people actually run rather than a private variant — the dishonesty class `arm-honesty` closed on the comparison side. It reaches `add-loop`/`add-loop-enumerate` ONLY; `raw` and `spec-kit` are asserted clause-free, because lifting the controls along with the treatment destroys the comparison in the other direction. It also tells the agent to skip harness bookkeeping (task-tracker calls, sleep-polling) — 3.9 further minutes that deliver nothing to the workload. Two guards, both enumerative: the skill-tree guard globs the repo layout rather than naming the three trees, so a fourth is covered the day it lands (three hand-mirrored trees with no parity test is the lock-reclaim failure class, and it has already cost this project a publish); an empty-root test proves the enumeration cannot pass vacuously. What this does NOT claim: that a prose clause changes tool-call shape. That is Gate A's measurement, one re-run away, and the §3 least-sure flag says so. Task read-batching: frozen @ v1, gate PASS, 2868 tests green across both floors. author: Tin Dang
…e call The pay1-4 flamegraph fold (2026-07-26) measured 4.9 of direction's 31 minutes spent building a single PLAN.md through 45 successive Edits — 45 round-trips to write one file. `advance --fill` already batched ONE section for ONE crossing; a direction bundle is three sections and a freeze. `draft --from <bundle>` writes §1 + §3 + §4 in one call, all-or-nothing. It reuses `_fill_and_advance`'s snapshot-and-restore verbatim in structure: any refusal on any path — SystemExit from `_die` included — restores the original bytes before re-raising, so a rejected draft never leaves a half-written bundle. The bundle is delimited by its own `## <n> ·` headings and parsed by the ONE canonical scan, so no second grammar enters the codebase. `--run-red` is the harder half. "The red suite ran red before the build" has always been a discipline the agent ASSERTS; the engine now OBSERVES it, and a suite that PASSES at freeze time is `red_suite_green`, a refusal. Design for failure: an argv list and never `shell=True`, bounded by `--red-timeout` (default 300s), and NO retry — a suite run is not idempotent-cheap, so a retried timeout could double whatever side effect hung it. Anything that prevents an honest verdict is a refusal, never a silent skip. `--freeze` chains cmd_freeze rather than reimplementing it, so contract_not_drafted, unflagged_freeze, boundary_unfilled and the scope floors all still decide. `advance --fill` is untouched and keeps its own contract. Three corrections the plan did not foresee: section bodies bound at the next `## ` OR bare `---` (heading-to-heading would have swallowed the rules between sections); `_bundle_sections` keeps out-of-range headings so a §9 is REFUSED rather than dropped; and `engine_pin.py` is a four-way twin in its own right. The last was caught by test_tree_parity, and test_corpus_slim's census caught that a twin-parity test of mine duplicated a strictly wider existing sweep. Both were removed rather than accommodated — no ceiling was raised. Skill cookbook gains the verb at a NET -118 bytes, funded by compression. Engine mirrored to all four twins; ENGINE_MD5 re-aimed (ENGINE_PKG_MD5 unchanged — add_engine/ was not touched). GATE PENDING: the task declares `risk: high`, and the engine refused the auto-gate (unguarded_high_risk_auto) — a human owns this one. 2882 tests green across both floors. author: Tin Dang
risk: high routed this gate to a human, and the engine refused the auto-path (unguarded_high_risk_auto) exactly as designed. The scope tripwire then caught the commit-message file under tmp/ — a consumed, gitignored transient genuinely outside the task's write-set, so it was reverted rather than covered by widening §5 Scope to include bookkeeping. Gate: PASS, reviewed by Tin Dang, target hit. 2882 green across both floors. Three lessons filed: a shelling-out verb needs argv/timeout/no-retry · an anti-sprawl census that refuses a new test is usually right (delete the duplicate, never raise the ceiling) · a fixture assertion must anchor the way the parser anchors it. author: Tin Dang
…unproven ones PROJECT.md carries project-wide `invariants:` binding every task, but a TASK had no way to publish one of its own — so a downstream node inherited nothing, and "the payout timeout is bounded" lived in prose no build was answerable to. §3 gains an OPTIONAL `Invariants (published):` block. Each entry must cite a proving test — `(proof: `path::test_name`)` — and the freeze refuses one that does not, naming which invariant is unproven. A citation whose file does not resolve under the project root is refused identically: a citation to nothing is the same failure wearing a citation, and this project has already paid once for `turn_ceiling` — declared in every arm, asserted equal by a test, read by nothing. An invariant no test can fail is a comment. Grandfathered by absence, copying boundary_unfilled's shape exactly: a §3 with no block freezes as it does today, and the template's line carries a placeholder so it deliberately does NOT match the parser — a freshly scaffolded task still freezes untouched. The floor is opt-in; opting in is what binds you. Path resolution is fail-closed with is_relative_to, mirroring _declared_scope. Slotted after unflagged_freeze so the cheaper structural refusals fire first; validate-then-write, so a refusal writes nothing. DOGFOOD: this task's own direction bundle was authored by `add.py draft --from ... --run-red --freeze --cross` — the verb its predecessor shipped. One call ran the suite, observed it RED, wrote §1+§3+§4, froze, and crossed to build. The grandfather test earned its keep immediately: it caught that four sibling tests were passing on contract_not_drafted rather than on the floor under test, because the fixture only partially replaced the template. Engine mirrored to four twins, template to four twins, ENGINE_MD5 re-aimed. 2892 tests green across both floors. Gate PASS. author: Tin Dang
…t a store A published invariant needs a consumer, or publishing one is pointless. `new-task --depends-on <a>` now prints every invariant the new node inherits, attributed to the node that owns it, walking the TRANSITIVE depends_on closure so a grandparent's invariant binds too. It is a VIEW. Nothing is copied into the new task's PLAN.md and no key enters state.json — the moment an inherited invariant is copied it can drift from the ancestor that owns it, and the copy is what the builder reads. `graph --signals` already settled this project's answer: the graph is a view, not a store. Fail-soft: an ancestor whose PLAN.md is missing or unreadable contributes nothing and never blocks the new task. A broken neighbour must not stop a healthy new node, and the guard for an unproven invariant lives at the ancestor's own freeze. An empty inheritance prints nothing at all — a header with no rows is noise on every task that inherits none. TWO DEFECTS OF MINE, both caught by existing guards rather than by me: 1. for/else. The print loop first landed between the `if milestone:` body and its `else:`, which binds the else to the FOR — it then fired on every loop completing without break, so new-task printed BOTH "linked to milestone" and the orphan nudge. Valid Python, silently wrong; test_v8_1_orphan_guard caught it. The loop now sits after the whole if/else with a comment saying why it must. 2. Fresh-checkout twins. invariants-publish's template guard demanded all four twins EXIST, but two are gitignored dogfood trees absent from a fresh checkout. It passed locally and failed only in test_ci_tooling_mirror_gap's nested run. Now follows test_tree_parity's exists-skip convention: the git-tracked pair must always be present and identical, a dogfood twin is compared only if it is there. That guard had already gated, so this is a post-gate repair to a shipped guard, recorded as a TDD delta. Direction bundle authored by `add.py draft --run-red --freeze --cross`. Engine mirrored to four twins, ENGINE_MD5 re-aimed. author: Tin Dang
Tamper tripwire fired first: §4 declares `Tests live in: add-method/tooling/`, which makes the whole engine directory a protected test surface, so the legitimate build edits to add.py and engine_pin.py read as tamper. Resolved via the sanctioned re-cross, and filed as a delta — three engine tasks in this milestone have now each paid that toll. author: Tin Dang
…ts DESIGN.md PLAN.md deliberately persists the interface and nothing else — "reason everything else in-context, don't write essays". That is right for most tasks and wrong for exactly one kind: a node that publishes an invariant its dependents inherit. Someone will later be told "you must not break this" with no way to learn why, and the reasoning is gone with the conversation that had it. So the floor is proportional rather than universal: if you bind your neighbours, you leave them the reasoning. Publish nothing and nothing changes — a non-publisher gates exactly as before and no DESIGN.md is created for it. "Every task writes a design doc" is the ceremony this method spent whole milestones removing. Written at BUILD, not direction, on purpose: the design that survives is the one the implementation actually took. Demanding it before the build is how you get a document describing a plan nobody followed. design_missing names the path to write. design_empty closes the touch-a-file bypass — a touched file is not reasoning. Both fire after _tamper_guard, so a cheat still outranks a missing document, and both apply to RISK-ACCEPTED as well as PASS: a waiver must not launder a missing design. Both reads are fail-soft; an unreadable file never becomes a traceback at a gate. What this does NOT do is judge the reasoning. A DESIGN.md saying "TODO" passes, and §1's assumption says so: imposing a shape on prose is the move that produces filled-in templates instead of thought. The floor makes an omission visible at a human seam; if audits later show empty designs, the fix is a reviewer prompt in the gate report, not a stricter parser. This task declares `Tests live in:` as the FILE rather than the directory, and gated with no tamper trip — confirming the lesson invariant-inherit paid a re-cross to learn: naming the engine directory as the test surface makes every legitimate build edit read as tamper. 2908 tests green across both floors. Gate PASS. author: Tin Dang
…fore the gate §4's Rigor rule already separates GATED rows from prose build-guidance, and it has one hole: an author can ENUMERATE an edge case — visibly promising it — and never write the test. The row reads as coverage to every later reader and nothing ever disagrees with it. Enumerating a case you do not cover is worse than not enumerating it, because it buys credit for work that was not done. At a completing gate, every `[edge]`-tagged row must name a test present in the §4-declared suite, or carry `[edge — waived: <reason>]`. A blank waiver is refused: the escape hatch costs a sentence. Both refusals apply to RISK-ACCEPTED as well as PASS, and both fire after the tamper and design floors so a cheat still outranks a bookkeeping gap. Deliberately narrow. `[GATED]` rows are NOT policed here — green-ness is the §6 evidence floor's job, and double-enforcing would refuse on a naming mismatch. A §4 with no `[edge]` rows is untouched. An unreadable declared suite contributes no names and never tracebacks at a gate. What it CANNOT do, stated in §1 rather than implied: the engine does not run the suite at gate time, so presence of the test name is the mechanical proxy for "covered". A `def test_x(): pass` stub clears this floor — that is the cheat class the earned-green refute-read exists for. This closes enumerate-and-forget, nothing wider. DOGFOOD: this task's own §4 carries three [edge] rows, all written, so its gate ran through the floor it ships. Also folds design-at-build's PLAN.md read into ONE read shared by both floors — the first draft re-read the file for §4 behind a walrus and a stray `or True`. 2917 tests green across both floors. Gate PASS. author: Tin Dang
The two open criteria both need a run, not code: - read-batching's clause ships and is guarded across three skill trees, but "the parallel-turn count is greater than zero" is a MEASUREMENT — it needs a pay1-4 re-run (Gate A) against the 2026-07-26 baseline. - algo-workload has not been started; it needs a paid head-to-head campaign. Ticking either without the run would be exactly the dishonesty arm-honesty and meter-provenance were built to stop. author: Tin Dang
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| from __future__ import annotations | ||
|
|
||
| import os | ||
| import pathlib |
| from benchmark import judge, tamper | ||
| from benchmark.ambiguity import is_implementation_write | ||
| from benchmark.arms.loader import ARM_NAMES | ||
| from benchmark.arms.loader import ALL_ARM_NAMES, ARM_NAMES |
…ution unearned Gate A re-ran pay1-4 on the add arm (n=1) against the 2026-07-26 baseline: $22.15 -> $9.94, 326 -> 163 turns, and R-payout-timeout-bounded — the one requirement ADD missed in the baseline — is now covered. 18/18, zero regressions, tests_weakened and assertions_lost both 0. Three things this does NOT establish, recorded so nobody reads the table as more than it is: 1. `add.py draft` was called ZERO times. The add-loop wrapper names `add.py freeze` and never learned the new verb, so direction-one-shot — the change this milestone rests on — remains unmeasured. That is arm-honesty pointed inward. 2. The parallel-turn exit criterion is UNMEASURABLE in this harness. A census of 138 transcripts / 7560 assistant turns — every arm, every campaign ever recorded — finds EVERY turn carrying exactly one tool_use. The '0 of 209 direction turns parallelized' finding that motivated read-batching was an instrument artifact, not a fact about ADD. The criterion is marked [~] with the census rather than ticked or silently dropped. 3. Direction's share of turns is not reported at all: the fold's seam detector never resets after the first freeze, and the campaigns used different verbs, so the 18.7%->54.6% reading is an artifact. What Gate A does support: Agent 8->0 and SendMessage 7->0, matching BATCH_CLAUSE's skip-harness-bookkeeping half. The rest is confounded at n=1. author: Tin Dang
Collaborator
Author
Gate A — 2026-07-27 (
|
| metric | baseline 2026-07-26 | Gate A | |
|---|---|---|---|
| cost, 4 WMs | $22.15 | $9.94 | −55% |
| assistant turns | 326 | 163 | −50% |
| requirement coverage | 17/18 | 18/18 | R-payout-timeout-bounded now covered |
| oracle pass rate | — | 0.9 / 1.0 / 1.0 / 1.0 | 0 regressions |
| Read / Edit calls | 64 / 87 | 26 / 44 | |
| Agent / SendMessage | 8 / 7 | 0 / 0 | matches BATCH_CLAUSE's skip-bookkeeping half |
Cost and coverage are real. The causal story is not earned, and three limits are recorded in MILESTONE.md rather than left implicit:
-
add.py draftwas called ZERO times. Theadd-loopwrapper namesadd.py freezeand never learned the new verb exists, sodirection-one-shot— the change this milestone rests on — is still unmeasured. This isarm-honestypointed inward, and it must be fixed before Gate B or Gate B measures nothing either. -
The parallel-turn exit criterion is unmeasurable in this harness. A census of 138 transcripts / 7,560 assistant turns — every arm, every campaign ever recorded — finds every turn carrying exactly one
tool_use. The "0 of 209 direction turns parallelized" finding that motivatedread-batchingwas an instrument artifact, not a fact about ADD. The criterion is marked[~]with the census attached rather than ticked or quietly dropped. -
Direction's share of turns is not reported. The fold's seam detector flips at the first
freezeand never resets, and the campaigns used different verbs (9 freezes vs 4, plus 3advance), so the 18.7%→54.6% reading is a detector artifact.
n=1, against measured within-arm variance (41/36/85/47 direction turns across four milestones) that already exceeds this effect — and the engine gained four gate floors in the same window. The comparison arm is also still un-rerun, so no ADD-vs-spec-kit claim is refreshed here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The pay1–4 campaign (n=1, 2026-07-26) produced a result worth taking seriously: spec-kit shipped the payments track perfectly at 1/5 of ADD's cost ($4.20 / 12.7 min vs $22.15 / 67.8 min). Folding ADD's own transcripts into a call-stack flamegraph located the spend — direction is 38.7% of billed context and 45.7% of wall-clock, and only 6.8 of those 31 minutes is reasoning. The rest is structure: 7.3m of strictly serial Reads, 4.9m building one PLAN.md through 45 successive Edits, 3.9m of harness bookkeeping. Across 209 direction turns, ZERO emitted more than one tool call. The engine itself is 1.1%, so trimming verbs buys nothing.
Then the instrument turned out to be lying, in both directions.
Part 1 — the instrument
spec-kitnever ran spec-kit.prompt_wrapper = "raw", andspecify initwithout--integrationdefaults to Copilot in non-interactive sessions, so its commands landed in.github/prompts/where the runner's agent never looks. Zerospecs/artifacts in every automated campaign. 493 tests were green throughout, because nothing compared an arm's setup against its wrapper. Every comparative number published was "ADD vs a competent agent with no method", labelled as a method.gsdcarried the identical shape.tests_weakenedcould only read pytest-style asserts — ADD's wm6 had 0 bare asserts and 189self.assertX, scoring 0.0 against spec-kit's 175/0.checklist.pyand was blind tooracle/*.py.meter_versionis now an md5 over the scoring modules' content, because a version someone must remember to bump is theturn_ceilingfailure mode.Part 2 — the
direction-velocitymilestone (8 of 9 tasks, all gated PASS)read-batching— the guide mandated "ONE silent draft" for the write side of direction and said nothing about the read side. All three skill trees now carry a "Batch the sweep" clause, andBATCH_CLAUSEreaches the measured arm so the benchmark scores the method people actually run.rawandspec-kitare asserted clause-free.direction-one-shot—add.py draft --from <bundle>writes §1+§3+§4 in one call, all-or-nothing.--run-redmakes the engine observe the red suite; a suite that PASSES at freeze time is a refusal. Argv list, nevershell=True; bounded timeout; no retry — a suite run is not idempotent-cheap.invariants-publish·invariant-inherit— a §3 may publish invariants, each citing a resolvable test or the freeze refuses; dependents inherit them as a view, never a copy.design-at-build·edge-rigor— a node that binds its neighbours owes them a DESIGN.md; an enumerated[edge]row must resolve to a real test or carry a stated reason.The last four tasks were authored by
draftitself — one call per direction phase, against the 45-Edit baseline.What this does NOT claim
Two exit criteria are deliberately unticked, because both need a run and not code:
direction-one-shotcarries the milestone.algo-workloadis not started.No cost claim in this PR is re-measured. The comparison arm is only now honest, so the next campaign is the first one whose numbers are publishable.
Evidence
2917 tests green across
add-method/tooling/andbenchmark/tests/, including the nested fresh-checkout run. Four engine twins byte-identical,ENGINE_MD5re-aimed.direction-one-shotdeclaresrisk: high; the engine refused its auto-gate and it was recorded by a human.Four defects in this work were caught by existing guards rather than by me — a
for/elsethat madenew-taskprint the orphan nudge for every attached task, a template guard demanding gitignored dogfood twins exist, a duplicated parity assert, and an unmirroredengine_pin.py. The duplicates were removed; no ceiling was raised.