From c45b2e420729c1890c78b51b0154d2b23b9efefc Mon Sep 17 00:00:00 2001 From: Darian Ngo Date: Sat, 25 Jul 2026 20:01:14 -0500 Subject: [PATCH] The pair parser reads JSON lists, and the flywheel repair is measured parsePredictedPairs matched parentheses only, so a run answering FINAL_ANSWER as a JSON list of lists -- the notation json.dumps produces -- scored 0.0 with every pair present and none counted. Observed on a live run: the answer was correct and the report called it a reasoning failure. Both bracket forms are now accepted; the pair shape still has to match, so no new class of text is admitted, and the 15 existing scoring tests stay green. The defect is worth more than the run that surfaced it. It had nothing to do with the module under test and would have misread any future benchmark run answering in valid JSON. The measured result for the spatial-flywheel provenance repair (#194) lands in PROVENANCE_CITATION_AB_REPORT.md rather than in a pull request body, which is rule 12's fourth resting place. One cold OOLONG-Pairs query per arm, identical graph, arms differing in one file: the spatial-flywheel addendum before and after the repair. Control refused 24 of 24 cited hashes and cached 0 insights. Treatment refused none and cached 220. Rescored with the fixed parser, F1 goes 0.800 to 1.000. The control discriminated, so this is a finding rather than noise. Two figures, per rule 7: estimate $0.25 printed before the first paid call, actual $0.4591. The overage is a blind first arm -- it returned F1 1.0 with zero sub-LLM calls, because the freshly ingested corpus carried all 220 categories and the module short-circuits when q.category is set, so nothing classified, nothing wrote, and the guard never fired. Its output was noise and the record says so. Standing is stated rather than implied: n=1 per arm establishes that the pre-repair write is refused and the repaired one is not, on the same instrument, and does not establish a distribution or a claim about answer quality, where one sample separates 0.800 from 1.000 and cannot say why. --- .../PROVENANCE_CITATION_AB_REPORT.md | 83 +++++++++++++++++++ src/benchmarks/oolong/scoring.ts | 11 ++- 2 files changed, 92 insertions(+), 2 deletions(-) diff --git a/docs/benchmarks/PROVENANCE_CITATION_AB_REPORT.md b/docs/benchmarks/PROVENANCE_CITATION_AB_REPORT.md index 4647ff1..5c3ea33 100644 --- a/docs/benchmarks/PROVENANCE_CITATION_AB_REPORT.md +++ b/docs/benchmarks/PROVENANCE_CITATION_AB_REPORT.md @@ -288,3 +288,86 @@ PAID, token-scoped (ingest + embed + spawn + judge + teardown), OOLONG price constants. The `module`/`hybrid` arms require the Appendix A module at `modules/provenance-citation-discipline/`. `scripts/exp_citation_metadata.ts` holds the metadata-copy variant. + +--- + +## The spatial-flywheel retrieval repair, measured (dated entry — July 26, 2026) + +**Claim:** the default protocol module prescribed a provenance write the default +guard refuses, and the repair at `41eff03` closes it. + + provenance writes refused: 24/24 hashes (control) → 0 (treatment) + insights cached: 0 (control) → 220 (treatment) + F1: 0.800 (control) → 1.000 (treatment) + · control discriminated · target: the run's writes are accepted + (docs/architecture/SELF_DESCRIBING_SURFACES.md, the T1 closure) + +### What was run + +One cold OOLONG-Pairs query per arm, city `dublin`, truth 6 pairs, through the +shipped path (`executeScoredQuery` → `/api/rlm-stream` → the RLM worker). Arms +differ in one file: `modules/spatial-flywheel/addendum.txt` at `41eff03^` +against `41eff03`. Version A against version B of a shipped artifact — the +rule-20-safe carve-out, not a new-versus-null baseline. + +Both arms faced an identical cold graph: 220 `:Question` nodes, **0** carrying +`category`, no `DERIVED_INSIGHT` edges, `sourceNodeIds` intact. Verified between +runs. + +### Results + +| | control (`41eff03^`) | treatment (`41eff03`) | +|---|---|---| +| provenance writes | **refused** — 24 cited hashes not retrieved | none refused | +| insights cached | 0 | **220** | +| F1 · precision · recall | 0.800 · 0.667 · 1.000 | **1.000 · 1.000 · 1.000** | +| sub-LLM calls | 1 | 5 | +| tool calls · iterations | 4 · 5 | 4 · 5 | +| cost | $0.1551 | $0.1979 | + +The control's refusal is the observation the code trace predicted: +`_verify_hashes_retrieved` admits only hashes in the run's retrieved set, and +`run_cypher` — where the module's step 1 obtains `sourceNodeIds` — feeds neither +the read nor the search bucket. The treatment's added step retrieves that +deduped union in one `get_ast_texts` call before any write, for one fetch of the +64-fetch budget at any catalog size. + +### Two figures, per rule 7 + +Estimate printed before the first paid call: **$0.25**. Measured actual: +**$0.4591** — $0.1061 + $0.1551 + $0.1979, all `gpt-5.4` through the RLM worker. +Coverage: the run's own telemetry, one process, no orphaned consumer (the queue +was empty and two 45-hour hung drills were killed before the stack came up). + +The overage is the first arm. It returned F1 1.0 with **zero** sub-LLM calls, +because the freshly ingested corpus carried all 220 categories and the module's +step 3 short-circuits when `q.category` is set — so nothing classified, nothing +wrote, and the guard never fired. A blind arm, and its output was noise +(rule 11). `oolong:flywheel-prep --confirm-strip` produced the real cold start. + +### A scorer defect this run exposed + +The treatment first scored **F1 0.000, 0 predicted pairs**, and that reading was +false. It had answered `FINAL_ANSWER: [["q_0013", "q_0050"], …]` — a JSON list +of lists, the notation `json.dumps` produces — while `parsePredictedPairs` +matched parentheses only. Every pair was present and none was counted. + +Any run answering in valid JSON scored 0.0 and was reported as a reasoning +failure. The parser now accepts both bracket forms; the pair shape still has to +match, so no new class of text is admitted. Rescoring the same logged answer +gives 6/6 and F1 1.000, which is the figure in the table. + +This is worth more than the arm that produced it: the defect had nothing to do +with the module under test and would have misread any future benchmark run. + +### Standing + +n=1 per arm. What is established is that the pre-repair module's write is +refused and the repaired module's is not, with the control discriminating on the +same instrument — not a distribution over runs, and not a claim about the +repair's effect on answer quality, where a single sample separates 0.800 from +1.000 and cannot say why. + +Stack: compose project `trellis-tests-approved`, ports 5443/7697/6389, isolated +from the `implement-trellis-flywheel-benchmark` containers that were running from +another worktree on the default ports and were not touched. diff --git a/src/benchmarks/oolong/scoring.ts b/src/benchmarks/oolong/scoring.ts index d334dbb..9cc16d8 100644 --- a/src/benchmarks/oolong/scoring.ts +++ b/src/benchmarks/oolong/scoring.ts @@ -56,12 +56,19 @@ export function cityTruth(dataset: OolongDataset, city: string): Set { return truth; } -// Extracts (q_x, q_y) tuples from FINAL_ANSWER and canonicalizes each to +// Extracts (q_x, q_y) pairs from FINAL_ANSWER and canonicalizes each to // (LOC id, HUM id) using the dataset's category index, so tuple ordering // mistakes by the agent don't mask a semantically correct pair. +// +// BOTH BRACKET FORMS. The module asks for a Python list of tuples and a run +// may answer with a JSON list of lists instead -- semantically the same pairs, +// in the notation json.dumps produces. Accepting only parentheses scored such +// an answer 0.0 and reported it as a reasoning failure; observed on a live run +// 2026-07-26, where every pair was present and none was counted. The pair shape +// still has to match, so widening the brackets admits no new class of text. export function parsePredictedPairs(finalAnswer: string, categoryOf: Map): Set { const predicted = new Set(); - const tupleRe = /\(\s*['"]?(q_\d+)['"]?\s*,\s*['"]?(q_\d+)['"]?\s*\)/g; + const tupleRe = /[([]\s*['"]?(q_\d+)['"]?\s*,\s*['"]?(q_\d+)['"]?\s*[)\]]/g; for (const match of finalAnswer.matchAll(tupleRe)) { const [, x, y] = match; if (categoryOf.get(x) === 'LOC' && categoryOf.get(y) === 'HUM') predicted.add(pairKey(x, y));