From 0ddad61aa08c1f8f115be37ba1f4635feef224b6 Mon Sep 17 00:00:00 2001 From: ojassug Date: Tue, 11 Aug 2026 13:39:58 +0530 Subject: [PATCH] docs+feat(audit): say what cannot be done, say what is checked, stop keeping two copies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the last three audit items — H2, M1 and M11 — which the audit filed as decisions rather than tasks because each had a legitimate "narrow the product" answer and a legitimate "build more" one. Reasoning, including the options refused, in DECISIONS §46. H2 — decided: report why, do not narrow the accepted set. Twelve of nineteen recognised extensions cannot produce a non-zero reduction under any flag combination, and that was indistinguishable from a file with nothing worth compressing. Rejecting `--language go` would be the stronger honesty signal and would also delete a working behaviour — pass-through is byte-identical and harmless. So every language is still accepted, and the run now says when elision cannot reduce it: `trace.languageSupport` carries `supported`, `unsupported`, `unsupportedLanguages`, `noneSupported` and a `reason`, and `validate()` raises an info issue, `LANGUAGE_NOT_ELIDIBLE`, that does not vote on the verdict. Same correction M5a made for budgets, one layer down. Three things this cost, all worth knowing. The predicate had to be derived from the gate rather than guessed: a first version asked "does the item yield symbols or markers?" and called Go supported, because a trivial Go file yields exactly one — `import:fmt`, an incidental match by the TypeScript import regex. The answer is exactly `supportsRegionElision`, since a symbol-bearing item cannot be elided whole (§43) and a symbol-free item's whole-item elision destroys every content marker and fails the same gate a step later. Measured, that predicts 3 of 17 probed languages — TypeScript, JavaScript, Python — which is the audit headline and the corpus agreeing independently. The field also had to be threaded through four separate whitelists that each enumerate their keys: `validate()`'s return, `createValidationReport`, `buildTrace` and `createOptimizationTrace`. Three dropped it silently, every time presenting as `trace.languageSupport: undefined` with everything else correct; the test asserts on the trace rather than on `validate()` for that reason. And a friendly CLI notice was written, then removed. The CLI prints the trace to stderr as a JSON document and consumers parse the whole stream — four of this repo's own tests among them. Prepending prose broke them. The explanation now lives inside the report as a `reason` field, so it is both machine-readable and readable and stderr stays parseable. M1 — decided: correct the documentation, do not wire the compiler API. The TypeScript "AST-lite validator" builds no AST; it is a lexer detecting unbalanced brackets and unterminated strings. Probed against the shipped code rather than taken from the audit — three audit claims in this project have failed that test (§40, §42, §45) — all of it reproduced: `const x = ;`, `import from "x";`, `let 123abc = 5;`, `const a = 1 +++++ 2;` and plain English prose all pass; only `super(; }` fails. Python is stronger and still passes prose; JSON is a real parser. `ts.createSourceFile` was refused on cost, not principle: `typescript` is a dev dependency today and promoting it to runtime costs install size and parse latency against a lexer running in single-digit milliseconds. Instead README and CLAUDE.md now say "bracket/quote integrity" and carry a per-language table of what each validator does and does not catch, and `test/unit/validator-guarantee.test.ts` pins every row as a characterization test — strengthen a validator and it fails on purpose, and the table moves with it. M11 — decided: retire the narratives and the root planning artifacts. Twelve files, 226 KB; markdown 31 files -> 19, markdown:src 1.40:1 -> 0.95:1. `docs/retired-documents.md` maps each file to where its conclusion lives and gives the `git show` command to read the original. The premise was stale and measuring it first changed what the decision was about. M11 was filed as 4.1:1; measured before acting it was already 1.40:1 — and not because the docs had shrunk (they had grown to 726 KB) but because src/ grew faster. Since 32.8% of src/ is comment prose, prose:code actually ran ~2.6:1. That reframes the finding: the problem is not bytes, it is two copies of an argument kept in sync by hand. In-source commentary is not that, and none of it was touched. Twenty-five source and test comments cite a retired document — the check the option called for, and the thing that nearly made this a bad change. They are marked `[retired]` rather than re-pointed: the citation names something git still holds, whereas re-pointing 25 citations at DECISIONS sections by hand would risk mapping some of them to the wrong place, trading a volume problem for a correctness one. CHANGELOG.md and DECISIONS.md keep their older citations untouched, each with a note saying why — they record what was true when written. Measurement: 574 of 574 corpus rows identical to the pre-change engine across 17 fields, same frozen corpus, varying only dist/. H2 refactored `selectElisionRegions` to derive its gate from a shared predicate, so this was a real risk rather than a formality. The corpus recipe moves again and both changes are mine: typescript 60 -> 61 (src/core/validation/language-support.ts) and prose 29 -> 18 (twelve documents retired, one added). `collect.js` refused on both before measuring anything. Suite: 599 passing (was 566), typecheck and lint clean. Every audit item is now closed. What remains is the architectural work in the status doc's §5, chiefly Phase 1c. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 49 ++ CLAUDE.md | 48 +- DECISIONS.md | 129 ++++ NOTES-FOR-DOCS.md | 675 ------------------- README.md | 35 +- ROADMAP.md | 11 +- docs/audit-remediation-status.md | 68 +- docs/issue-2-content-type-contract-design.md | 458 ------------- docs/phase-0-measurement-baseline.md | 209 ------ docs/phase-1-stabilization-summary.md | 415 ------------ docs/phase-1d-drift-investigation.md | 435 ------------ docs/phase-1d-granularity-design.md | 359 ---------- docs/phase-1d-semantic-gate-disposition.md | 313 --------- docs/phase-4b-lever-disposition.md | 287 -------- docs/phase-4b-pathless-code-scope.md | 546 --------------- docs/retired-documents.md | 56 ++ purposed architecture changes.md | 100 --- src/core/constraints/directives.ts | 2 +- src/core/elision/index.ts | 2 +- src/core/elision/regions.ts | 40 +- src/core/engine/index.ts | 3 + src/core/ledger/drift-tracker.ts | 4 +- src/core/model/constructors.ts | 12 +- src/core/model/types.ts | 47 ++ src/core/trace/index.ts | 4 + src/core/validation/index.ts | 26 + src/core/validation/language-support.ts | 82 +++ src/gateway/proxy.ts | 4 +- src/stages/compression/token-hashing.ts | 2 +- study.md | 101 --- test/integration/bench.test.ts | 2 +- test/unit/bench/m1_reverification.test.ts | 2 +- test/unit/bench/runner.test.ts | 2 +- test/unit/benchmark-harness-route.test.ts | 2 +- test/unit/constraint-prose-scope.test.ts | 2 +- test/unit/declared-language.test.ts | 2 +- test/unit/drift-unwitnessed-elision.test.ts | 2 +- test/unit/gateway.test.ts | 4 +- test/unit/language-support.test.ts | 140 ++++ test/unit/markdown-marker-allowlist.test.ts | 2 +- test/unit/python-content-probe.test.ts | 4 +- test/unit/validator-guarantee.test.ts | 93 +++ tokendamper-headroom-known-issues.md | 213 ------ tools/corpus-harness/README.md | 2 +- tools/corpus-harness/recipe.json | 6 +- tools/corpus-harness/seam2.js | 2 +- 46 files changed, 821 insertions(+), 4181 deletions(-) delete mode 100644 NOTES-FOR-DOCS.md delete mode 100644 docs/issue-2-content-type-contract-design.md delete mode 100644 docs/phase-0-measurement-baseline.md delete mode 100644 docs/phase-1-stabilization-summary.md delete mode 100644 docs/phase-1d-drift-investigation.md delete mode 100644 docs/phase-1d-granularity-design.md delete mode 100644 docs/phase-1d-semantic-gate-disposition.md delete mode 100644 docs/phase-4b-lever-disposition.md delete mode 100644 docs/phase-4b-pathless-code-scope.md create mode 100644 docs/retired-documents.md delete mode 100644 purposed architecture changes.md create mode 100644 src/core/validation/language-support.ts delete mode 100644 study.md create mode 100644 test/unit/language-support.test.ts create mode 100644 test/unit/validator-guarantee.test.ts delete mode 100644 tokendamper-headroom-known-issues.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 17d93be..46e7bbc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,11 @@ All notable changes to this project will be documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +> **Entries citing retired documents are deliberate.** Audit M11 retired twelve narrative files; +> older entries still name them, because a changelog records what was true at the time and +> rewriting it would falsify that. See `docs/retired-documents.md` for where each conclusion +> lives now and how to read the original out of git. + ## [Unreleased] Commits on `main` beyond the `v1.1.0` tag (`807f6f0`). Not yet tagged or released; run @@ -39,6 +44,20 @@ Commits on `main` beyond the `v1.1.0` tag (`807f6f0`). Not yet tagged or release fields. Wave 2 changes no stage output. See DECISIONS §44. ### Added +- **A 0% run now says whether the language could ever have been reduced — audit H2.** Twelve of + nineteen recognised extensions cannot produce a non-zero reduction under any flag combination, + and that was indistinguishable from a file with nothing worth compressing. `trace.languageSupport` + now carries `supported`, `unsupported`, `unsupportedLanguages`, `noneSupported` and a `reason`, + and `validate()` raises an **info** issue (`LANGUAGE_NOT_ELIDIBLE`) that does not vote on the + verdict. Every language is still accepted — pass-through is byte-identical, and refusing it + would remove a working behaviour to make a point. + + Measured, elision reduces **3 of 17** probed languages (TypeScript, JavaScript, Python), which + is the audit's headline and the corpus agreeing independently. The predicate is + `supportsRegionElision` and nothing looser: a first attempt asked "does the item yield symbols?" + and called Go supported, because a trivial Go file yields exactly one — `import:fmt`, an + incidental match by the TypeScript import regex. + - **`optimize` accepts multiple paths and directories — audit H5**: `tokendamper optimize a.ts b.ts` and `tokendamper optimize ./src`. This is what makes the 0/1 knapsack reachable: `createContextBundle` produced exactly one item for every shipping entry point, prefix locking @@ -60,6 +79,23 @@ Commits on `main` beyond the `v1.1.0` tag (`807f6f0`). Not yet tagged or release Phase 1c). This delivers the mechanism; §3.1 stands between it and the outcome. See DECISIONS §43. ### Removed +- **Twelve narrative documents, 226 KB — audit M11.** `NOTES-FOR-DOCS.md`, `study.md`, + `purposed architecture changes.md`, `tokendamper-headroom-known-issues.md`, and the eight + `docs/phase-*` / `docs/issue-2-*` files. Markdown drops from 31 files to 19, and markdown:src + from 1.40:1 to **0.95:1**. `docs/retired-documents.md` maps each file to where its conclusion + now lives and gives the `git show` command to read the original. + + **The 4.1:1 premise was stale**: measured before acting it was already 1.40:1, and not because + the docs had shrunk — they had grown to 726 KB — but because `src/` grew faster. Since **32.8% + of `src/` is comment prose**, prose:code actually ran ~2.6:1. The in-source commentary is + deliberately kept: the failure mode M11 names is two copies of an argument kept in sync by hand, + and a comment next to its code is not that. + + Twenty-five source and test citations to retired documents are marked `[retired]` rather than + re-pointed — the citation names something git still holds, and re-pointing 25 of them by hand + would risk mapping some to the wrong place. `CHANGELOG.md` and `DECISIONS.md` keep their older + citations untouched, each with a note saying why: they record what was true when written. + - **Three knobs that were parsed, validated and then read by nothing — audit H4.** `--max-output-tokens` and `--max-latency-ms` (with `TOKENDAMPER_MAX_OUTPUT_TOKENS` and `TOKENDAMPER_MAX_LATENCY_MS`) reached no consumer anywhere in the pipeline; @@ -219,6 +255,19 @@ Commits on `main` beyond the `v1.1.0` tag (`807f6f0`). Not yet tagged or release constant-time. ### Changed +- **The documented guarantee is "bracket/quote integrity", not "syntax validity" — audit M1.** + The TypeScript validator builds no AST; it is a lexer detecting unbalanced brackets and + unterminated strings. Probed against the shipped code, it passes `const x = ;`, + `import from "x";`, `let 123abc = 5;`, `const a = 1 +++++ 2;` and plain English prose, failing + only on `super(; }`. `README.md` gains a per-language table of what each validator does and does + not catch; `CLAUDE.md` says the same; `test/unit/validator-guarantee.test.ts` pins every row as + a characterization test, so strengthening a validator fails the test on purpose and the table + has to move with it. + + Wiring the real TypeScript compiler API was **refused on cost, not principle**: `typescript` is + a development dependency today, and promoting it to runtime costs install size and parse latency + against a lexer that runs in single-digit milliseconds. + - **Gateway test seams are parameters, not environment variables — audit M8.** `TOKENDAMPER_MOCK_UPSTREAM=true` made the proxy return the caller's own optimized prompt with a 200 as though a model had written it, and `NODE_ENV === 'test'` waived the missing-credentials diff --git a/CLAUDE.md b/CLAUDE.md index f44f755..30bb593 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,7 +7,16 @@ Context file for Claude Code. Read this before touching the codebase. TokenDamper is a **deterministic context optimization engine** for AI coding assistants (Claude Code, Codex, Gemini CLI, Aider). TypeScript, CommonJS, Node >=20.19, MPL-2.0. It sits between a developer tool and an LLM provider API and reduces token count while -preserving syntax validity and provider prompt-cache alignment. +preserving **bracket/quote integrity** and provider prompt-cache alignment. + +**Not "syntax validity" — audit M1, closed by measurement.** The TypeScript "AST-lite validator" +builds no AST; it is a lexer detecting unbalanced brackets and unterminated strings. Probed +against the shipped code, it *passes* `const x = ;`, `import from "x";`, `let 123abc = 5;`, +`const a = 1 +++++ 2;` and plain English prose, and fails only on `super(; }`. Python is +meaningfully stronger (missing colons, malformed `def`, bad dedent) and still passes prose; JSON +is a real parser and is correct. `test/unit/validator-guarantee.test.ts` pins every one of those +as a characterization test, and the README carries the same table — if you strengthen a +validator, that test fails on purpose and both need updating together. Three entry modes: **CLI**, **local Gateway HTTP proxy**, **MCP server (stdio)**. @@ -138,11 +147,11 @@ and takes precedence over budget-derived knapsack selection. from **114 of 264 files to 12** with **zero** prose casualties. - **Measured end state:** every uncovered-language bucket goes to **0.00%** reduction, and **258 of 258** rows in the AST-covered and prose buckets are **byte-identical** to - baseline. `docs/phase-0-measurement-baseline.md`, DECISIONS §33–§34. + baseline. `docs/phase-0-measurement-baseline.md`, DECISIONS §33–§34. [retired] - **The Gateway keeps within-payload dedup and loses cross-turn sole-copy dedup.** `resolveRecoverableElisions` substitutes recoverable elisions back before the gate runs, so they are structurally invisible to it. The lost case is the one §9 of - `docs/phase-1-stabilization-summary.md` already called a marker the model cannot resolve. + `docs/phase-1-stabilization-summary.md` already called a marker the model cannot resolve. [retired] - **Still open:** a symbol-free code file the **pruner** removes is invisible to drift (the `!after` branch is a deliberate exemption — selection is not elision); and `isCodeExtension` remains a hardcoded 19-entry list that decides whether a real source @@ -180,8 +189,10 @@ and takes precedence over budget-derived knapsack selection. > (this is invariant 10 applied to budgets, and H2 is the same question one layer down), and the > corpus A/B method in the status doc §2 is the one that caught its own two false greens. -Full detail in `tokendamper-headroom-known-issues.md`; proposed fixes in -`purposed architecture changes.md`. Summary: +Both documents that used to hold the detail here — `tokendamper-headroom-known-issues.md` and +`purposed architecture changes.md` — were retired by audit M11. Their live content is below and +in `docs/audit-remediation-status.md`; `docs/retired-documents.md` says how to read the originals +out of git. Summary: - **~~Gateway bypasses validation entirely~~ — FIXED (Phase 1.0b).** `src/gateway/proxy.ts` now routes through `core/engine.optimize()`, so validators, `DriftTracker`, @@ -244,7 +255,7 @@ Full detail in `tokendamper-headroom-known-issues.md`; proposed fixes in A case for splitting fallback may still exist — the success path's newline join is lossy for multi-item bundles, which is why the Gateway must map `finalBundle` positionally (invariant 9) — but it is a **different defect with different evidence**. Do not scope - Phase 1b from the -1.39%. See `NOTES-FOR-DOCS.md`. + Phase 1b from the -1.39%. See `NOTES-FOR-DOCS.md`. [retired] **Phase B settled both halves (DECISIONS §35).** The live one was neither: `rawInput` is a *decoded string*, so `readFileSync(path, 'utf8')` turned invalid bytes into U+FFFD before any stage ran and the fallback echo could not restore them — a Latin-1 `vimspell.sh` came back @@ -255,8 +266,9 @@ Full detail in `tokendamper-headroom-known-issues.md`; proposed fixes in Gateway bypasses `emittedOutput` — so it is pinned by `test/unit/fallback-render.test.ts` rather than fixed. - **Issue 3 / Phase 1d — investigated 2026-08-03, threshold unchanged, remedy undesigned.** - Full record: `docs/phase-1d-drift-investigation.md`; read §10 and §12 before citing any - benchmark number from it. **The threshold is not the defect; do not tune it.** + Full record was `docs/phase-1d-drift-investigation.md` (retired — `docs/retired-documents.md`); + its §10 and §12 carried the caveats on its benchmark numbers, so treat any figure quoted from + it as needing that context. **The threshold is not the defect; do not tune it.** - Bench reality at `targetReductionRatio: 0.30`: `avgReduction` **0.00%**, `fallbackRate` 0.40, all ten fixtures byte-identical. The 7.82% that briefly appeared was the estimator mismatch, not a saving (§12, DECISIONS §19). @@ -286,7 +298,7 @@ Full detail in `tokendamper-headroom-known-issues.md`; proposed fixes in is **retracted** — on re-run Headroom hit a 20-second backend timeout and failed open. Same 0%, different mechanism. Do not cite it as corroboration. - **The semantic gate was investigated 2026-08-04 and precondition (a) is disposed of: - `docs/phase-1d-semantic-gate-disposition.md`. Nothing implemented.** §18's proposed + `docs/phase-1d-semantic-gate-disposition.md`. Nothing implemented.** §18's proposed [retired] markers for code (brace balance, function/class boundaries, imports) are measured to be near-constants under the shipped selector — it preserves them by construction — so they would replace one decorative constant with four. Only comments and docstrings vary. @@ -312,7 +324,7 @@ The agreed direction is three scoped changes (no rewrite): dispatches per item, so a bundle-level tag would key the transform and the check at different granularities, reproducing Issue 2's shape while appearing to fix it. Bundles are heterogeneous (a 12 KB JSON tool result next to a one-line question), and - `statistics.contentTypeCounts` is already the bundle-level view. See `NOTES-FOR-DOCS.md`. + `statistics.contentTypeCounts` is already the bundle-level view. See `NOTES-FOR-DOCS.md`. [retired] The planner-level gate (§3.6 of the design doc) is still **not** implemented: nothing in `src/core/planner/` reads `contentType`. 2. Per-stage checkpointing replacing the single global validate→fallback gate (**Phase 1c, @@ -323,7 +335,7 @@ The agreed direction is three scoped changes (no rewrite): can originate in an item no stage touched (that is how DECISIONS.md §17 was found, on turn 1 with nothing transformed). Constraint retention and drift are also bundle-scoped set comparisons with no per-stage attribution. Establish attribution first. See - `NOTES-FOR-DOCS.md` and `docs/phase-1-stabilization-summary.md` §8. + `NOTES-FOR-DOCS.md` and `docs/phase-1-stabilization-summary.md` §8. [retired] 3. ~~Split fallback into **raw passthrough** vs. **bundle rendering**.~~ **DONE (Phase B, DECISIONS §35)** — though the live defect was not the one this item names. The fallback branch already returned `request.rawInput`; what made that *not* byte-identical was that @@ -358,7 +370,7 @@ scoring, MMR, AST folding and Prometheus metrics on top of a pipeline that curre Gateway path: the consumer is a stateless provider API with no rehydration mechanism, so elided content is deleted, not referenced. Do not infer the exemption from `elided` or `originalContentHash`; `token-hashing` sets both and must stay fully scored. - See DECISIONS.md §16 and the §16 entry in `NOTES-FOR-DOCS.md`. + See DECISIONS.md §16 and the §16 entry in `NOTES-FOR-DOCS.md`. [retired] - **This repo is its own corpus. Freeze it before measuring, or the measurement moves under you.** Every reduction figure in this project is measured over `src/**/*.ts` and the repository's own `*.py` — the same files a session edits while it works. A re-run after @@ -374,7 +386,7 @@ scoring, MMR, AST folding and Prometheus metrics on top of a pipeline that curre that converts fallbacks into reductions changes the denominator and can make a strictly worse rule look better on the mean); and the corpus is ~94% TypeScript, which is not a neutral sample for anything language-dependent — a docstring rule that costs 0.45pp here - costs 6.8pp on real Python (`docs/phase-1d-semantic-gate-disposition.md` §2). + costs 6.8pp on real Python (`docs/phase-1d-semantic-gate-disposition.md` §2). [retired] - **Classification has a blast radius over items no stage touched.** `validate()` runs `validateBundleAst` over *every* item in the final bundle, so changing what `classifyContent` returns can fail an item nothing transformed. To see it, measure @@ -413,6 +425,12 @@ scoring, MMR, AST folding and Prometheus metrics on top of a pipeline that curre Start here for anything audit-related; it is the doc kept current. `ARCHITECTURE.md` (canonical, frozen) · `ROADMAP.md` · `DECISIONS.md` · `CHANGELOG.md` · -`max_audit.md` (the audit itself — note several of its proposed fixes were measured wrong; see -§40 and §42) · `docs/architecture/milestone_*.md` · `docs/v1_deployment_audit.md` · +`max_audit.md` (the audit itself — note several of its *reachability* claims were measured wrong; +see §40, §42 and §45) · `docs/architecture/milestone_*.md` · `docs/v1_deployment_audit.md` · `tokendamper-benchmark/BENCHMARK_RESULTS.md` + +**`docs/retired-documents.md`** — audit M11 retired twelve narrative documents (~230 KB) whose +conclusions already lived in `DECISIONS.md` and the status doc. That file maps each one to where +its conclusion now lives and gives the `git show` command to read the original. Citations marked +`[retired]` in source comments and here point at documents in git history, not missing files — +they are still accurate about what was measured and when. diff --git a/DECISIONS.md b/DECISIONS.md index 1a476c1..5d7dcf9 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -4,6 +4,13 @@ This document records the architectural decisions behind TokenDamper. It should grow over time. Any future architectural change must update this document before implementation. +> **Citations to retired documents are deliberate and are not broken links.** Audit M11 retired +> twelve narrative files whose conclusions had already been folded into the entries below. Older +> entries still cite them by name, because those citations were accurate when the entry was +> written and this is an append-only record — rewriting them would falsify the history it exists +> to keep. `docs/retired-documents.md` maps each file to where its conclusion lives now and gives +> the `git show` command to read the original. + ## 1. Why immutable `ContextBundle`? ### Decision @@ -3017,3 +3024,125 @@ CRLF line terminators". So aggregate reduction figures on this repo are not comparable across a commit boundary, not only across a corpus-size change. Only the per-row A/B over one frozen corpus is. + +--- + +## 46. Three Decisions: Say What Cannot Be Done, Say What Is Actually Checked, Stop Maintaining Two Copies + +Audit H2, M1 and M11. The audit filed these as *decisions rather than tasks* — each had a +legitimate "narrow the product" answer and a legitimate "build more" answer, and the choice was +not the auditor's to make. Recorded here with the option taken and the option refused. + +### H2 — decided: report why, do not narrow the accepted set + +Twelve of nineteen recognised extensions cannot produce a non-zero reduction under any flag +combination. Two gates, both single-language-family and neither threshold-controlled: +`selectElisionRegions` returns `[]` outside TypeScript/JavaScript and Python, and a whole-item +elision has to survive a measurement gate that a Go `func` or a C function gives it nothing to +work with. + +**Refused: narrowing the accepted set.** Rejecting `--language go` would be the stronger honesty +signal, and it would also delete a working behaviour — pass-through is byte-identical and +harmless. Taking away something that works, to avoid saying something true, is the wrong trade. + +**Taken: say it.** `trace.languageSupport` carries `supported`, `unsupported`, +`unsupportedLanguages`, `noneSupported` and a `reason`; `validate()` raises an **info** issue, +`LANGUAGE_NOT_ELIDIBLE`, which does not vote on the verdict. This is the same correction M5a made +for budgets, one layer down: `reductionRatio: 0` cannot distinguish *"nothing was compressible"* +from *"nothing could have been"*, and only one of those is about the user's file. + +Three things this cost, all worth recording: + +**The predicate had to be derived from the gate, not guessed.** The first version asked *"does +the item yield symbols or content markers?"* — reasonable, and wrong. A trivial Go file yields +exactly one symbol: `import:fmt`, an incidental match by the TypeScript import regex. It +witnesses nothing about the function bodies, and Go still cannot reduce, but the predicate called +it supported. The answer is exactly `supportsRegionElision`, because every other elision route +terminates in a refusal: a symbol-bearing item cannot be elided whole (§43), and a symbol-free +item's whole-item elision destroys every content marker and fails the same gate one step along. +Measured, that predicts **3 of 17** probed languages — TypeScript, JavaScript, Python — which is +the audit's headline and the corpus baseline agreeing independently. + +**The field had to be threaded through four separate whitelists**, each of which enumerates its +keys: `validate()`'s return, `createValidationReport`, `buildTrace` and `createOptimizationTrace`. +Three of them dropped it silently, and the symptom every time was `trace.languageSupport: +undefined` with everything else correct. `test/unit/language-support.test.ts` asserts on the +**trace**, not on `validate()`, for exactly that reason. + +**A friendly notice line was written, and then removed.** The CLI prints the trace to stderr as a +JSON document, and consumers parse the whole stream — this repository's own integration tests +among them. Prepending prose broke four of them. The explanation now lives *inside* the report as +a `reason` field, which is both machine-readable and readable, and stderr stays parseable. A +channel with a contract is not improved by making it friendlier. + +### M1 — decided: correct the documentation, do not wire the compiler API + +The TypeScript "AST-lite validator" builds no AST. It is a lexer — a good one, tracking strings, +template interpolation, comments and regex literals — that detects unbalanced brackets and +unterminated strings, and nothing else. + +Probed against the shipped code rather than taken from the audit, since three audit claims in +this project have failed that test (§40, §42, §45). All of it reproduced exactly: + +| Input | Verdict | +|---|---| +| `const x = ;` | **PASS** | +| `function f(a: , b) { return 1; }` | **PASS** | +| `import from "x";` | **PASS** | +| `let 123abc = 5;` | **PASS** | +| `const a = 1 +++++ 2;` | **PASS** | +| `ceci nest pas du code` | **PASS** | +| `super(; }` | FAIL | + +Python is meaningfully stronger — missing colons, malformed `def`, bad dedent, stray leading +indentation — and still passes English prose. JSON is a real parser and is correct. + +**Refused: wiring `ts.createSourceFile`.** It would make the guarantee real, and `typescript` is +already a development dependency. It is refused on cost: promoting it to a *runtime* dependency +costs install size, and parsing costs latency against a lexer that runs in single-digit +milliseconds. That is a trade someone may want to revisit; it is not an oversight. + +**Taken: say what is true.** `README.md` gains a per-language table and the sentence that the +guarantee on TypeScript — the family where compression actually runs — is **bracket and quote +integrity**, not syntax validity. `CLAUDE.md` says the same. `test/unit/validator-guarantee.test.ts` +pins every row as a characterization test, so the documented guarantee is executable: strengthen +a validator and the test fails on purpose, and the README table has to move with it. + +Two consequences stated in the README rather than left implicit: a passing check is not a promise +the output compiles, only that it is no more unbalanced than the input; and real inputs are +frequently already invalid, which is why the sub-item check is *relative*. + +### M11 — decided: retire the narratives and the root planning artifacts + +Twelve files, **226 KB**, markdown from 31 files to 19. `docs/retired-documents.md` maps each to +where its conclusion lives and gives the `git show` command to read the original. + +**The premise was stale, and measuring it first changed what the decision was about.** M11 was +filed as a **4.1 : 1** documentation-to-code ratio. Measured immediately before acting, it was +**1.40 : 1** — and the improvement was not real: markdown had *grown* to 726 KB, while `src/` grew +faster. Worse, **32.8% of `src/` is comment prose** (165 KB of 518 KB), so counting honestly, +prose ran about **2.6 : 1** against code. The volume had not gone anywhere; some of it had moved +into the source files. + +That reframes the finding usefully. The problem M11 names is not bytes, it is **two copies of an +argument that have to be kept in sync by hand**. In-source commentary is not that — it sits next +to the code it describes and moves with it. Retired: the standalone narratives. Kept: every line +of the in-source commentary. + +**Twenty-five source and test comments cite a retired document**, which is the check the option +called for and the thing that nearly made this a bad change. They are marked `[retired]` rather +than re-pointed: the citation names a document and section that existed and that git still holds, +whereas re-pointing 25 citations at DECISIONS sections by hand would risk mapping some of them to +the wrong place — trading a volume problem for a correctness one. + +`CHANGELOG.md` and `DECISIONS.md` keep their older citations untouched, and each now carries a +note saying why. They are append-only records of what was true when written; editing them to +match today would falsify the history they exist to preserve. This entry is subject to the same +rule. + +### What was not done + +`cli/bench-table-renderer.ts:97` still prints a `risk` column sourced from `riskTolerance`, which +H4 established no stage reads. It is now the only reader of that field, and a benchmark column +implies the row's numbers depend on it. Small, real, and left alone here because changing what a +benchmark table reports is a measurement change, not a documentation one. diff --git a/NOTES-FOR-DOCS.md b/NOTES-FOR-DOCS.md deleted file mode 100644 index 7da1b89..0000000 --- a/NOTES-FOR-DOCS.md +++ /dev/null @@ -1,675 +0,0 @@ -# Notes for Docs - -Corrections to planning documents, recorded here rather than by editing those documents in -place. Each entry names the document, what it says, what is actually true, and why. - -Planning docs in this repo have repeatedly been read as current state after going stale. -Appending here keeps the original proposal intact as a historical record while making the -correction discoverable next to it. - ---- - -## `purposed architecture changes.md` — Change 1 specifies the wrong seam - -**Date:** 2026-08-02 -**Status of the correction:** approved, implemented in Phase 1a / Issue 2 - -**What the document says.** Change 1 ("Make content-type a first-class planning input") -specifies that *"`ContextBundle` should carry a content-type tag (e.g. `json`, `code`, -`prose`, `logs`) determined at ingestion."* - -**What is actually true.** The tag belongs on `ContextItem`, not `ContextBundle` — and it -already exists there. - -1. `ContentType` is defined at `src/core/model/types.ts:19` and carried on `ContextItem` at - `:53`. `classifyContent()` (`constructors.ts:392`) already sets it at ingestion. -2. `selectValidator()` — the function that rejects the corrupting placeholder — dispatches - **per item**, with precedence `language` → `path` extension → `contentType`. A - bundle-level tag would key the transform and the check at different granularities, which - is the exact mismatch that produced Issue 2. It would reproduce the bug's shape while - appearing to fix it. -3. Bundles are heterogeneous by construction. A Gateway bundle is N messages: one may be a - 12 KB JSON tool result, the next a one-line prose question. A scalar bundle tag forces a - choice between over-conservatism (skip the stage if *any* item is JSON) and corruption - (apply a JSON contract to prose). -4. `ContextBundle` already carries the correct bundle-level view: - `statistics.contentTypeCounts`, a `Record` histogram - (`types.ts:90`). A scalar `bundle.contentType` would duplicate and contradict it. - -**Consequent reframing.** Issue 2 is not "add content-awareness." The awareness exists: -`token-hashing.ts:75` copies `item.contentType` onto the item it produces and never reads it -to decide whether the transform is safe. The real defects are that four construction sites -bypass `classifyContent()` and hardcode a literal, that eliding stages never consult the tag -they already propagate, and that nothing binds how a stage elides to which validator will -judge the result. - -**Where the approved design lives:** `docs/issue-2-content-type-contract-design.md`. - ---- - -## `DECISIONS.md` §16 — the recoverable-elision rationale does not hold on the Gateway path - -**Date:** 2026-08-02 -**Status of the correction:** approved, implemented in Commit B of Issue 2 - -**What the document says.** §16 ("Recoverable References Are Not Semantic Drift") justifies -exempting `cleanup:session-dedup` elisions from drift on the grounds that *"a dedup marker -is a pointer: the full text is retained in the session store under `originalContentHash` and -is restorable on demand. Nothing is irrecoverably lost, so nothing should be scored as -loss."* - -**Where that holds.** On the MCP path, where `rehydrate_context` exists and a client can -actually resolve the reference. §16 stands there. - -**Where it does not.** On the Gateway path, which is the *only* path -`cleanup:session-dedup` currently runs on. The consumer there is a stateless provider API: - -- It has no rehydration mechanism and will never call `rehydrate_context`. -- Each request is independent, so content sent in an earlier turn is not available to the - model in this one. Prompt caching does not help — it reuses computation for identical - prefix bytes, and an elision changes those bytes. -- `rehydrateRefs` — the only input that triggers rehydration inside the stage — is never set - by any caller in `src/`. The rehydration branch is unreachable in the product. - -So content elided from an outbound payload is not pointed at, it is **deleted**. Cross-turn -elision of a sole copy is lossy compression wearing a pointer's clothes, and `DriftTracker` -scoring it 0.60 was correct all along. The exemption was granting a pass to precisely the -case that deserved scoring — the same shape as the hardcoded `fallbackUsed: false` that -Phase 1.0a removed: a safety property asserted without being evaluated. - -**The correction.** `recoverable: true` is now set only when an intact copy of the content -survives elsewhere in the **same outbound payload**. The stage preserves the first -occurrence of duplicated content so the copies after it reference something demonstrably -present in this request. That is a verifiable precondition rather than an assumed one. - -**Measured cost:** `docs/phase-1-stabilization-summary.md` §9. Cross-turn dedup on code and -logs drops from ~99% to 0% with a fallback; within-payload duplication still deduplicates at -~66%. The Gateway's near-term dedup value is likely close to zero on realistic traffic. - -**Not changed:** MCP behavior. `cleanup:session-dedup` only runs under `session_dedup` -planner mode, which only the Gateway sets, so the stage never executes on the CLI or MCP -paths. The change is path-agnostic in code and Gateway-only in effect. - ---- - -## `docs/phase-1-stabilization-summary.md` §9 — the predicted flip happened, plus one it missed - -**Date:** 2026-08-02 -**Status of the correction:** measured, implemented in Commit C of Issue 2 - -**What the document says.** §9 records the cross-turn `tool_output.json` row still reading -99.14% after Commit B, notes that this is the §2.2 vacuity rather than a surviving -exemption, and predicts: *"Commit C (the relabel) is expected to flip that row to 0% and a -fallback ... It is listed here so the change is attributable when it happens rather than -read as a regression introduced by the relabel."* - -**What happened.** The prediction held. Measured through the real proxy path, two turns per -session, same methodology as §9. **These are Gateway figures, derived from HTTP body byte -lengths — unaffected by the token-estimator unification (`1b1e999`), do not re-correct -them:** - -| Payload (cross-turn, sole copy) | Before Commit C | After Commit C | -|---|---|---| -| `tool_output.json` | 13,785 / 13,982 = **98.59%**, no fallback | **0.00%**, fallback | -| `codebase.py` | 0.00%, fallback | 0.00%, fallback | -| `sample_logs.txt` | 0.00%, fallback | 0.00%, fallback | - -Within-payload duplication is unchanged at ~66% with no fallback on all three, which is the -result that matters for the relabel being safe: it does not disturb the case where a -referent demonstrably survives in the same request. - -**What §9 did not anticipate.** The relabel also introduced a **false positive**, caught by -measurement rather than by the design: - -`validate()` runs `validateBundleAst` over **every item in the final bundle**, not only the -items a stage changed. So a newly computed tag can fail an item that nothing touched. With -`contentType` computed, a message quoting a code snippet classified as `code`, and -`selectValidator` maps `code` to the TypeScript validator — so on **turn 1**, where -`cleanup:session-dedup` has no previous block hashes and cannot elide anything, this fell -back: - - Here's the fix. It's the guard that's missing: - - ```ts - const a = 1; - ``` - -Three apostrophes leave an odd number of quote characters open and the message is rejected -as an unterminated string literal. The same message with one fewer contraction passes. - -That is fixed in Commit C1 (`DECISIONS.md` §17) by reclassifying fenced content as -`markdown`, which selects no validator. C1 landed **before** the relabel so the Gateway was -never in the regressed state. - -**The general lesson, worth carrying into per-stage checkpointing (Phase 1c).** A -content-type tag is not only an input to the transform; it is an input to the *check applied -to everything in the bundle*. Any future change to classification has a blast radius over -untouched items, and the way to see it is to measure turn 1 — where nothing is transformed, -so every failure is a false positive by construction. - ---- - -## `purposed architecture changes.md` — Change 2 assumes a validation failure is attributable to a stage - -**Date:** 2026-08-02 -**Status of the correction:** design input for Phase 1c, which is **not started** - -**What the document says.** Change 2 ("Replace the single global validate→fallback gate with -per-stage checkpointing") proposes: *"Validate incrementally after each stage in the Linear -Engine ... On a stage-level validation failure, roll back only that stage's transform and -keep the output of prior stages."* - -**What that assumes.** That a validation failure can be attributed to the stage that caused -it. For a class of failures it cannot, and this was demonstrated rather than theorized -during Issue 2. - -**1. A failure can originate in an item no stage touched.** `src/core/validation/index.ts` -runs `validateBundleAst(after)` over **every item in the final bundle**, not only the items -a stage changed. The fenced-prose defect in `DECISIONS.md` §17 was found exactly this way: -with `contentType` newly computed, a message quoting a code snippet failed the TypeScript -validator on **turn 1** of a Gateway session, where `cleanup:session-dedup` has no previous -block hashes and cannot elide anything. Nothing had been transformed. There was no stage to -roll back, and rolling one back would not have helped. - -**2. Two of the four checks are bundle-scoped, not item-scoped.** Constraint-directive -retention compares the `before` directives against all `after` item content joined into one -string; `DriftTracker` computes `S_k` from whole-bundle symbol and marker *sets*. Neither -produces a per-stage or per-item attribution as written. Drift in particular is a set -comparison — a symbol dropped by stage 2 and a symbol dropped by stage 4 are -indistinguishable in the result. - -**Consequence for the design.** "Roll back only the failing stage" needs a prior answer to -*which stage failed*, and for these cases the honest answer is "none of them" or "not -determinable." A checkpointing implementation that assumes attributability will roll back an -innocent stage and report a cause that is not the cause — the same class of error as the -hardcoded `fallbackUsed: false` that Phase 1.0a removed, and the vacuous JSON checks that -Commit C removed: a verdict asserted without being derived. - -The suggested first step is to establish attribution before building rollback on it — -validate the *delta* a stage produced rather than the whole bundle, and decide explicitly -what happens to a failure that predates every stage. Change 2's estimate that this "would -likely convert at least 2 of the current 0%-fallback cases into partial reductions" should -be re-derived afterwards; both cited cases (`tool_output.json`, `session.json`) have since -changed behavior under `b11dcb0` and `ac16cec`. - -**Also recorded in:** `docs/phase-1-stabilization-summary.md` §8 (1c), and `CLAUDE.md` as a -gotcha, because those are read. - ---- - -## Every CLI/bench/MCP reduction figure recorded before `1b1e999` was inflated - -**Date:** 2026-08-03 -**Status of the correction:** measured, implemented in `1b1e999` (DECISIONS.md §19) - -**What the documents say.** `CLAUDE.md` records under Gotchas: *"Token counting is currently -`content.length / 4`. It is an estimate, not exact."* Several records cite reduction figures -produced on the CLI or bench path. - -**What was actually true.** Token counting was `content.length / 4` **on one side of every -comparison and `EnhancedHeuristicTokenizer` on the other.** `createContextBundle` measured -the input bundle with the tokenizer; `core/trace`, `attemptAutomatedRehydration`, -`gateway/proxy.ts` and three of the five stages measured their output with -`Math.ceil(len / 4)`. The heuristic runs 11–22% above `len / 4` on this corpus, so -byte-identical output measured as an 11–22% saving. - -**Which figures this invalidates, and which it does not.** - -| Record | Status | -|---|---| -| `docs/phase-1d-drift-investigation.md` §10 `avgReduction: 7.8217%` | **Was already flagged as fabricated in that document.** Now 0.0000%. See §12 there. | -| `docs/phase-1d-drift-investigation.md` §10 per-fixture 9.89%–17.92% | Same. All ten fixtures were byte-identical; every figure is 0.00%. | -| `tokendamper-benchmark/BENCHMARK_RESULTS.md` `tokenBefore`/`tokenAfter` trace lines | **Invalid.** Every pair was the estimator gap on unchanged content — see below. | -| `tokendamper-benchmark/BENCHMARK_RESULTS.md` reduction percentages | **Valid.** `run_benchmark.py` counts both sides with `tiktoken` `cl100k_base` on the actual text, independent of TokenDamper's internal estimator. | -| `docs/phase-1-stabilization-summary.md` §9 (66.15%, 66.37%, 66.21%, 99.14%, 0.00%) | **Valid. Do not re-correct these.** Gateway path, derived from HTTP body byte lengths. The Gateway's internal counters do shift unit (`rawTokens` 8,470 → 10,059 on a 36 KB payload) but its `dedupRatio` moves only 49.79% → 49.82%, because both of its sides already used the same estimator. | -| `NOTES-FOR-DOCS.md` §"the predicted flip happened" (98.59%, 0.00%, ~66%) | **Valid. Do not re-correct these.** Same Gateway measurement, same provenance. | - -**The `BENCHMARK_RESULTS.md` trace lines specifically.** All four pairs are the heuristic -count of the input against the `len / 4` count of the *same* input: - -| Payload | Recorded `tokenBefore` → `tokenAfter` | Implied saving | Measured now | -|---|---|---|---| -| `sample_logs.txt` | 3029 → 2724 | 10.07% | **3029 → 3029**, fallback, output byte-identical | -| `tool_output.json` | 3974 → 3009 | 24.28% | **3974 → 3974**, fallback, output byte-identical | -| `codebase.py` | 5029 → 4235 | 15.79% | **5029 → 5029**, fallback, output byte-identical | -| `session.json` | 4679 → 4049 | 13.47% | **4679 → 4679**, fallback, output byte-identical | - -Re-measured through the real CLI at `--target-reduction-ratio 0.3`, build at `1b1e999`. - -> **Correction 2026-08-04 (4b.0), one row only.** Those four rows were measured the way the -> harness invoked the CLI at the time — bytes piped to `optimize -`. With no path the engine -> cannot resolve a language, so *all four* fell back; that is the pathless defect, not a -> property of the payloads. The harness now passes a path, and `codebase.py` changes: -> **5,029 → 3,310, no fallback** (27.61% by `cl100k`, 34.18% by the engine's own estimator — -> publish the former). The other three rows stand exactly as written: logs and JSON fall back -> on the path route too, for reasons unrelated to the path. Engine frozen at `95056df` across -> both runs, corpus frozen by `sha256` manifest. -> -> The "reduction percentages are **Valid**" row above (`cl100k`, independent of the internal -> estimator) remains true as a statement about *how* the harness counted — but every -> TokenDamper percentage it published before this date was measured on the wrong route and -> understates it. `BENCHMARK_RESULTS.md` has been regenerated. - -**Accuracy, separately.** `EnhancedHeuristicTokenizer` is named as though it improves on -`len / 4`. Scored against `cl100k_base` over the ten bench fixtures plus the four Gateway -payloads, it does not: mean absolute error **24%** against `len / 4`'s **17%** (max 56% vs -44%). That did not change which estimator was adopted — `TokenizerAdapter` is the seam a -real BPE tokenizer plugs into, and the planner already denominates knapsack weights and -1,024-token cache blocks in adapter units — but it is a real open item, and the fix is now a -one-line change to `DEFAULT_TOKENIZER`. DECISIONS.md §19. - ---- - -## `CLAUDE.md` Issue 5 — the −1.39% is a benchmark-harness artifact, not a fallback defect - -**Date:** 2026-08-03 -**Status of the correction:** measured against source; **Phase 1b is not yet rescoped** - -**What the document says.** `CLAUDE.md`: *"**Issue 5:** on fallback, `session.json` emits -**−1.39%** — output is *larger* than input. Fallback re-renders `currentBundle` instead of -echoing raw input bytes."* `docs/phase-1-stabilization-summary.md` §1b repeats it and scopes -Phase 1b around it. - -**What is actually true.** Two things, both checkable. - -**1. Fallback already echoes raw input bytes.** `src/core/fallback/index.ts` returns -`output: request.rawInput` when `validation.shouldFallback`. It is the **success** branch -that re-renders, joining `currentBundle.items` with newlines. Measured through the CLI at -`--target-reduction-ratio 0.3`, all four harness payloads fall back and all four emit output -byte-identical to their input: - -``` -sample_logs.txt : inBytes=10896 outBytes=10896 identical=true fallbackUsed=true -tool_output.json: inBytes=12036 outBytes=12036 identical=true fallbackUsed=true -codebase.py : inBytes=16937 outBytes=16937 identical=true fallbackUsed=true -session.json : inBytes=16193 outBytes=16193 identical=true fallbackUsed=true -``` - -**2. The −1.39% comes from the harness comparing two different strings.** -`tokendamper-benchmark/run_benchmark.py:75-77` special-cases session payloads: - -```python -is_session = file_name.endswith(".json") and "session" in file_name -if is_session: - messages = json.loads(raw_text) - orig_tokens = count_tokens(json.dumps(messages)) -``` - -`json.dumps` re-serializes and collapses the file's pretty-printing, so the "original" side -is measured on a **shorter string than the one TokenDamper was given**. TokenDamper receives -`raw_text` and, on fallback, echoes it back verbatim. Reproduced exactly: - -``` -raw file chars=16131 tokens=6285 <- what TokenDamper received and echoed -json.dumps chars=15785 tokens=6199 <- what the harness called "original" -=> reported reduction = -1.39% -``` - -That is `(6199 − 6285) / 6199`. The −1.39% is the pretty-printing the harness discarded, to -two decimal places. - -**Why this matters beyond the number.** It is the same defect class as the estimator -mismatch recorded above — a ratio whose two sides measure different things — one layer up, -in the Python harness. It is the seventh instance of the project's recurring pattern. - -**Consequence for Phase 1b.** Phase 1b is currently scoped as *"split fallback into raw -passthrough vs. bundle rendering"* on the strength of this figure. The figure does not -support it: the fallback path is already a raw passthrough. There may still be a case for -the split — the **success** path renders `items.join('\n')`, which is lossy for any -multi-item bundle and is why the Gateway must map `finalBundle` positionally instead -(invariant 9) — but that is a different defect with different evidence, and Phase 1b should -be re-derived from it rather than from Issue 5. **Not done here; flagged for decision.** - -**Not changed:** the `-1.39%` figure in `tokendamper-benchmark/BENCHMARK_RESULTS.md`. It is -an accurate record of what that harness printed; the correction is to its interpretation. - ---- - -## Two defects found while implementing Phase 1d granularity, neither caused by it - -**Date:** 2026-08-03 -**Status:** one fixed (`20ac438`), one open - -### 1. `TypeScriptValidator` has no regex-literal mode — **fixed 2026-08-04, DECISIONS §26** - -> Closed by adding a regex-literal mode. The count below was **7 of 64**, not the three files -> named — `src/cli/diff-renderer.ts`, `src/cli/html-reporter.ts` and -> `src/core/ledger/drift-tracker.ts` were also rejected. All 64 `src/` and 46 `test/` sources -> now pass, pinned by a corpus test. The paragraph on `scanBraceSpans` still holds and is now -> a deliberate duplication: the validator copies that scanner's punctuation rule rather than -> sharing code with it, because `scanBraceSpans` decides what the product removes. - - -**What the code implies.** `src/core/validation/ast/ts-validator.ts` presents itself as a -bracket/quote/comment scanner sufficient to judge TypeScript syntax, and `validate()` runs -over every item in a bundle. - -**What is actually true.** It does not track regex literals, so bracket characters inside one -are counted as brackets. Minimal reproduction: - -``` -const re = /([^)]+/; -> INVALID (AST_UNBALANCED_BRACKET) -const x = (a + b) / 2; -> VALID -``` - -`src/core/model/constructors.ts`, `src/core/elision/regions.ts` and -`src/core/topology/dependency-graph.ts` all fail their own project's validator today for -this reason. In the 52-file CLI sweep it accounted for 2 of the 30 fallbacks. - -**Why it matters beyond yield.** It is a **false negative in a safety check**, so it fails -closed rather than open — the pipeline falls back and the user gets their input. That is the -right direction to fail, which is why this is recorded rather than rushed. But it means the -TS validator cannot be relied on as a backstop for anything inside a regex literal, and -`elideRegions`'s post-condition is relative (no *new* issues) partly because of it. - -**Consequence for the region scanner.** `scanBraceSpans` in `src/core/elision/regions.ts` -*does* track regex literals, deliberately: the validator's blind spot means it could not -catch a region boundary the scanner got wrong inside one. - -### 2. Python delivered over stdin is invisible to the whole validator layer — **open, scoped** - -> Scoped 2026-08-04: `docs/phase-4b-pathless-code-scope.md` (renamed from -> `phase-1e-…`; the file said 1e and its contents said 4b, so the label is now **4b** -> everywhere). **4b.0 has since landed** — the benchmark harness passes a path; the engine -> defect below is untouched and 4b.1–4b.3 are not started. Two -> corrections to the entry below. The suggested remedy — "a content-based fallback when no -> path is available" — is right about *where* but the narrow version of it is wrong: resolving -> a language inside `selectValidator` without also correcting `contentType` raises drift on -> 14 of 20 measured files and pushes one over the gate, because pathless Python classifies as -> `markdown` or `text` and `extractMarkers` then harvests its `#` comments as headings — -> **1,025 fabricated markers across 43 files**. And the entry treats this as one defect; it is -> two, the second currently dormant and made material by fixing the first. - - -**What `CLAUDE.md` implies.** That `tokendamper optimize ` treats the two -input forms equivalently. - -**What is actually true.** They diverge at classification, because `createContextBundle` -derives `contentType` from content *and* `sourcePath`, and stdin has no path: - -``` -stdin, no path contentType=text language=undefined validator=null regions=0 -file arg, .py path contentType=code language=undefined validator=python regions=19 -``` - -With `validator=null`, `validateItemAst` returns `valid: true` without running anything, -`resolveElisionSyntax` falls through to the classifier, and `selectElisionRegions` returns -nothing. A Python file piped to `optimize -` is not validated, not segmented, and cannot be -compressed at sub-item granularity. - -This is why `tokendamper-benchmark/run_benchmark.py` sees no improvement from granularity: -it invokes `[cmd, "optimize", "-"]` and pipes the text. The same file passed as an argument -reduces 34.76%. - -**It is also a vacuous-check instance in the classic shape** — `validateItemAst` reporting -`valid: true` on content it never examined — and `CLAUDE.md`'s note that -"`classifyContent` already sets it at ingestion" is true but does not say that the answer -depends on how the bytes arrived. - -**Not fixed here.** The fix is a content-based fallback when no path is available, and it has -a blast radius over every item in every bundle (the `DECISIONS.md` §17 lesson), so it wants -its own change and its own turn-1 measurement. - -## `docs/issue-2-content-type-contract-design.md` §2.2 — the remedy it specifies closes half the hole - -**Status: corrected in code (DECISIONS §22, §23). The design doc is left as written.** - -The doc's §2.2 says: - -> It is invisible today only because the Gateway hardcodes `contentType: 'text'`, which makes -> `selectValidator` return `null`, so **no validator runs at all** on Gateway items. - -The diagnosis is right and the implemented remedy — replace the literal with -`classifyContent` (`ac16cec`) — follows from it. But the doc treats "the tag is now real" as -equivalent to "a validator now runs", and those are only the same thing if the classifier is -correct. It was not: - -``` ---- TS with an unterminated string literal --- - with path (CLI file arg) contentType=html validator=typescript valid=false issues=1 - no path (Gateway message) contentType=html validator=NULL valid=true issues=0 -``` - -`classifyContent` returned `html` for 46 of this repository's 57 TypeScript sources -(`looksLikeHtml`'s greedy `[\s\S]*`, plus probes running before extension checks — §22), and -`selectValidator` has no `html` branch. So the JSON half of §2.2's claim holds and the code -half does not. On the CLI the path-extension branch rescues it; a provider message has no -path, so nothing does. - -Two things follow that the doc does not anticipate: - -1. Fixing the classifier is necessary but not sufficient. The mechanism — a classifier free - to emit a tag dispatch has never heard of, failing *silently* — survives any number of - regex fixes. §23 binds them with a total `Record` and records - `validated: false` on the result so "nothing looked" stops reading as "it passed". -2. `CLAUDE.md`'s Issue 2 entry inherited the overstatement and has been corrected in place. - -## Phase 1a's closure claim needs the same qualification wherever it appears - -`tokendamper-headroom-known-issues.md` and `purposed architecture changes.md` both describe -Issue 2's content-type work as closing the Gateway validation gap. Read as "JSON payloads are -now validated and scored as JSON", that is accurate and measured. Read as "Gateway items are -now validated", it is not, and was not at any point: - -- Before `ac16cec`: hardcoded `text` → `selectValidator` → `null` → nothing ran. -- After `ac16cec`, for code: classified `html` → `selectValidator` → `null` → nothing ran. -- After §22/§23, for code: classified `code` if a filename is present, otherwise `text` or - `yaml` → still `null` for pathless content, but now reported as `validated: false` and - counted in `trace.astCoverage`. - -The remaining gap is deliberate: DECISIONS §17 removed content-only code detection because -its only signal was a markdown fence, and the resulting verdict flipped on apostrophe parity -in the surrounding prose. Nothing here re-opens that. The change is that the gap is now -*stated* by the engine instead of being indistinguishable from a clean bill of health. - ---- - -## `DECISIONS.md` §18 — the structural markers it proposes for code would be constants too - -**Date:** 2026-08-04 -**Status of the correction:** measured; **nothing implemented**. Full record: -`docs/phase-1d-semantic-gate-disposition.md`. - -**What the document says.** §18's Future Revisit Conditions name the remedy for an inert -`R_struct`: *"until `extractMarkers` learns structural markers that (a) live in the content -and (b) are meaningful for code — nesting depth, function and class boundaries, import -blocks, brace balance."* `docs/phase-1d-granularity-design.md` §8 carries the same list as -precondition **(a)**, "the real fix". - -**What is actually true.** Four of those five candidates do not vary under the selector that -shipped. Each was computed on the original and on the **real CLI output** of every file that -reduces, over two frozen corpora (this repo's 68 sources; 39 `pip` Python files): - -| proposed marker | differs before/after (repo) | differs (pip) | -|---|---|---| -| brace balance | 1 / 29 | 0 / 20 | -| paren balance | 0 / 29 | 1 / 20 | -| function headers | 0 / 29 | 2 / 20 | -| class headers | 0 / 29 | 0 / 20 | -| import lines | 0 / 29 | 2 / 20 | -| max nesting depth | 20 / 29 | 13 / 20 | -| **comment starts** | **14 / 29** | **14 / 20** | -| **docstring delimiters** | 1 / 29 | **17 / 20** | - -The reason is the same one that makes `filepath:` inert: `selectElisionRegions` preserves -signatures and brace balance **by construction**, because that is how it was designed to pass -the gate. Adding these markers replaces one decorative constant with four. Nesting depth does -vary, but on *every* successful body elision — as a retention term it penalises the transform -for having worked, which makes it a compression detector, not a loss detector. - -The only markers that move are comments and docstrings. So §18's remedy, followed to its -measurement, lands on the same content class as the "cheap" docstring exclusion it was -contrasted against. **(a) is not the real fix and (b) the stopgap; (a) is the scored version -of (b)** — it grades partial loss instead of vetoing, and it covers the whole-item path where -a region rule does nothing. If it is ever built it belongs beside `R_AST` as an explicit -information-class term: a "structural integrity" ratio computed from comment density would be -a lie in the metric's own vocabulary. - -**Also corrected: `HumanEval/0` is not the live instance.** `DECISIONS.md` §20 and the -design's §8 both present it as the measured hole. It is closed on the shipped path — -`selectElisionRegions` returns `[]`, the item falls to whole-item hashing, `S_k` pins at 0.60 -and the input is echoed verbatim. The live instances are different in kind and larger: -`src/index.ts` is elided **whole** at 86.15% with `S_k = 0.0000` because `extractSymbols` -finds no symbols in a barrel file and an empty *before* set is scored as perfect retention; -and 42% of this repo's elided function bodies (86% of `pip`'s) contribute no symbols at all, -so their removal cannot move the metric. Quote the class, not the fixture. - -**One consequence for `DECISIONS.md` §24.** Its `prose:WHOLE S_k=0.00 saved=78.4%` and -`logs:WHOLE S_k=0.00 saved=97.5%` rows are produced by that same default: `extractSymbols` -returns the empty set on `README.md`, `SECURITY.md` and `sample_logs.txt`, and -`sample_logs.txt` has exactly one marker (`filepath:`), so **neither term examined anything**. -§24's conclusion stands — a static content-type gate is still the wrong shape, and the saved -bytes are real — but "passes at `S_k = 0.00`" is not evidence of safety there and should not -be cited as if it were. Markdown is the exception: `README.md` yields 20 real markers, so -`R_struct` does genuine work on it. - ---- - -## `docs/phase-1d-semantic-gate-disposition.md` §3 — the ninth invariant-10 instance, and one detail it got wrong - -**Date:** 2026-08-05 -**Status of the correction:** measured, implemented (DECISIONS §28) - -**What the document says.** That `R_AST` and `R_struct` "default to `1.0` when the *before* -set is empty", which scores "nothing to measure" as "perfect retention" and lets -`src/index.ts` be deleted whole at 86.15% with `S_k = 0.0000`. - -**What is actually true.** The conclusion holds and is now fixed. The mechanism was stated -one step too loosely, and the difference matters to anyone touching this next. - -Measured on `src/index.ts` through the real `DriftTracker`: - -``` -symbolsBefore 0 markersBefore 1 R_AST 1.0 R_struct 1.0 S_k 0.0000 -markers before: ["filepath:src/index.ts"] -``` - -Only **`R_AST`** is at its empty-set default. `R_struct`'s before-set is *not* empty — it -holds one marker, `filepath:src/index.ts`, harvested from `item.path`. So the file is not a -case of "both components found nothing"; it is one component finding nothing and the other -finding a marker that **elision cannot destroy**, because `item.path` is metadata and no -content transform touches it. `R_struct` reports `1.0` as a genuine measurement of a quantity -that is preserved by construction. - -That is why the fix needed two parts rather than one. Renormalising away the empty component -would have left `R_struct = 1.0` vouching for the file on its own, and `S_k` would still have -been 0. Evidence had to exclude metadata-derived markers (`extractContentMarkers`) as well as -handle the empty set. §18's separate point — that `R_struct` is decorative for code — is the -same observation arriving from the yield side rather than the safety side. - -**Also corrected: "empty before-set" is not one rule.** Enforcing it wherever a before-set was -empty gutted prose (4 failing tests, including both Gateway cross-turn dedup cases and the -bench baseline). For prose the empty set is *inapplicability*, not extractor failure. The -shipped rule is scoped to items an AST validator covers. Prose and pruner-removed items remain -unwitnessed and are now reported on `DriftCoverage` rather than enforced. - ---- - -## `docs/phase-4b-pathless-code-scope.md` §6/§7 — the risk register missed the effect that dominated - -**Date:** 2026-08-05 -**Status of the correction:** measured, implemented (DECISIONS §29). Full record: that -document's own §8. - -**What the document says.** §6 lists five risks for the declaration/probe work, of which -risk 2 is *"new validation means new fallbacks are possible"* — expected to cost yield through -`PythonValidator` rejecting fragments. §7 projects the pip corpus from `2.42% → up to 14.11%`. - -**What is actually true.** Zero fallbacks came from syntax. The six files that reduce under -bare stdin and fall back once declared are all `SEMANTIC_DRIFT_UNMEASURABLE` — five TypeScript -barrels and `pip`'s `status_codes.py` — and that is **DECISIONS §28 reaching a route it had -never reached**. §28's refusal is conditional on an AST validator covering the item; nothing -covers a pathless item; so over stdin the barrel file that fix is recorded as saving was still -being elided whole at `S_k = 0.0000` with no fallback. `index.ts`: 135 → 18 tokens, -`astCoverage.checked: 0`, `unwitnessedItems: []`. - -The operative consequence is not about yield. **A pathless item is not merely unoptimized, it -is unprotected** — every safety property that dispatches on a validator is inert on it. Anyone -scoping 4b.2 from §7's token figures is scoping it from the weaker half of the argument. - -**Also: §7's projection is not comparable to the landed measurement**, and neither refutes the -other. `2.42% → 14.11%` is the engine's own estimator (24% MAE, CLAUDE.md) over a 39-file -selection; the landed `0.02% → 12.34%` is `cl100k_base` over a 45-file selection re-frozen on -2026-08-05 because the earlier scratch corpus was gone. Quote §8's table, which names its -tokenizer and its manifest. - ---- - -## `docs/phase-4b-pathless-code-scope.md` §6 risk 2 — disposed of by a third option it did not list - -**Date:** 2026-08-06 -**Status of the correction:** measured, implemented (DECISIONS §31). Full record: that -document's §9. - -**What the document says.** §6's risk 2: *"Enabling `PythonValidator` on pathless fragments -will reject some of them… Needs its own before/after count on the session corpus, and a -decision about whether a fragment that fails indentation should fall back or be reported as -uncheckable."* - -**What is actually true.** Neither branch of that question is the answer. A fragment that fails -the indentation rule should not be **detected**: - -> A probe may only claim content the validator for that language already accepts. - -The asymmetry that makes this right is the one §29 established from the other side. A -declaration is the caller's assertion, so failing on content that does not parse is correct and -informative. A detection is our own guess, so content that does not parse is evidence *against -the guess*, not against the content. Failing closed on a guess converts a heuristic into a -fallback generator on live traffic — the exact trade §17 refused when it removed fence-based -detection. - -Consequently the before/after fallback count risk 2 asked for is **zero by construction**, and -measured zero: Gateway turn 1 and turn 2 are byte-identical before and after, and the file -route is unchanged on all 45 frozen `pip` files. - -**Two numbers worth carrying.** The confirmation step **fires** — bad indentation, an -unterminated string and a truncation mid-argument each clear the structural rule and are each -rejected by the parser — and it **costs nothing**: all six `pip` files the probe declines parse -fine, so the structural rule turned them down, not the validator. A check that never fires and -a check that changes every outcome are both suspicious; this one is neither, and both halves -were measured rather than assumed (invariant 10). - -**Also: the residual hole is a safety hole, not a yield hole.** An undetected pathless Python -file is unprotected, not merely unoptimized — `pip`'s `status_codes.py` is elided whole and -unwitnessed over stdin at 44 → 27 tokens while the file route refuses it. Anyone scoping 4b.3 -or a wider probe from the yield tables is again reading the weaker half of the argument, which -is the same mistake §8 recorded for 4b.2. - ---- - -## `docs/phase-4b-pathless-code-scope.md` §5 — 4b.3 names the wrong buckets, and one marker kind that does not exist - -**Date:** 2026-08-06 -**Status of the correction:** measured, implemented (DECISIONS §32). Full record: that -document's §10. - -**What the document says.** §5's 4b.3: *"Undetected Python, and pathless code in any other -language, still harvest `#` and `- ` lines as structural markers from the `text` and `unknown` -buckets."* - -**What is actually true.** - -`- ` lines are **not harvested at all**, and never were. `collectMarkers` gates exactly three -kinds: `heading:`, `fence:` and `section:`. There is no bullet branch. - -And the fabrication is not in `text` or `unknown`. Measured pathless over five frozen corpora: -62 `text`-classified files, one `html` and one `logs` yield **zero** gated markers between -them, while **591** fabricated headings come from 9 shell scripts and **45** more from the 4 -`pip` files 4b.2 declines — all of them classified **`markdown`**, because `looksLikeMarkdown` -fires on a single `#` heading and a shell script's first `# Copyright …` line is one. - -§2's original `py/text 23 files 308 headings` was measured before §22's classifier fix and -4b.2's probe moved that population. What is left sits where an allowlist edit cannot reach it -without gutting the 25 real documents that yield 477 genuine headings. - -**The consequence is worse than a mis-scoped step.** On a frozen `tclConfig.sh` the fabricated -markers put `S_k` at exactly `0.400` — passing, since the gate is `> 0.40` — and the file is -deleted whole, 1,877 → 19 tokens, `fallbackUsed: false`, `astCoverage.checked: 0`, while -`DriftCoverage` reports `structMeasured: true` and `measured: true` on 79 comment lines. The -markers do not merely inflate the score; **they forge the evidence that the score measured -anything**, which is the reporting §28 added specifically so this class would be visible. - -**Carry this into any successor of §28.** That decision deferred "should drift certify an item -nothing covers?" as a product question about prose. The population is not prose — it is every -language the AST-lite suite does not implement. Pinned as a `KNOWN DEFECT` characterization -test so the behaviour cannot be changed by accident. diff --git a/README.md b/README.md index cb60834..a1ce39a 100644 --- a/README.md +++ b/README.md @@ -167,12 +167,43 @@ Raw Input -> Stateless 0/1 Knapsack Planner -> Linear Engine -> Topology Pruning & Delta Compression - -> AST Validators + -> Syntax Checks (see "What validation actually checks") -> Explicit Fallback (on constraint violation) -> Final Output + Explainability Trace ``` -Note: the `AST Validators` and `Explicit Fallback` steps above run in **all three** modes, Gateway included, since Phase 1.0b. (A previous version of this note claimed Gateway mode skipped them; it does not.) What differs on the Gateway is the *stage list*, not the checks — it plans only `cleanup:session-dedup`. See the Gateway proxy status notice at the top. +Note: the `Syntax Checks` and `Explicit Fallback` steps above run in **all three** modes, Gateway included, since Phase 1.0b. (A previous version of this note claimed Gateway mode skipped them; it does not.) What differs on the Gateway is the *stage list*, not the checks — it plans only `cleanup:session-dedup`. See the Gateway proxy status notice at the top. + +## What validation actually checks + +The internal validators are called "AST-lite". **They are not parsers and they do not build an +AST** — the one exception is JSON. Stating the guarantee precisely, because it is the product's +headline property: + +| Content | What is checked | What is *not* | +|---|---|---| +| **TypeScript / JavaScript** | Bracket, quote and comment balance, by a lexer that tracks strings, template interpolation and regex literals | Everything else. `const x = ;`, `import from "x";`, `let 123abc = 5;` and plain English prose all **pass** | +| **Python** | The above, plus missing colons, malformed `def`, bad dedent and stray leading indentation | Plain English prose still passes | +| **JSON** | Fully parsed — this one is a real check | — | +| **Everything else** | Nothing. No validator covers it | Reported on `trace.astCoverage`, never silently counted as a pass | + +So the guarantee this product offers on TypeScript — the language family where compression +actually runs — is **bracket and quote integrity**, not syntax validity. An elision that lands +somewhere syntactically nonsensical but balanced is caught by the drift metric, if at all, not by +the syntax check. + +Two consequences worth stating plainly: + +- **A passing check is not a promise the output compiles.** It is a promise the output is no + more unbalanced than the input. +- **Real inputs are often already invalid**, and that is deliberate: a truncated completion + prompt is a first-class input, so the sub-item check is *relative* — an elision must not + introduce a new problem, rather than produce provably valid code. + +Wiring the real TypeScript compiler API would change this. It is not done: `typescript` is a +development dependency today, and making it a runtime one costs install size and parse latency +against a lexer that runs in single-digit milliseconds. That is a deliberate trade, not an +oversight — see `docs/audit-remediation-status.md`. ## License TokenDamper is licensed under the Mozilla Public License 2.0 (MPL-2.0). See [LICENSE](./LICENSE). diff --git a/ROADMAP.md b/ROADMAP.md index b0dbd90..d6d1040 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -146,10 +146,13 @@ before scheduling any of it.** Each is a decision with a legitimate "narrow the `fallbackUsed: false`. It was live, on the one path the Gateway saves anything on. - **M7** — measure savings against the bytes on the wire rather than the newline-joined render. Still open, and no longer blocked on C4. **Only if B keeps the Gateway.** -- **M1 + M4b + M11** — docs last, once A–D are answered. Rename "syntax validity" to - "bracket/quote integrity" (the TS validator passes `const x = ;` and plain English prose); - correct the README's remaining untrue claims; retire the phase narratives to git history. - Docs currently run **4.1:1** against source. +- ~~**M1 + M11**~~ — **done.** "Syntax validity" is now "bracket/quote integrity" in the README + and `CLAUDE.md`, with a per-language table of what each validator does and does not catch, + pinned by `test/unit/validator-guarantee.test.ts`; the phase narratives are retired to git + history. The **4.1:1** figure was stale by the time it was acted on — measured before the + cleanup it was **1.40:1**, and not because the docs shrank (they had grown to 726 KB) but + because `src/` had grown faster. Counting the 33% of source that is comment prose, prose:code + was ~2.6:1. --- diff --git a/docs/audit-remediation-status.md b/docs/audit-remediation-status.md index 58684c6..fba3e66 100644 --- a/docs/audit-remediation-status.md +++ b/docs/audit-remediation-status.md @@ -16,16 +16,20 @@ typecheck and lint clean. | **1** | C1a, C1b, C2, M6 | ✅ merged | | **2** | M5a, M5b, H4, M8, M9, M10 + M5 minor | ✅ **done — DECISIONS §44** | | **3** | C3/H1 ✅, H6 ✅, H5 ✅, C4 ✅ | ✅ **done — C4 in DECISIONS §45** | -| Decisions | H2, M1, M11 | ⬜ not addressed | +| Decisions | H2, M1, M11 | ✅ **decided and done — DECISIONS §46** | -**Every task-shaped audit item is now closed.** What remains is three *decisions* (H2, M1, M11) -and the architectural work in §5 — chiefly Phase 1c, which is the binding constraint on -multi-file value. +**Every audit item is now closed.** What remains is the architectural work in §5 — chiefly +Phase 1c, which is the binding constraint on multi-file value. + +The three decisions were taken as: **H2 — report why** (keep accepting every language, but say +when elision cannot reduce it, rather than narrowing the accepted set); **M1 — correct the +docs** (say "bracket/quote integrity", do not wire the compiler API); **M11 — retire the +narratives and the root planning artifacts**. Each is argued in §46. ### Corresponding DECISIONS entries §36 license · §37 C1a · §38 C2 · §39 M6 · §40 C1b · §41 C3/H1 · §42 H6 · §43 H5 · §44 Wave 2 · -§45 C4 +§45 C4 · §46 H2/M1/M11 --- @@ -193,15 +197,45 @@ Learned the hard way during Waves 0–3. `messages[2].content` came back as the string `"{\"__td_block__\":\"[TokenDamper Elided: …]\"}"` with `fallbackUsed: false` and `tokensSaved: 42`. That is the one case the Gateway saves anything on at all, so C4 was live on precisely the path the mode exists for. -- **H2 — language coverage.** 3 of 19 declared languages can produce a non-zero reduction. - Decision, not a task: narrow the accepted set, or make a declared-but-unsupported language - report *why* instead of falling back mutely. - - **Wave 2 built the shape of the answer without applying it here.** M5a's response now carries - `budgetApplied` and a `notice` so a 0% result says whether anything ran. A declared-but- - unsupported language is the same question one layer down — the run reports 0% and does not say - the language has no validator behind it. `trace.astCoverage` already holds the fact. -- **M1 — "AST-lite validator".** The TypeScript validator is a bracket/quote matcher. Say that in - user-facing docs, or wire the real compiler API. -- **M11 — documentation volume.** 4.1:1 against source. This file is meant to *replace* scattered - status prose, not add to it; retire superseded phase narratives to git history. +- ~~**H2 — language coverage.**~~ **Decided: report why. DECISIONS §46.** Every language is + still accepted — pass-through is byte-identical and refusing it would remove a working + behaviour to make a point. What changed is that the run now says when elision cannot reduce + the input: `trace.languageSupport` carries `supported`/`unsupported`/`noneSupported` and a + `reason` string, and `validate()` raises an **info** issue, `LANGUAGE_NOT_ELIDIBLE`. + + Measured, elision reduces **3 of 17** probed languages — TypeScript, JavaScript, Python — + which matches both the audit headline and the corpus. The predicate is `supportsRegionElision` + and nothing looser: a first attempt asked "does the item yield symbols?" and called Go + supported, because a trivial Go file yields exactly one — `import:fmt`, an incidental match by + the TypeScript import regex. + +- ~~**M1 — "AST-lite validator".**~~ **Decided: correct the docs, do not wire the compiler + API. DECISIONS §46.** `README.md` and `CLAUDE.md` now say **bracket/quote integrity** and + carry a per-language table of what each validator does and does not catch. + `test/unit/validator-guarantee.test.ts` pins all of it as characterization tests, verified + against the shipped validator rather than copied from the audit. + + Wiring `ts.createSourceFile` was rejected on cost, not principle: `typescript` is a *dev* + dependency today, and promoting it to runtime costs install size and parse latency against a + lexer that runs in single-digit milliseconds. Revisit if the guarantee needs to be stronger. + +- ~~**M11 — documentation volume.**~~ **Decided: retire the narratives and the root planning + artifacts. DECISIONS §46.** Twelve files, **226 KB**, 31 markdown files down to 19. + `docs/retired-documents.md` maps each to where its conclusion lives and how to read the + original out of git. + + **The 4.1:1 premise was stale.** Measured before the cleanup it was **1.40:1**, and not + because the docs had shrunk — they had grown to 726 KB — but because `src/` grew faster. + Counting the **32.8%** of `src/` that is comment prose, prose:code ran ~2.6:1. After the + retirement, markdown:src is **0.95:1**. + + The in-source commentary was deliberately left alone. It is the part that sits next to the + code it explains and is maintained with it; the retired files were the part that had to be + kept in sync by hand, which is the failure mode M11 actually names. + + **Twenty-five source and test comments cite a retired document.** They are marked + `[retired]` rather than rewritten: the citation names a document and section that existed and + git still has, whereas re-pointing 25 citations at DECISIONS sections by hand would risk + mapping some of them to the wrong place. `CHANGELOG.md` and `DECISIONS.md` keep their older + citations untouched for the same reason — they are append-only records of what was true when + written, and each now carries a note saying so. diff --git a/docs/issue-2-content-type-contract-design.md b/docs/issue-2-content-type-contract-design.md deleted file mode 100644 index 35cb075..0000000 --- a/docs/issue-2-content-type-contract-design.md +++ /dev/null @@ -1,458 +0,0 @@ -# Issue 2 — Content-Type Contract (Design Proposal) - -> **STATUS: APPROVED 2026-08-02**, with the seam changed from `ContextBundle` to -> `ContextItem`. Written against `d51a01b`. Landing as two commits: (1) placeholders and the -> chokepoint, (2) the relabel. -> -> This supersedes `purposed architecture changes.md` Change 1, which specified a -> bundle-level tag. See `NOTES-FOR-DOCS.md`. - ---- - -## The reframing (read this first) - -**The tag was never missing.** - -`ContentType` has existed on `ContextItem` since the model was frozen -(`src/core/model/types.ts:19`, field at `:53`), and `classifyContent()` has been setting it -at ingestion the whole time. `compression:token-hashing` **already reads it** — at -`token-hashing.ts:75` it copies `item.contentType` onto the very item it produces. It -carries the tag through the transform and never once consults it to decide whether the -transform is safe. - -So Issue 2 is not "add content-awareness to the pipeline." It is **"stages ignore the -awareness they already have."** Three concrete defects follow from that framing, and none of -them is a missing field: - -1. Four construction sites bypass `classifyContent()` and hardcode a literal - (three `'text'` in the Gateway, one `'code'` in bench). -2. Eliding stages never consult the tag they already propagate. -3. Nothing binds *how a stage elides* to *which validator will judge the result* — the two - are written independently, which is why they disagree. - -Framing it as "add a tag" would have produced a second tag alongside the working one, at a -coarser granularity than the validator that has to agree with it. That is how this bug is -reproduced, not fixed. - ---- - -## 0. What I found on the proxy side first - -You asked me to check whether `61bd685` introduced content-type handling on the Gateway -before designing anything. - -**It did not.** None of the 12 files in that commit added, moved, or decided content-type -behavior. There is **no parallel implementation to reconcile** — Issue 2 still has one -contract to design. - -There is, however, a **pre-existing wrong default** on that path, older than `61bd685`: - -| Site | Value | Set by | -|---|---|---| -| `src/gateway/proxy.ts:441` (OpenAI messages) | `contentType: 'text'` hardcoded | pre-existing | -| `src/gateway/proxy.ts:546` (Anthropic system) | `contentType: 'text'` hardcoded | pre-existing | -| `src/gateway/proxy.ts:568` (Anthropic messages) | `contentType: 'text'` hardcoded | pre-existing | -| `src/bench/evaluator.ts:45,54` | `contentType: 'code'` hardcoded | pre-existing | - -`classifyContent()` — the ingestion classifier that already exists — is called from exactly -one place, `createContextBundle()` (`constructors.ts:67`), which is the **single-item CLI -path only**. Every multi-item construction site bypasses it and hardcodes a literal. - -One thing `61bd685` *did* decide implicitly, and it should be named: pinning the Gateway to -`session_dedup` mode is a **stand-in for content-type awareness**. It avoids -token-hashing-on-JSON by avoiding token-hashing entirely. That pin is what this work should -make unnecessary — the mode can stay, but its justification disappears. - ---- - -## 1. Where I disagree with the framing - -### 1.1 `ContextBundle` is the wrong seam. The tag belongs on `ContextItem` — where it already is. - -The brief says content-type "becomes a first-class tag on `ContextBundle`". I think that is -the wrong place, for four reasons drawn from the code: - -**(a) It already exists on `ContextItem`, and is already set at ingestion.** -`ContentType` is defined at `src/core/model/types.ts:19` as -`'text' | 'markdown' | 'code' | 'html' | 'json' | 'yaml' | 'logs' | 'unknown'`, and -`ContextItem.contentType` at line 53. `classifyContent()` sets it. The tag is not missing. - -**(b) The validator that rejects the placeholder dispatches per item, not per bundle.** -`selectValidator(item)` (`src/core/validation/ast/index.ts`) resolves with precedence -`item.language` → `item.path` extension → `item.contentType`. If the tag governing the -*transform* were bundle-level while the *check* is item-level, the producer and the checker -would be keyed at different granularities. That mismatch is the exact defect class that -produced this bug. Putting the tag on the bundle would preserve the shape of the bug while -appearing to fix it. - -**(c) Bundles are heterogeneous by construction.** A Gateway bundle is N messages: message 0 -may be a 12 KB JSON tool result, message 1 a one-line prose question. A scalar bundle tag -forces a choice between over-conservatism (skip the stage entirely if *any* item is JSON, -losing reduction on the prose) and corruption (apply a JSON contract to prose, or worse the -reverse). Neither is acceptable, and the choice only exists because the tag is at the wrong -level. - -**(d) `ContextBundle` already carries the correct bundle-level summary.** -`statistics.contentTypeCounts` (`types.ts:90`) is a `Record` histogram. -A scalar `bundle.contentType` would duplicate and contradict it. - -### 1.2 The bug is not a missing tag. It is an unconsulted one. - -`compression:token-hashing` **already reads and propagates** `item.contentType` — at -`token-hashing.ts:75` it copies it onto the item it produces. It just never consults it when -deciding eligibility. Its four eligibility rules check `preserveKinds`, `role === 'system'`, -`metadata.elided`, and `content.length`. There is no content-type rule. - -So the actual defect is three things, none of which is "the tag does not exist": - -1. Four construction sites bypass `classifyContent()` and hardcode a literal. -2. Eliding stages do not consult the tag they already carry. -3. Nothing binds *how a stage elides* to *which validator will judge the result*. - -Fixing only (1) makes things **worse** — see §2. - -### 1.3 Scope correction: `session-dedup` has the same defect as `token-hashing` - -This is the finding that most changes the shape of the task. - -The brief scopes Issue 2 to `compression:token-hashing`. But `cleanup:session-dedup` writes -`[TokenDamper Elided: ref=… bytes=… kind=…]` into item content with equally no idea what it -is overwriting. That string is not valid JSON either. - -It is invisible today only because the Gateway hardcodes `contentType: 'text'`, which makes -`selectValidator` return `null`, so **no validator runs at all** on Gateway items. The bug -is masked by the mislabel, not absent. - -Consequence: **relabelling the Gateway without also fixing the dedup placeholder would break -the Gateway's only working stage.** The contract must cover every eliding stage, not just -token-hashing. - ---- - -## 2. Measurements - -Measured against `tokendamper-benchmark/test_data/tool_output.json` (12,036 bytes, parses -clean) in a Gateway-shaped two-item bundle, at `d51a01b`. - -### 2.1 Drift and AST delta from correcting the label - -| contentType | Validator selected | Stage | AST valid | drift `S_k` | Fallback | -|---|---|---|---|---|---| -| `text` *(today)* | **NONE** | session-dedup | true | 0.0000 | no | -| `text` *(today)* | **NONE** | token-hashing | true | 0.0000 | no | -| `json` *(relabelled)* | `json` | session-dedup | **false** | 0.0000 | no | -| `json` *(relabelled)* | `json` | token-hashing | **false** | **0.6000** | **yes** | - -**Drift delta from the relabel:** - -- `session-dedup`: **0.0000 → 0.0000** (no change). Symbols before/after `32 / 32` — the - Phase 1.0b recoverable-elision exemption substitutes the original content back, so - correcting the label does *not* push dedup over the threshold. -- `token-hashing`: **0.0000 → 0.6000**, over the 0.40 threshold. Symbols `32 / 0`. - -Per your instruction: the token-hashing move to 0.60 **is the drift gate, and is expected**. -It is not a failure of this contract and I am not tuning the contract to avoid it. It is the -correct behavior for a lossy stage that destroys 32 JSON key symbols. I am reporting it -rather than designing around it. - -The dedup row staying at 0.0000 is worth noting explicitly: **the Phase 1.0b exemption is -precisely what makes the relabel safe for the Gateway's current single stage.** Without it, -the relabel would have pushed dedup drift up as well, and the Gateway would have regressed. - -### 2.2 The uncomfortable finding: today's safety net is vacuous on JSON - -The `text` rows above show AST `true` and drift `0.0000` for *both* stages — including -token-hashing, which is actively corrupting the payload. - -Both checks are passing **vacuously**: - -- AST: `contentType: 'text'` with no `language` and no `path` → `selectValidator` returns - `null` → the item is not validated at all. -- Drift: `DriftTracker.extractSymbols` only harvests `jsonkey:` symbols when - `item.contentType === 'json'`. A JSON blob labelled `text` yields **zero** symbols (it - has no `function`/`class`/`const` patterns), so retention is vacuously 1.0 and drift 0.0. - -This qualifies the Phase 1.0b claim in `docs/phase-1-stabilization-summary.md`: the Gateway -safety net is real and does fire (proven on the constraint-directive case), but **on -JSON-shaped content specifically it is currently blind**. The mislabel is not a cosmetic -defect; it is silently disarming both validators. That should be recorded regardless of -what happens to the rest of this design. - -### 2.3 Placeholder format candidates - -| Candidate | Bytes | JSON-valid | Round-trips through existing `rehydrateText` | -|---|---|---|---| -| `` *(current)* | 77 | **no** | exact | -| `""` (quoted) | 79 | yes | **BROKEN** — `Unexpected non-whitespace character after JSON` | -| `{"__td_block__":"…"}` (wrapper) | 83 | yes | returns unchanged — not rehydrated at all | - -**This is the constraint that binds the design.** `TokenHasher.rehydrateText` uses -`/]+)>/g` and substitutes raw content in place. -For the quoted form it matches the *inner* placeholder and produces -`"{…raw JSON object…}"` — a quoted string wrapping a JSON object, which is invalid. For the -wrapper form the regex does not match at all, so nothing is rehydrated. - -**Placeholder format and rehydration are one contract, not two.** Any format change must -land with its matching reverse transform in the same change, or round-trip breaks. - -Byte-stability was verified: two independent `TokenHasher` instances produce an identical -placeholder for identical content, because it is `sha256(content)` and nothing else. No -position, session, turn, or timestamp enters the string. (`createBlockPlaceholder` does -record `createdAt: Date.now()` in the block *metadata*, but that never reaches the emitted -bytes — this must stay true.) - ---- - -## 3. The proposed contract - -### 3.1 The tag - -Keep `ContextItem.contentType`. Add nothing to `ContextBundle`. - -Change instead: **every construction site must classify rather than hardcode.** Route the -four hardcoded sites through `classifyContent()`, so the Gateway and bench paths get the -same treatment the CLI already gets. - -`ContextBundle.statistics.contentTypeCounts` remains the bundle-level view, and becomes -accurate for the first time as a side effect. - -### 3.2 The tag wins; the classifier only fills a vacuum - -**Superseded 2026-08-02.** This section originally proposed "tag as hint, content as ground -truth", with an independent probe that could override the tag. That was wrong twice over, -and both errors are worth recording because they were caught in code review rather than by -tests. - -**Error 1 — it created a producer/checker split.** A probe that can override the tag means -the encoder decides by content while the checker (`selectValidator`, `extractSymbols`) -decides by tag. That is the same granularity mismatch this document rejects in §1.1 when -arguing against the bundle-level seam, reintroduced one layer down. It shipped in `29f66b3` -and was live: a TypeScript item whose content parses as JSON (an object literal) was -JSON-wrapped while being validated as TypeScript. - -**Error 2 — the probe was a duplicate, not a second opinion.** `classifyContent` already -decides JSON via `looksLikeJson` (`constructors.ts:468`), which is a structural-opener check -followed by `JSON.parse` — *precisely* the algorithm the bespoke probe implemented. There -were never two authorities with different strengths; there were two copies of one algorithm. -The Gateway's problem was never an unreliable tag, it was that on that path the tag is -**never computed at all**. - -**The rule, as implemented:** - -1. `selectValidator(item)` is authoritative. If it returns a validator, its language decides - the syntax — `json` for `JsonValidator`, `raw` otherwise. Full stop. -2. Only when it returns `null` — the item has no governing authority whatsoever — consult - `classifyContent`, the canonical ingestion classifier. It may only tighten `raw -> json`. -3. There is no third step and no bespoke probe. `parsesAsJsonDocument` was deleted. - -This still fixes the Gateway before the relabel, because the Gateway's items have no -`language`, no `path`, and `contentType: 'text'`, so `selectValidator` returns `null` and -step 2 applies. But it does so by calling the one classifier rather than by keeping a rival -implementation. - -### 3.3 Placeholder contract per content type - -| Content type | Placeholder form | Rationale | -|---|---|---| -| `json` | `{"__td_block__":""}` | Valid JSON; self-describing; survives embedding in a larger structure; distinguishable from user data by the reserved key | -| `code`, `prose`, `logs`, `text`, `markdown`, others | `>` (unchanged) | Current behavior preserved; no cache disturbance on paths that work today | -| unknown / ambiguous | **refuse to elide** | See §3.5 | - -I prefer the wrapper object over the quoted string for `json`. Both are JSON-valid, but the -quoted form is indistinguishable from a legitimate user string that happens to contain that -text, whereas a reserved object key is unambiguous — and unambiguous reverse-mapping is what -rehydration needs. - -**Every form must remain a pure function of `sha256(content)`.** No position, index, session -id, turn number, or timestamp. This is the cache-alignment requirement and it is -non-negotiable: `cache-aware.ts` pins the prefix by accumulating `estimateItemTokens` over -items in order, so an unstable placeholder both busts the provider cache and perturbs -prefix-pin boundaries. - -Note the JSON form is ~6 bytes longer than the bare form. Negligible against a 12 KB block, -but it does shift `estimateItemTokens` and therefore knapsack weights and the 1024-token -prefix accumulation. Worth asserting in a test rather than assuming. - -### 3.4 Enforcement — and an honest limit - -You asked for the invalid case to be **unrepresentable**, not merely checked. I want to be -straight about how far that is achievable here. - -**TypeScript cannot statically prove a `string` is valid JSON.** `ContextItem.content` is a -`string`. No type-level encoding makes "emits invalid JSON for a JSON item" a compile error -without changing content to a parsed representation, which is a far larger change than this -task and would fight the immutable-bundle model. Anyone promising you pure type-level -unrepresentability here is overselling. - -What is achievable is a **single chokepoint with an inescapable post-condition** — three -layers: - -**Layer 1 — branded type.** Placeholder rendering returns a branded `Placeholder` type, not -`string`. Raw strings become unassignable to elided-item content, so a stage cannot -hand-roll a marker. This catches the careless case at compile time. - -**Layer 2 — one chokepoint.** All eliding stages call a shared -`elideItem(item, hash): ContextItem | null` rather than constructing elided items -themselves. Today `session-dedup`, `token-hashing` and `delta-compression` each build their -own — which is why the same bug exists in more than one of them. One implementation, one -place to be correct. - -**Layer 3 — post-condition inside the chokepoint.** `elideItem` renders the placeholder, -constructs the candidate item, then runs `validateItemAst()` on it *before returning*. If -the candidate fails, it returns `null` and the caller keeps the original item. - -**Correction after implementation (2026-08-02).** Layer 3 is *not* what makes this safe, and -the original wording above oversold it. Measured: only `JsonValidator` rejects a bare -``; the TypeScript and Python AST-lite validators both **accept** it. Since -JSON is exactly the case Layer 1 renders safely, **`post_condition_rejected` is currently -unreachable**. - -The load-bearing mechanism is **Layer 1, correct-by-construction rendering**: the stage does -not choose the encoding, so it cannot express the invalid case. Layer 3 is retained as -defense in depth against a future renderer bug or a stricter validator, and a test pins the -reasoning so it fails loudly if that changes. - -Corollary worth recording: placeholder injection into TypeScript or Python content would not -be caught by AST validation at all. Only drift would flag it. That is a pre-existing -weakness in the AST-lite validators, not something this contract introduces — but it does -mean the AST gate is JSON-only in practice. - -### 3.4.1 Refusal semantics - -A refusal **skips that item and the stage continues**. It never aborts the stage. - -Per-stage checkpointing (Phase 1c) does not exist yet, so a stage-level abort would surface -as `status: 'failed'`, which the engine converts into `STAGE_EXECUTION_FAILED` and a -whole-pipeline fallback. That would convert a placeholder defect into a fallback defect — -the exact failure mode this phase exists to remove. - -The caller keeps the original item and records the skip. Trace reporting per stage: - -| Metric | Meaning | -|---|---| -| `itemsSkipped` | total items the chokepoint declined | -| `skippedNoSavings` | rendered content was not shorter than the original | -| `skippedPostConditionRejected` | validator rejected the candidate (currently unreachable) | - -The stage `notes` string carries the same breakdown in prose, so a stage that correctly -declined every item is visibly distinct from a stage that ran and was rolled back — which -today are indistinguishable in the trace. - -Cost: one extra AST validation per elided item. `validateItemAst` already enforces a 5 ms -SLA and `JSON.parse` on 12 KB is microseconds. The existing per-stage validation already -walks every item, so this is a constant-factor increase on an existing traversal. - -### 3.5 Misdetection behavior and failure direction - -Direction: **conservative — refuse to transform, never emit-and-hope.** - -| Misdetection | Outcome | -|---|---| -| JSON blob mislabelled `prose`/`text` | Caught by the §3.2 content probe → JSON contract applied → correct placeholder | -| Prose mislabelled `json` | JSON validator rejects the bare placeholder → Layer 3 refuses → item kept whole. Lost reduction, no corruption | -| Markdown containing fenced code | No JSON validator selected; bare placeholder; validator is `null` or TS → unchanged from today | -| JSONL / log file that is really JSON lines | Whole-content `JSON.parse` fails (JSONL is not a single document) → treated as logs → bare placeholder → no validator → no corruption. Reduction is achieved, structure is not claimed | -| Ambiguous / unknown | Refuse to elide | - -In every row the failure mode is **lost reduction, not emitted corruption**. That is the -direction you asked for, and it falls out of Layer 3 rather than being separately -engineered: anything the validator would reject simply does not get emitted. - -### 3.6 How the planner consults it - -The brief wants the planner to consult content-type *before* transforms run. I agree with -the intent but think planner-level gating alone is insufficient, for the §1.1(c) reason: -the planner selects `stageIds` for the **whole bundle**, while eligibility is **per item**. -Gating there means "skip token-hashing if any item is JSON", which discards reduction on -every prose item in a mixed bundle. - -Proposed split: - -- **Planner (coarse, bundle-level):** consult `statistics.contentTypeCounts`. If *every* - eligible item is a type a stage cannot transform, drop the stage from `stageIds` and say - so in the trace. This is an honesty-and-cost optimization — it avoids scheduling a stage - that will no-op, and makes the trace explain why. -- **Chokepoint (fine, per-item):** the §3.4 enforcement decides each item. - -Both, not either. The planner's job is not scheduling work that cannot succeed; the -chokepoint's job is guaranteeing correctness. Only the second is load-bearing for the bug. - -### 3.7 What changes in the trace - -Today a JSON payload produces one aggregate fallback with AST and drift reasons commingled — -which is exactly why fixing the placeholder alone would look like a failure. - -Additions: - -- Per-stage `itemsSkipped` with a reason breakdown (`content_type_ineligible`, - `post_condition_rejected`), so a stage that correctly declines is visibly distinct from a - stage that ran and got rolled back. -- Planner records why a stage was dropped from `stageIds`. -- The bundle content-type histogram, so a trace is self-explaining without re-deriving it. -- Separate `astIssues` from `driftScore` in the fallback reason rather than joining them - into one string. Required for the §4 acceptance criterion to be checkable at all. - ---- - -## 4. Acceptance criteria - -Per your instruction, success is defined as **the AST error no longer appearing in the -trace** — not as reduction going above zero. - -**Primary (must hold):** - -1. On `tool_output.json` and `session.json`, no `JSON_SYNTAX_ERROR` appears in the trace for - any stage-emitted placeholder. -2. Round-trip: rehydrating an elided JSON item reproduces the original content **byte for - byte**. -3. Every emitted placeholder is byte-identical across turns and across independent - `TokenHasher` instances for identical content. -4. No stage can emit an item that fails its own validator — asserted by a test that calls - the chokepoint directly with hostile inputs. - -**Explicitly NOT criteria:** - -- Reduction > 0% on `tool_output.json` / `session.json`. Per §2.1, token-hashing on JSON - measures **drift 0.60 against a 0.40 threshold**, so those payloads may still legitimately - fall back on drift *after* the AST error is gone. That is the drift gate doing its job. - Judging this work by reduction would score a correct fix as a failure. -- Any change to the drift threshold. Out of scope here. - ---- - -## 5. Test strategy - -Contract tests (these I can write before approval, as agreed): - -1. **Placeholder validity, table-driven** — for each content type, the rendered placeholder - passes the validator `selectValidator` picks for that item. -2. **Round-trip byte-identity** — property/fuzz test over generated JSON documents - (nested objects, arrays, embedded quotes, unicode, empty containers): elide → rehydrate → - assert `===` original. Extends `test/unit/fuzz-diff-debt.test.ts` rather than adding a - parallel harness. -3. **Cache stability** — identical content ⇒ identical placeholder bytes across independent - hashers, across turns, and independent of item position within the bundle. -4. **Post-condition inescapability** — a deliberately wrong stage that tries to emit a bare - placeholder for a JSON item; assert the chokepoint refuses and the original item survives. -5. **Misdetection matrix** — one case per §3.5 row; assert the outcome is lost reduction and - never invalid emitted content. -6. **Regression: `session-dedup` on JSON** — the §1.3 defect; currently masked by the - mislabel, must not reappear after relabelling. -7. **Relabel delta guard** — pins the §2.1 numbers, so a future change to `extractSymbols` - or `selectValidator` that silently re-disarms the JSON path fails loudly. - ---- - -## 6. Open questions for you - -1. **Wrapper object vs quoted string** for the JSON placeholder (§3.3). I recommend the - wrapper for unambiguous reverse-mapping; it costs ~4 bytes more than the quoted form. -2. **Scope of the relabel.** Correcting `bench/evaluator.ts`'s hardcoded `'code'` will change - benchmark AST dispatch and therefore the baseline numbers. Include it here, or isolate it - so this work is not entangled with benchmark movement? -3. **`session-dedup` inclusion.** §1.3 says it must be in scope or the relabel regresses the - Gateway. Confirm you want that absorbed here rather than tracked separately. -4. **Gateway enablement** stays out of scope per the brief. For the record, the bar for - enabling `token-hashing` there would be: this contract landed, round-trip proven, **and** - the drift gate resolved for lossy stages on JSON — which §2.1 shows is a separate - problem this design does not solve. diff --git a/docs/phase-0-measurement-baseline.md b/docs/phase-0-measurement-baseline.md deleted file mode 100644 index 235beab..0000000 --- a/docs/phase-0-measurement-baseline.md +++ /dev/null @@ -1,209 +0,0 @@ -# Phase 0 — Measurement Baseline, and Seam 2 Measured - -> **Date:** 2026-08-06 · **Status: harness landed, corpus frozen, seam 2 measured.** -> No engine change. Engine: `dist` built from `2df850d`, tree dirty with the harness itself -> (recorded as `dirty: true` in the manifest — this is an instrumented run, not a release -> baseline). Corpus: 289 files under `sha256` manifest, 578 CLI runs, 0 failures. -> Tokens are the **engine estimator** throughout, because every figure here comes from a CLI -> trace; `cl100k` aggregates are not quoted. -> -> **Headline: seam 2 is not dead. It is the best-separating lever measured so far** — 114 code -> files misclassified as markdown drop to 12, with **zero** prose casualties. DECISIONS §32 -> deferred on the assumption that a classifier change carries blast radius over prose. Measured, -> it does not. -> -> **Second headline, unplanned: §32 is not a pathless-route defect.** It reaches the file -> argument, and the worst case found is 57,037 → 19 tokens on a file passed by name. - ---- - -## 1. What landed - -`tools/corpus-harness/` — `collect.js` (freeze + pin), `measure.js` (verify + run both routes + -dump traces), `seam2.js` (this note's analysis), `recipe.json`, `README.md`. - -The guarantees it adds are in the README. The two that already paid for themselves: - -- **Asserted bucket counts.** The first collection run reported three mismatches immediately. - One was real (see §2), two were my recipe being wrong about this machine. -- **A pin that shows dirt.** Every run records the commit, a hash over all 116 `dist/**/*.js`, - and whether the tree was dirty. - -## 2. The corpus, and a near-miss worth recording - -289 files, nine buckets, chosen so the population Phase A actually tests is present rather -than assumed: - -| bucket | n | extension in `isCodeExtension`? | AST-lite covers it? | -|---|---|---|---| -| shell `.sh` | 40 | yes | no | -| perl `.pl` | 40 | **no** | no | -| tcl `.tcl` | 40 | **no** | no | -| c `.c/.h` | 30 | yes | no | -| rust `.rs` | 3 | yes | no | -| css `.css` | 10 | yes | no | -| python `.py` | 45 | yes | **yes** | -| typescript `.ts` | 56 | yes | **yes** | -| prose `.md` | 25 | — | n/a (negative set) | - -No Ruby, Go, SQL or Java exist on this machine; Perl and Tcl replace them and turn out to be -the more interesting cases anyway (§3). The TypeScript bucket is **56, not the 64** quoted in -`docs/phase-4b-*.md` — a 1 KB floor excludes eight sources — so aggregates here are not -directly comparable to those documents. - -**The near-miss.** Selection is "sort by path, take the first N", which is deterministic and -was silently wrong: `.agents/` sorts before everything, so the first run filled all 40 prose -slots with prior agent scratchpads and selected **zero** hand-written documents. The prose -bucket is the negative set the entire seam-2 question rests on. Deterministic is not -representative, and the failure is invisible unless you look at what a bucket caught. - -## 3. Baseline - -`--target-reduction-ratio 0.3`, both routes, all 289 files. - -| bucket | route | n | reduce | fallback | ast=0 | drift `measured:true` | saved% | -|---|---|---|---|---|---|---|---| -| shell | file | 40 | 0 | 40 | 0 | 15 | 0.00% | -| shell | stdin | 40 | 20 | 20 | **40** | 40 | 17.28% | -| perl | file | 40 | **34** | 6 | **40** | 13 | **81.91%** | -| perl | stdin | 40 | **34** | 6 | **40** | 13 | **81.91%** | -| tcl | file | 40 | 9 | 31 | **40** | 40 | 3.95% | -| tcl | stdin | 40 | 9 | 31 | **40** | 40 | 3.95% | -| c | file | 30 | 0 | 30 | 0 | 28 | 0.00% | -| c | stdin | 30 | 2 | 28 | 30 | 28 | 0.80% | -| rust | file | 3 | 3 | 0 | 0 | 3 | 21.40% | -| rust | stdin | 3 | 0 | 3 | 3 | 3 | 0.00% | -| css | file | 10 | 3 | 7 | 0 | 3 | 9.58% | -| css | stdin | 10 | 6 | 4 | 10 | 3 | 11.13% | -| python | file | 45 | 22 | 23 | 0 | 45 | 14.98% | -| python | stdin | 45 | 21 | 24 | 3 | 45 | 14.88% | -| typescript | file | 56 | 21 | 35 | 0 | 56 | 14.13% | -| typescript | stdin | 56 | 0 | 56 | **56** | 56 | 0.00% | -| prose | file | 25 | 2 | 23 | 25 | 25 | 0.85% | -| prose | stdin | 25 | 2 | 23 | 25 | 25 | 0.85% | - -Where this corpus overlaps prior work it reproduces it: TypeScript over stdin reduces nothing -and is wholly uncovered (§29's case for `--language`), Python's two routes are near-identical -(4b.2's probe working), and exactly two prose files reduce over stdin (the lever-disposition's -`CODE_OF_CONDUCT.md` / `SECURITY.md` pair). - -## 4. Finding — §32 reaches the file-argument route - -`isCodeExtension` lists 19 extensions. `.pl` and `.tcl` are not among them, so a Perl or Tcl -file **passed by name** falls past every extension branch into the content probes and is -classified `markdown` on both routes: - -``` -shell file:{"contentType":"code"} stdin:{"contentType":"markdown"} -rust file:{"contentType":"code"} stdin:{"contentType":"text"} -perl file:{"contentType":"markdown"} stdin:{"contentType":"markdown"} -tcl file:{"contentType":"markdown"} stdin:{"contentType":"markdown"} -prose file:{"contentType":"markdown"} stdin:{"contentType":"markdown"} -``` - -§32, the scope document and CLAUDE.md all frame this as a defect of the pathless routes, with -"the file route refuses it" as the reassuring half. For any language whose extension is not in -that list, there is no reassuring half. The route is not the variable; **membership of a -hardcoded list of 19 extensions** is. - -## 5. Finding — the two defect shapes separate cleanly, and Perl carries both - -| bucket/route | classified `markdown` | of those, `measured:true` | classified `text` | of those, `measured:true` | reduced | -|---|---|---|---|---|---| -| perl / file | 13 | **13** | 27 | 0 | 7 md + **27 text** | -| perl / stdin | 13 | **13** | 27 | 0 | 7 md + **27 text** | -| tcl / file | 40 | 40 | 0 | — | 9 | -| shell / stdin | 40 | 40 | 0 | — | 20 | - -Two distinct failures, and Perl is the first corpus member to exhibit both at once: - -- **The §32 shape** (13 files): classified `markdown`, `#` comments harvested as headings, - drift reports `measured: true` on fabricated evidence. -- **The §28 shape** (27 files): classified `text`, no markers, no validator, drift honestly - reports `measured: false` — **and the item reduces anyway**, because §28 reports rather than - enforces outside validator-covered items. - -The worst case is the second shape, on the file route: - -``` -Unicode_Collate_Locale_ja.pl 57,037 -> 19 tokens (100.0%) fallbackUsed false - contentType text astCoverage.checked 0 driftScore 0 driftCoverage.measured false -``` - -A 57,037-token file deleted whole, by name, with every reporting field correctly saying that -nothing checked it. §32's `tclConfig.sh` (1,877 → 19) is the same defect two orders of -magnitude smaller. Nothing here is lying; nothing here is stopping it either. - -## 6. Finding — seam 2 measures viable - -Ground truth: the 264 code-bucket files must not be `markdown`; the 25 prose files must be. -The candidates are **shape** discriminators, not count thresholds — the disposition already -established that counts point the wrong way (`tclConfig.sh` 79 markers vs -`CODE_OF_CONDUCT.md` 12). - -| candidate | code → markdown (of 264) | prose → markdown (of 25) | -|---|---|---| -| V0 — what ships | **114** | 25 | -| V1 — require a non-`#` marker | 11 | **24** ← loses `CODE_OF_CONDUCT.md` | -| **V2 — V1, with the list regex repaired** | **12** | **25** | -| V3 — a heading needs corroboration | 12 | 25 | -| V4 — any two distinct signals | **7** | **25** | - -**V2 and V4 both hold all 25 prose files while removing ~90% of the misclassification.** -V4 separates best; V2 is the smaller change. Neither has a prose casualty. - -Residual leaks under V2 are honest rather than spurious — two shell scripts with `- ` lists, -three Tcl files whose `[...]` command syntax matches the markdown link regex, four pip files, -and three of this repo's own TypeScript sources whose doc comments genuinely contain fenced -markdown. - -**This contradicts the premise §32 deferred on.** §32 seam 2: *"could require more than one `#` -line, but that is a classifier change with blast radius over every prose item"*. The measured -blast radius over prose is **zero files** — because the fix is not a count threshold, which is -the form §32 imagined and the form the disposition separately proved wrong. - -## 7. Finding — the shipped list regex does not match markdown lists - -`RE_LIST` is `/(^|\n)(- |\* |\d+\.)\s+\S/`. The alternation already consumes the space after -`-`, and `\s+` then demands another one: - -``` -"- item" shipped false fixed true -"- item" shipped true fixed true -"* item" shipped false fixed true -"1. item" shipped true fixed true -``` - -So `- item` and `* item` — the two commonest list forms — do not match, and only ordered lists -and double-spaced bullets do. Measured, 21 of 25 prose files trip the shipped rule against -25 of 25 under the repair. This is why V1 loses `CODE_OF_CONDUCT.md` and V2 does not: V1 leans -on a rule that half-works. - -Not fixed here. It is a one-character class of change with a real blast radius on -classification, and it belongs to whoever implements the seam, measured under this harness. - -## 8. What this changes for Phase A - -1. **Seam 2 is back on the table and should be measured against seam 3, not assumed worse than - it.** It is cheap, it is the only lever with a measured zero-cost negative set, and it - attacks the fabrication at its source rather than compensating downstream. -2. **It is a mitigation, not the fix.** V2/V4 would convert the 13 `markdown`-shaped Perl files - into `text`-shaped ones — moving them from the §32 defect to the §28 defect, where they - still reduce unwitnessed. `Unicode_Collate_Locale_ja.pl` is already `text` and a classifier - change does not touch it. **Seam 3 remains load-bearing.** -3. **Phase A's population is bigger than "pathless".** The file route needs measuring for every - language outside those 19 extensions, and the `astCoverage` reassurance in CLAUDE.md's - invariant 10 entry should be re-read with §4 in hand. -4. **`isCodeExtension` is itself a finding.** A hardcoded 19-entry list decides whether a real - source file is checked at all, and `.pl`/`.tcl`/`.rb`/`.lua`/`.r`/`.swift`/`.kt` are outside - it. - -## 9. Correction owed to earlier documents - -- `docs/phase-4b-pathless-code-scope.md` §1 ("The CLI file argument is the only route that - works") and CLAUDE.md's Phase 4b.2 note ("the file route refuses it") are true for Python and - false in general. Both should say *the file route works for extensions in - `isCodeExtension`*. -- DECISIONS §32's seam-2 sentence should record that the seam was measured on 2026-08-06 and - that its blast radius over prose is zero under a shape discriminator, against 114 → 12 on - code. diff --git a/docs/phase-1-stabilization-summary.md b/docs/phase-1-stabilization-summary.md deleted file mode 100644 index 6e23fd4..0000000 --- a/docs/phase-1-stabilization-summary.md +++ /dev/null @@ -1,415 +0,0 @@ -# Phase 1.0 and Issue 2 — Summary Report - -> ## ⚠️ Scope: this is **not** all of Phase 1 -> -> This document covers **Phase 1.0** (Gateway stabilization, `1.0a` + `1.0b`) and -> **Issue 2** (the content-type contract). Those are the parts that are done. -> -> The original Phase 1 brief also contains **1b** (byte-identical fallback), **1c** -> (per-stage checkpointing) and **1d** (the drift-threshold investigation). None of those -> are covered here, and as of `4335c31` none of them have been started. See -> §8 "Outstanding work" for the real list — and read that section before treating anything -> here as a statement about Phase 1 as a whole. - -> ## ⚠️ Point-in-time record — not a live specification -> -> This document records work done **2026-08-01 → 2026-08-02**, across commits -> `4b11d7e..4335c31`. It is a historical account of what was done and why. It is **not** a -> description of current behavior and it is **not** a spec to implement against. -> -> Planning docs in this repo have twice been mistaken for current state after going stale. -> Before relying on any statement here, verify it against the source. Where this document -> and the code disagree, **the code is right and this document is out of date.** -> -> The load-bearing decisions from this report are duplicated into `CLAUDE.md` (invariants) -> and into comments at the relevant call sites, because those are read and this is not. -> -> ### Superseded: the drift-exemption rationale in §5.3 and §2.1 -> -> §5.3 justifies exempting dedup elisions from drift on the grounds that a marker is a -> pointer to restorable content. **That rationale does not hold on the Gateway path**, and -> the §2.1 measurement showing dedup at `S_k = 0.0000` was therefore reporting a pass that -> had not been earned. -> -> The Gateway's consumer is a stateless provider API. It has no rehydration mechanism, never -> calls `rehydrate_context`, and has no memory of a previous turn's request. Content elided -> from an outbound payload is not pointed at — it is deleted, and the model cannot resolve -> the marker by any means available to it. Cross-turn elision of a sole copy is lossy -> compression, and drift scoring it 0.60 was correct all along. -> -> Corrected in Commit B: `recoverable: true` is now set only when an intact copy survives -> elsewhere in the **same outbound payload**, which is the only version of the claim the -> stage can verify. See §9 for the measured cost. - -Covers Phase **1.0a** and **1.0b** of the Gateway stabilization work, plus the supporting -commits landed alongside them. - -**Status:** both phases complete, merged to `main`, CI green. -**Range:** `4b11d7e..61bd685` (2026-08-01 → 2026-08-02). - -> **Naming note.** The `1.0a` / `1.0b` labels originate in commit messages, not in -> `ROADMAP.md`. The roadmap's own "Phase 1" list was dropped as already-fixed (see -> `ROADMAP.md:207`). Treat these labels as a work-stream sequence, not a released -> milestone numbering. - ---- - -## 1. The problem both phases address - -`src/gateway/proxy.ts` called `runSessionDedupStage()` directly. It never invoked the -planner, the validators, `DriftTracker`, `ConfidenceLedger`, `DebtTracker`, or the -fallback resolver. - -Consequences before this work: - -- Invariant 3 (**fail-open fallback**) and invariant 5 (**drift threshold `S_k <= 0.40`**) - held for CLI and MCP modes only. They did not exist on the Gateway path. -- `cleanup:session-dedup` was the only stage that ever ran on production Gateway traffic, - and it ran with **zero** syntax/drift safety net. -- The Gateway also violated invariant 4 (**only `stage-registry` imports concrete - stages**). -- `fallbackUsed: false` in the Gateway trace was a **hardcoded literal**, surfaced to MCP - clients verbatim through the `tokendamper://session/{id}` resource. - -Phase 1.0a addressed the dishonest reporting. Phase 1.0b fixed the underlying gap. - ---- - -## 2. Commit timeline - -| Commit | Date | Summary | -|---|---|---| -| `4b11d7e` | 08-01 | `fix(planner)`: knapsack mode triggers on `targetReductionRatio` (precursor) | -| `ed4d141` | 08-02 | **Phase 1.0a** — stop asserting `fallbackUsed` on the proxy path | -| `1e839a2` | 08-02 | `docs`: correct v1.1.0 baseline, gateway-bypass limitation, tiktoken status | -| `b932180` | 08-02 | `docs`: add `CLAUDE.md` + stabilization planning docs | -| `2042ce6` | 08-02 | `chore`: add TokenDamper vs. Headroom benchmark harness | -| `aba84df` | 08-02 | `fix(test)`: tolerate expected fallback on Python fixtures (CI repair) | -| `61bd685` | 08-02 | **Phase 1.0b** — route the proxy path through `core/engine.optimize()` | - ---- - -## 3. Phase 1.0a — stop asserting an unevaluated safety property - -**Commit:** `ed4d141` · 3 files, +34 / −3 - -### Problem - -Both proxy recording sites wrote `fallbackUsed: false` into every `SessionTurn`. Because -`src/adapters/mcp/index.ts` serializes `session.turns` verbatim, that literal reached MCP -clients as though it were a computed result. It claimed a safety property that had never -been checked. - -### Change - -- `SessionTurn.fallbackUsed` made **optional** (`src/gateway/types.ts`). -- The field is **omitted** at both proxy recording sites rather than set to `false`. -- Rationale: an absent field is honest ("not evaluated"); a `false` is not. - -### Verification - -Regression test added to `test/unit/gateway.test.ts` asserting the field is absent — -failing before the change, passing after. - -### Deliberate non-goal - -1.0a did **not** attempt to make the Gateway safe. It only stopped the codebase from -lying about it, and explicitly deferred the repair to 1.0b. - ---- - -## 4. Interlude — CI repair (`aba84df`) - -Not part of either phase, but it blocked them. - -The precursor commit `4b11d7e` made the planner enter knapsack mode on -`targetReductionRatio`, fixing Issue 1 (budgets supplying only `--target-reduction-ratio` -silently resolved to `pass_through` with zero stages and guaranteed 0% reduction). - -That fix had a known, documented consequence: the knapsack stages now genuinely execute -against the humaneval Python fixtures and trip the `S_k <= 0.40` drift threshold -(Issue 3). Five tests hard-asserting `fallbackRate === 0` on those fixtures began failing -across all three Node matrix jobs. - -**Resolution:** updated the assertions in `test/integration/bench.test.ts`, -`test/unit/bench/m1_reverification.test.ts` and `test/unit/bench/runner.test.ts` to expect -the now-real 100% fallback rate on code content, each commented back to Issue 3. No -production logic changed. - ---- - -## 5. Phase 1.0b — route the proxy through the engine - -**Commit:** `61bd685` · 12 files, +435 / −87 - -### 5.1 Two landmines found during investigation - -Both would have shipped silently if the change had been made naively. - -**(a) It would have turned the Gateway into a no-op.** -`DEFAULT_CONFIG.budget` is `targetReductionRatio: 0` with no `maxInputTokens` -(`src/config/schema.ts:20`), so `plan()` returns `pass_through` with an **empty** -`stageIds`. Separately, `plan()` never listed `cleanup:session-dedup` at all. Routing -through `optimize()` as-is would have run **zero stages** and removed the Gateway's only -working feature. - -**(b) Drift would have vetoed deduplication on exactly the payloads it helps most.** -Drift is `1 − (0.6·symbolRetention + 0.4·markerRetention)`, computed bundle-wide. -Replacing a message with `[TokenDamper Elided: ref=…]` drops every AST symbol that message -contributed, so the score scales with *how much* was deduplicated. A representative code -payload measured **`S_k = 0.60`** against the 0.40 threshold — a forced fallback. - -### 5.2 Changes - -| File | Change | -|---|---| -| `src/core/model/types.ts` | Added `'session_dedup'` to `OptimizationMode` (additive union member) | -| `src/core/planner/index.ts` | New `session_dedup` mode planning exactly `['cleanup:session-dedup']`, selected via `config.planner.defaultMode`, taking precedence over budget-derived knapsack | -| `src/stages/cleanup/session-dedup.ts` | Tags its elisions `recoverable: true` | -| `src/core/ledger/drift-tracker.ts` | `resolveRecoverableElisions()` substitutes pre-optimization content for recoverable items before scoring | -| `src/gateway/proxy.ts` | Builds an `OptimizationRequest` and calls `optimize()`; shared `runGatewayOptimization()` helper replaces duplicated stage-call blocks in both provider paths | -| `src/gateway/types.ts` | `fallbackUsed` comment updated — now a computed value | -| `test/unit/planner.test.ts` | `session_dedup` mode plans only the dedup stage, even with a knapsack-triggering budget | -| `test/unit/drift-tracker.test.ts` | Recoverable elision exempt; lossy elision still scored | -| `test/unit/gateway.test.ts` | Fail-open byte-identity test; code-payload dedup test; 1.0a test inverted to assert a computed `fallbackUsed` | -| `CHANGELOG.md`, `DECISIONS.md`, `CLAUDE.md` | Documentation (DECISIONS §16 records the drift rationale) | - -### 5.3 Key design decisions - -**Planner pinned to `session_dedup`.** Cross-turn deduplication is the only transform -currently safe for live provider payloads. `compression:token-hashing` writes bare -`` markers into JSON-shaped message content (Issue 2), so broadening the -Gateway's stage list is gated behind content-type tagging. Pinning the mode also gave -`config.planner.defaultMode` — previously dead config — a real purpose. - -**Drift exempts recoverable references, not lossy ones.** A dedup marker is a *pointer*: -the full text is retained in the session store under `originalContentHash` and is -restorable on demand. Nothing is irrecoverably lost, so nothing should be scored as loss. -The exemption keys on an **explicit `recoverable` flag**, deliberately not inferred from -`elided` or `originalContentHash` — `token-hashing` sets both and must stay fully scored. -Full rationale and alternatives considered in `DECISIONS.md` §16. - -**Gateway uses `finalBundle`, never `emittedOutput`.** The fallback resolver renders a -bundle by joining item contents with newlines, which is not a valid provider API payload -(this is the mechanism behind Issue 5). Mapping `finalBundle` items positionally back onto -the parsed request preserves payload shape, and because the engine returns the *original* -bundle whenever fallback fires, the request body is reproduced byte-for-byte. The Gateway -therefore sidesteps Issue 5 structurally rather than by test enforcement. - -**`ConfidenceLedger` is per-request, not session-scoped.** Confidence decays as -`initialConfidence × 0.9^(currentTurn − lastAccessedTurn)`. A persistent ledger would drop -earlier turns below `validation.minimumConfidence` (default `1`) and force a fallback on -every turn after the first. Cross-turn confidence decay needs its own threshold policy and -was deliberately excluded. - -### 5.4 Verification - -Behavior was probed against the built output **before** the tests were written, so the -assertions encode observed behavior rather than assumptions. - -Gateway, two-turn sessions with repeated content: - -| Scenario | `fallbackUsed` | Elided | Body byte-identical | Tokens saved | -|---|---|---|---|---| -| Plain prose | `false` | yes | no | 4 | -| Imperative constraint directive | **`true`** | no | **yes** | 0 | -| Symbol-dense code | `false` | yes | no | 8 | - -Drift exemption, identical elision content: - -| Elision type | `S_k` | Fallback | -|---|---|---| -| `session-dedup` (recoverable) | **0.00** | no | -| `token-hashing` (lossy) | **0.60** | **yes** | - -The constraint-directive row is the guarantee that did not previously exist: validation -detects the dropped directive and the caller receives their original payload unchanged. -The drift table confirms the exemption is load-bearing — without it the code payload would -have fallen back — while lossy compression remains fully policed. - -**Suite:** 283 tests passing (up from 279; +4 new). `typecheck`, `lint`, `build` all clean. - ---- - -## 6. Net effect - -- Invariants 3 and 5 now hold on **all three entry modes**, not just CLI and MCP. -- Invariant 4 restored for the Gateway — it no longer imports a concrete stage. -- `fallbackUsed` carries a genuinely computed value on the proxy path again. -- Gateway reduction behavior is **unchanged** on the happy path; the change adds a safety - net rather than altering what gets compressed. -- `config.planner.defaultMode` is now functional instead of inert. - ---- - -## 7. Known limitations after Phase 1 - -Carried forward deliberately, not oversights: - -1. **Gateway runs one stage only.** Broadening to the knapsack set is gated on Issue 2 - (content-type tagging), which would otherwise corrupt JSON payloads in production. -2. **No cross-turn confidence decay** on the Gateway, for the `minimumConfidence` reason - above. -3. **Issue 5 unfixed for CLI/MCP.** The Gateway avoids it structurally, but the shared - fallback path still re-renders the bundle rather than echoing raw input bytes. -4. **Still a single global validate→fallback gate.** A failing stage discards all prior - valid reductions; per-stage checkpointing remains outstanding. -5. **`npm run format` fails on 94 files** repo-wide. Pre-existing, and not part of the CI - workflow (which runs typecheck → lint → build → test). - -## 9. Addendum (2026-08-02) — measured cost of correcting the drift exemption - -Commit B narrowed `recoverable: true` to elisions whose referent demonstrably survives in -the same outbound payload. Measured through the real Gateway proxy path, two turns per -session, `--mock-upstream`: - -> **These figures are unaffected by the token-estimator unification (`1b1e999`, -> DECISIONS.md §19) and must not be re-corrected.** They are derived from HTTP body byte -> lengths on the Gateway path, not from `summary.tokenEstimate`. The Gateway's internal -> counters do change unit — `rawTokens` 8,470 → 10,059 on a 36 KB payload — but its -> `dedupRatio` moves only 49.79% → 49.82%, because both of its sides already used the same -> estimator. The figures corrected by that commit are the CLI, MCP and bench ones; see -> `NOTES-FOR-DOCS.md` for the full inventory of which records are valid and which are not. - -**Cross-turn dedup — sole copy, no surviving referent (what the Gateway did before):** - -| Payload | Saved | Fallback | -|---|---|---| -| `tool_output.json` | 2,987 / 3,013 = **99.14%** | no | -| `codebase.py` | 0 / 4,238 = **0.00%** | **yes** | -| `sample_logs.txt` | 0 / 2,728 = **0.00%** | **yes** | - -**Within-payload duplication — a copy is preserved (what still deduplicates):** - -| Payload | Saved | Fallback | -|---|---|---| -| `tool_output.json` | 5,974 / 9,031 = **66.15%** | no | -| `codebase.py` | 8,434 / 12,707 = **66.37%** | no | -| `sample_logs.txt` | 5,413 / 8,176 = **66.21%** | no | - -**Reading these honestly.** Code and logs went from ~99% "savings" to 0% and a fallback. -That is not a regression: the prior number depended on sending the model markers it had no -way to resolve, and the fallback is the system correctly declining a lossy transform it -cannot justify. The real number replaced an inflated one. - -The `tool_output.json` cross-turn row still shows 99.14%, and that is **not** a sign the -exemption survives there — it is the §2.2 vacuity again. `contentType` is still hardcoded -`text` on that path, so `extractSymbols` harvests nothing from JSON and drift is 0.00 by not -looking. **Commit C (the relabel) is expected to flip that row to 0% and a fallback**, for -the same reason the Python row already does. It is listed here so the change is attributable -when it happens rather than read as a regression introduced by the relabel. - -**What this means for the Gateway.** After Commit C, cross-turn deduplication will -effectively stop contributing on any symbol-bearing content, which is most realistic agent -traffic. Within-payload exact duplication does occur — repeated tool schemas, a file pasted -into several messages — but it is not the common shape of a conversation, where each message -appears once. - -Said plainly: **the Gateway's near-term dedup value is likely close to zero.** That does not -make the Gateway pointless; it relocates its value to cache-aware prefix stability and to -the knapsack stages, which Issue 2 is the gate on. It does mean cross-turn deduplication -should not be cited as a headline capability until there is a mechanism that lets the model -resolve a marker — which, on a stateless provider API, there currently is not. - -## 8. Outstanding work - -> Replaces the former "Suggested next step" section, which named Issue 2. Issue 2 is now -> **done** (`29f66b3`, `e9ea50d`, `b11dcb0`, `642abcb`, `ac16cec`, `4335c31`) — though not -> at the seam it was specified at; see `NOTES-FOR-DOCS.md`. The §9 prediction about the -> `tool_output.json` row was confirmed on landing, also recorded there. - -Everything below is **not started** as of `4335c31`, verified against source rather than -recalled. Phase 1 is not finished. - -### 1b — byte-identical fallback (Issue 5) - -`session.json` emits **−1.39%** on fallback: the output is *larger* than the input, because -the fallback path re-renders `currentBundle` by joining item contents with newlines instead -of echoing the original bytes. - -The Gateway sidesteps this structurally — `src/gateway/proxy.ts` maps `result.finalBundle` -back onto the parsed payload and never touches `emittedOutput` — but that is a local -workaround on one path, not a fix. **The CLI and MCP paths still re-render.** The agreed -direction is to split fallback into raw passthrough (byte-identical echo, bypassing the -bundle render model) and bundle rendering (success path only), so byte-identity is -structural rather than test-enforced. - -### 1c — per-stage checkpointing - -Still a single global validate→fallback gate: one failing stage discards every prior valid -reduction. Cited twice as the reason elision refusal must skip an item and continue rather -than abort the stage (`docs/issue-2-content-type-contract-design.md` §3.4.1, and the -`elideItem` doc comment) — aborting today would convert a placeholder defect into a -whole-pipeline fallback. - -**Design input discovered during Issue 2, and it complicates the premise.** -`src/core/validation/index.ts` runs `validateBundleAst(after)` over **every item in the -final bundle**, not only the items a stage changed. Per-stage checkpointing assumes a -validation failure can be attributed to the stage that caused it. Sometimes it cannot: - -- A validation failure can originate in an item **no stage touched**. This is not - hypothetical — it is exactly how the fenced-prose defect in `DECISIONS.md` §17 was found. - With `contentType` newly computed, a message quoting a code snippet failed the TypeScript - validator on **turn 1**, where `cleanup:session-dedup` has no previous block hashes and - cannot elide anything. Nothing had been transformed, so no rollback could have fixed it. -- Two of the four checks in `validate()` are **bundle-scoped, not item-scoped**: constraint - directive retention compares `before` against all `after` content joined, and - `DriftTracker` computes `S_k` over whole-bundle symbol sets. Neither yields a per-stage or - per-item attribution as written. - -So "roll back only the failing stage" needs a prior answer to *which stage failed*, and for -a class of failures the honest answer is "none of them." A checkpointing design that assumes -attributability will silently roll back an innocent stage. Recommended first step is to -establish attribution — validate the delta a stage produced, not the whole bundle — before -building rollback on top of it. Recorded in `CLAUDE.md` as a gotcha as well, since that is -what gets read. - -### 1d — drift threshold investigation - -**Investigation done 2026-08-03; recorded in `docs/phase-1d-drift-investigation.md`. The -threshold is unchanged and the remedy is undesigned.** - -The brief: `codebase.py` aborts on `S_k = 0.60 > 0.40`; investigate what drives the score -and decide whether the threshold should be content-type-specific. What the measurement -found, in short: - -- `extractSymbols` returns **empty** on the optimized side — correctly, because - `token-hashing` replaces the item's whole content with a 77-byte placeholder. Nothing to - extract from, not a misparse. No validator participates in extraction at all. -- `0.60` is a **formula constant**, equal to `w_AST`, produced whenever `R_AST = 0` and - `R_struct = 1`. Identical to 4 dp across Python, TypeScript and JavaScript. -- The cause is **granularity**: `token-hashing` is whole-item, and `createContextBundle` - makes a single-item bundle for CLI/bench, so `R_AST` is a boolean rather than a ratio and - a single-item code bundle can never pass. Given granularity, the metric grades fine - (1-of-4 hashed → `S_k = 0.34`, passes; 2-of-4 → `0.47`, fails). -- Separately: for code, **`R_struct` is pinned at 1.0**, so 40% of the metric does no work - and `S_k` is capped at `0.60`. See `DECISIONS.md` §18. - -`DriftTracker` still has a single scalar `maxDriftThreshold` (default `0.40`, -`drift-tracker.ts:83`) with no content-type branching; `--max-drift` remains the only -override. On present evidence a content-type-specific *threshold* looks like the wrong -instrument — but that is an argument against the framing, not a decision. - -Three things are adjacent but are **not** 1d, and should not be counted as it: -`tokendamper-headroom-known-issues.md` Issue 3 states the problem and then *retracts* its -main supporting evidence (Headroom did not independently choose `router:noop`; it hit a -20-second backend timeout); `aba84df` changed tests to tolerate the abort rather than -investigate it; and `DECISIONS.md` §16 rejected a *path*-specific threshold for a different -problem. `docs/issue-2-content-type-contract-design.md` §4 explicitly puts threshold changes -out of scope. - -**Anything measured for 1d before now must be re-measured.** `b11dcb0` narrowed the drift -exemption and `ac16cec` made `DriftTracker` see JSON as JSON for the first time, so drift -does not behave as it did when 1d was written. - -One observation that fell out of the bench run for `4335c31`, offered as a starting point -rather than as analysis: at `targetReductionRatio: 0.30`, **nine of the ten bundled bench -fixtures fall back at exactly `S_k = 0.60`** — Python, TypeScript and JavaScript alike. That -value is what `1 - (0.60 × R_AST + 0.40 × R_struct)` yields when `R_AST = 0` and -`R_struct = 1`, i.e. total AST-symbol loss with structural markers fully intact. The -constant recurrence across languages suggests the score is being driven by one mechanism, -not by per-fixture content. - -### Phase 2 — security audit - -Not started. diff --git a/docs/phase-1d-drift-investigation.md b/docs/phase-1d-drift-investigation.md deleted file mode 100644 index 1da12b3..0000000 --- a/docs/phase-1d-drift-investigation.md +++ /dev/null @@ -1,435 +0,0 @@ -# Phase 1d — Semantic Drift Investigation (Diagnostic Record) - -> ## ⚠️ What this is, and what it is not -> -> This is the **investigation** half of Phase 1d, recorded **2026-08-03** at commit -> `3dc30f4`. The brief was: *"`codebase.py` aborts on semantic drift 0.60 > 0.40. -> Investigate what drives the score, then tell me whether the threshold should be -> content-type-specific rather than shared across prose, logs, and code. Do not change the -> threshold without making the case first."* -> -> **The threshold has not been changed, and this record does not propose changing it.** -> Every number below is measured, with the reproduction script described in §9. -> -> ### One measurement caveat that qualifies the whole document -> -> The benchmark fallback rate cited here (100%, 10/10 fixtures) was measured with the -> engine's automated-rehydration recovery path **switched off** — not deliberately, but -> because `src/bench/runner.ts:45` called `optimize(request)` with no options, so no -> `TokenHasher` and no `ConfidenceLedger` reached the engine and -> `attemptAutomatedRehydration` returned immediately on `if (!hasher && !ledger)`. See §8. -> -> **Corrected 2026-08-03 — the valve is now enabled and §10 supersedes the fallback rate: -> it drops to 0.40.** §10 also documents what enabling it exposed: the benchmark's -> `avgReduction` figure is fabricated by an estimator mismatch, and actual savings are zero -> either way. **Corrected again 2026-08-03 — the estimator is now unified (`1b1e999`, -> DECISIONS.md §19) and §12 supersedes every reduction figure in §10: they are all 0.00%, -> which is what the tooling now reports. Read §10 and §12 before citing any benchmark -> number from this document.** -> -> The per-fixture drift *mechanics* in §2–§6 are unaffected by it — they are computed from -> `DriftTracker` directly on before/after bundles, not from the benchmark's fallback -> outcome. - ---- - -## 1. Summary - -The prevailing hypothesis was that the `0.40` drift threshold is too sensitive for code. -That hypothesis is **not supported**. Three things are true instead: - -1. `extractSymbols` returns an **empty set** on the optimized side of every code fixture — - and that is *correct*, not a misparse. `compression:token-hashing` replaces the item's - entire content with a 77-byte placeholder, so there is nothing left to extract. -2. **No validator participates in symbol extraction.** Extraction is pure regex over - `item.content`. The "wrong validator is parsing it" hypothesis is inapplicable. -3. **`0.60` is a formula constant, not a measurement.** It is `w_AST` exactly, produced - whenever `R_AST = 0` and `R_struct = 1`. - -The real cause is **granularity** (§6): `token-hashing` is whole-item and all-or-nothing, -and `createContextBundle` produces a **single-item** bundle for CLI and bench input. Every -symbol therefore dies simultaneously, `R_AST` can only be `0` or `1`, and there is no -fractional case for the metric to measure. - -A separate, independent finding (§5) stands regardless of how the granularity problem is -solved: for code, **40% of the metric does no work**. - ---- - -## 2. Q1 — what `extractSymbols` returns on the optimized side - -Measured on the first three bundled bench fixtures, plan -`topology_knapsack` → `cleanup:constraint-preservation` → `pruning:topology-pruner` → -`compression:token-hashing` → `compression:delta-compression`. Only -`constraint-preservation` and `token-hashing` reported `changed`. - -| Fixture | Content before → after | symbolsBefore | symbolsAfter | -|---|---|---|---| -| `HumanEval/0` | 348 B → 77 B | 2 — `{fn:has_close_elements, import:typing}` | **0** — `{}` | -| `HumanEval/1` | 504 B → 77 B | 3 — `{fn:is, fn:separate_paren_groups, import:typing}` | **0** — `{}` | -| `HumanEval/2` | 328 B → 77 B | 1 — `{fn:truncate_number}` | **0** — `{}` | - -The optimized content in every case is a single placeholder: - -``` - -``` - -Empty is the right answer. A 77-byte hash contains no `def`, no `class`, no `import`. The -extractor is reporting accurately on content that no longer exists. - ---- - -## 3. Q2 — no validator is involved in extraction - -`DriftTracker.extractSymbols` reads `item.content` and applies eight regexes. It never -calls `selectValidator`, and it does not parse. The only `contentType`-sensitive branch in -the whole function is the `jsonkey:` harvest at `drift-tracker.ts:240`, gated on -`item.contentType === 'json'` — which never applies to code. - -Demonstrated by holding content fixed and varying the tag across every legal value: - -``` -contentType=code symbols n=4 {fn:render, fn:helper, type:Widget, import:os} -contentType=text symbols n=4 {fn:render, fn:helper, type:Widget, import:os} -contentType=json symbols n=4 {fn:render, fn:helper, type:Widget, import:os} -contentType=markdown symbols n=4 {fn:render, fn:helper, type:Widget, import:os} -contentType=unknown symbols n=4 {fn:render, fn:helper, type:Widget, import:os} -``` - -Identical across all five. Symbol extraction cannot be fixed by correcting a tag or a -validator, because neither reaches it. - ---- - -## 4. Q3 — `0.60` is a formula constant - -``` -S_k = 1 − (w_AST · R_AST + w_struct · R_struct) w_AST = 0.60, w_struct = 0.40 -``` - -With `R_AST = 0` and `R_struct = 1`: - -``` -S_k = 1 − (0.60 × 0 + 0.40 × 1) = 1 − 0.40 = 0.60 -``` - -`0.60` **is** `w_AST`, reproduced by construction. It carries no information about *how -much* was lost — only that `R_AST` reached zero. Measured identical to four decimal places -on every fixture, across Python, TypeScript and JavaScript. A value that is constant across -three languages and every payload size is not measuring the payload. - ---- - -## 5. Finding — for code, 40% of the metric is a constant - -**This finding is independent of the granularity cause and survives whatever fix is chosen -for it.** Recorded in `DECISIONS.md` §18. - -`extractMarkers` harvests: `filepath:` (from `item.path`), markdown headings, code fences, -`TD_PRESERVE:` directives, and section delimiters. For a source file with none of those in -its content, the marker set is exactly one entry — `filepath:` — derived from -`item.path`, which is **metadata**. Elision rewrites `content`; it never touches `path`. - -So `R_struct = 1.0` by construction for code: - -| Fixture | markersBefore | markersAfter | R_struct | -|---|---|---|---| -| `HumanEval/0` | 1 — `{filepath:src/has_close_elements.py}` | 1 — same | 1.0000 | -| `HumanEval/1` | 1 — `{filepath:src/separate_paren_groups.py}` | 1 — same | 1.0000 | -| `HumanEval/2` | 1 — `{filepath:src/truncate_number.py}` | 2 — same + a directive | 1.0000 | - -Three consequences: - -1. The `w_struct` term contributes a **fixed 0.40** to retention. It cannot vary, so it - cannot discriminate between a good transform and a catastrophic one. -2. `S_k` for code is therefore confined to **`[0.00, 0.60]`** — it is - `0.6 · (1 − R_AST)` in disguise, a one-variable metric wearing a two-variable formula. -3. The `0.40` threshold sits at **two-thirds of a maximum the metric can never exceed**. - -Reachable values, with `R_struct` pinned: - -| R_AST | R_struct | S_k | -|---|---|---| -| 1.0 | 1.0 | 0.0000 | -| 0.5 | 1.0 | 0.3000 | -| 0.0 | 1.0 | **0.6000** ← ceiling | -| 0.0 | 0.5 | 0.8000 (unreachable for marker-free code) | -| 0.0 | 0.0 | 1.0000 (unreachable for marker-free code) | - -This is not a badly-tuned threshold. It is a metric whose structural half does no work on -the content type the product exists to serve. Tuning the number would paper over that. - -> **Note.** The `R_struct < 1` rows above *are* reachable for Python today, but only via a -> defect: `extractMarkers` matches `/^#{1,6}\s+/` and Python comments satisfy it. See §7. - ---- - -## 6. The cause — transform granularity, not the metric - -`compression:token-hashing` maps over bundle items and, for each eligible one, replaces its -**entire content** with a single placeholder (`token-hashing.ts:37–90`). There is no -sub-item granularity: an item is hashed whole or not at all. - -`createContextBundle` (`constructors.ts:91`) builds `items = freeze([item])` — a **single -item** — from CLI and bench input. - -Composing those two facts: on the CLI and bench paths, one successful `token-hashing` -destroys every symbol in the bundle simultaneously. `R_AST` is therefore not a ratio in -practice; it is a **boolean**, `0` or `1`. `S_k = 0.60` is deterministic, exceeds `0.40`, -and falls back — **every time, structurally**. `token-hashing` can never succeed on a -single-item code bundle. - -**Measured caveat:** a file with no extractable symbols behaves differently, because -`R_AST` defaults to `1.0` when `symbolsBefore.size === 0`: - -``` -symbol-free code item : symbolsBefore=0 R_AST=1.0000 S_k=0.0000 fallback=false -``` - -So the rule is: *given at least one extractable symbol*, a single-item code bundle always -fails. A symbol-free file passes trivially — which is its own small dishonesty, since it -passes by having nothing to measure. - -### The metric grades correctly once it has granularity - -The same stage, same threshold, on multi-item bundles where some items are below -`token-hashing`'s 40-character eligibility floor and therefore survive intact: - -| Bundle shape | Items hashed | R_AST | R_struct | S_k | Outcome | -|---|---|---|---|---|---| -| 1 item (**the CLI/bench shape**) | 1 / 1 | 0.0000 | 1.0000 | **0.6000** | fallback | -| 2 items, both eligible | 2 / 2 | 0.0000 | 1.0000 | **0.6000** | fallback | -| 4 items, all eligible | 4 / 4 | 0.0000 | 1.0000 | **0.6000** | fallback | -| 4 items, 1 long + 3 short | 1 / 4 | 0.4286 | 1.0000 | 0.3429 | **ok** | -| 4 items, 2 long + 2 short | 2 / 4 | 0.2222 | 1.0000 | 0.4667 | fallback | -| 4 items, 3 long + 1 short | 3 / 4 | 0.0909 | 1.0000 | 0.5455 | fallback | - -The middle rows are the point: given a fractional `R_AST`, `S_k` moves smoothly and the -`0.40` threshold discriminates sensibly — one-of-four passes, two-of-four fails. The metric -is not broken. It is being fed a transform that only ever hands it `0` or `1`. - -Note also that the top three rows are identical regardless of bundle size. `S_k` is -scale-invariant here precisely because *all* symbols die in every case. - ---- - -## 7. A real extraction defect — in `extractMarkers`, not `extractSymbols` - -`extractMarkers` classifies any line matching `/^#{1,6}\s+/` as a markdown heading. **Python -comments match that pattern.** - -``` -1 item, python WITH # comments R_AST=0.0000 R_struct=0.3333 S_k=0.8667 FALLBACK -``` - -Two `#` comments push `S_k` to `0.8667` — *above the `0.60` ceiling* §5 establishes for -code, because the comment-derived pseudo-markers are destroyed along with the content while -`filepath:` survives. Drift on Python therefore scales with comment density. - -This is a straight bug and is fixed separately, ahead of any granularity work, because it -contaminates every Python measurement taken while designing that work. - ---- - -## 8. The fallback rate was measured with the recovery valve disabled - -`src/bench/runner.ts:45` calls `optimize(request)` with **no options**. The engine's -recovery path is: - -```ts -function attemptAutomatedRehydration(bundle, hasher?, ledger?, turn = 1) { - if (!hasher && !ledger) { - return undefined; - } - ... -``` - -With neither supplied it returns immediately. So on validation failure the engine's -documented ability to un-hash placeholders and re-validate **never runs** in the benchmark, -and the observed 100% fallback rate is a number taken with that machinery switched off. - -This is the fifth instance of the same pattern in this project — a result produced by -machinery that never executed, after the hardcoded `fallbackUsed: false`, the vacuous JSON -AST check, the vacuous JSON drift check, and the unreachable `post_condition_rejected`. It -is why `CLAUDE.md` invariant 10 exists. - -**Consequence for this record:** §2–§7 stand, because they are computed from `DriftTracker` -directly. The benchmark fallback *rate* does not stand until re-measured with the valve -enabled. - ---- - -## 9. Reproduction - -All figures come from two scripts run against `dist/` at `3dc30f4`: - -1. **Per-fixture drift decomposition** — loads the bundled fixture set, re-runs - `plan()` + `executeBuiltInStage()` step by step so the *pre-validation* bundle is - observable (the engine returns `request.bundle` on fallback, which would hide exactly - what is being measured), then prints the symbol and marker sets on both sides alongside - `DriftTracker.calculateDrift`'s own component ratios. -2. **Bundle-shape sweep** — synthesizes 1/2/4-item bundles with a controlled mix of items - above and below `token-hashing`'s 40-character floor, and runs the stage in isolation. - -Budget in both cases: `targetReductionRatio: 0.30` over `loadConfig()` defaults. Without a -budget the planner returns `pass_through` with zero stages and every figure is trivially -zero. - ---- - -## 10. Addendum (2026-08-03) — the recovery valve enabled, and what it exposed - -§8 said the 100% fallback rate was measured with `attemptAutomatedRehydration` switched -off. It has now been enabled — `src/bench/runner.ts` passes a fresh `TokenHasher` per run, -which is what the engine needs both to *make* the placeholders reversible and to reverse -them. Re-measured on the same ten fixtures at `targetReductionRatio: 0.30`: - -| Metric | Valve off | Valve on | Real? | -|---|---|---|---| -| `fallbackRate` | 1.00 | **0.40** | **yes** | -| `totalValidationIssues` | 11 | **4** | **yes** | -| `avgReduction` | 0.0000% | **7.8217%** | **no — see below** | -| Actual bytes saved | 0 | **0** | — | - -### The fallback drop is real - -Six of ten fixtures now recover instead of falling back. The engine detects the drift -failure, rehydrates the `` placeholder back to its original content, -re-validates, and passes. That machinery works and had simply never been switched on. - -### The reduction figure is fabricated - -**Every fixture's output is byte-identical to its input** — verified directly, not inferred: - -| Fixture | in bytes | out bytes | identical | in tokens | out tokens | `ceil(len/4)` | reported | -|---|---|---|---|---|---|---|---| -| `HumanEval/0` | 348 | 348 | **yes** | 106 | 87 | **87** | 17.92% | -| `HumanEval/1` | 504 | 504 | **yes** | 146 | 126 | **126** | 13.70% | -| `HumanEval/2` | 328 | 328 | **yes** | 91 | 82 | **82** | 9.89% | -| `HumanEval/3` | 446 | 446 | **yes** | 126 | 112 | **112** | 11.11% | -| `HumanEval/4` | 386 | 386 | **yes** | 109 | 97 | **97** | 11.01% | -| `CodeXGLUE/py/102` | 164 | 164 | **yes** | 48 | 41 | **41** | 14.58% | - -`out tokens` equals `ceil(len/4)` exactly in every row. The cause is that **two different -token estimators are in use**, and a reduction ratio compares one against the other: - -| Estimator | Sites | -|---|---| -| `EnhancedHeuristicTokenizer` | `constructors.ts:108` (`createContextBundle` — the **input** side), `:131` (`createBundleFromItems`) | -| Naive `ceil(len / 4)` | `engine/index.ts:398` (`attemptAutomatedRehydration`), `trace/index.ts:56`, `gateway/proxy.ts:505,627`, `constraint-preservation.ts:103`, `session-dedup.ts:186`, `delta-compression.ts:273` | - -The input bundle is measured with the tokenizer; every bundle a stage or the rehydrator -produces is measured with `ceil(len/4)`. On this corpus the tokenizer runs 11–22% above -`len/4`, so identical bytes register as an 11–22% saving. - -**This is not confined to the benchmark.** Any successful optimization on the CLI path -reports a reduction inflated by the same gap, because `createContextBundle` uses the -tokenizer and every stage's output bundle uses `ceil(len/4)`. It was invisible until now -only because everything was falling back, and on fallback `finalBundle = request.bundle`, -so both sides used the tokenizer and the ratio was a true 0%. - -The Gateway is *not* affected the same way: it builds its input bundle with `ceil(len/4)` -too, so both sides use the same estimator. The Gateway figures reported earlier in this -work (66%, 98.59%, 0%) were computed from actual HTTP body byte lengths, and stand. - -### What the valve does and does not buy - -It converts "fallback, 0% saved" into "success, 0% saved". Actual token savings on this -corpus remain **exactly zero** with the valve on, because the engine's recovery *is* the -undoing of the compression. That is a better outcome than a fallback — the fail-open path -is no longer being exercised as though it were normal operation — but it is not a reduction -win, and the `avgReduction` figure must not be cited as one. - -This is the sixth instance of the vacuity pattern in this project, and the first where the -fabricated value reports **success** rather than a passed check. It is why the delta was -reported before any design work rather than after. - -### Consequence for §5 and §6 - -Unchanged. Both are computed from `DriftTracker` on before/after bundles and do not touch -token estimates. The four remaining fallbacks are still drift at exactly `0.60` -(`CodeXGLUE/py/101`, `ts/201`, `js/301`) plus the pre-existing unclosed-bracket AST failure -on `CodeXGLUE/ts/202`. - -Note the recovery valve is, in effect, an accidental partial implementation of per-stage -rollback for one specific stage: it undoes `token-hashing` and keeps the earlier stages' -work. Phase 1c should account for it rather than build a second mechanism beside it. - ---- - -## 11. What this record does **not** conclude - -- **It does not answer the brief's actual question** — whether the threshold should be - content-type-specific. It answers the prerequisite: what drives the score. On the present - evidence a content-type-specific *threshold* looks like the wrong instrument, because the - problem is not that `0.40` is mis-set for code but that the metric's structural half is - inert for code (§5) and the transform hands it a boolean (§6). That is an argument - against the framing, not a decision. -- **It does not propose the granularity fix.** Sub-item hashing granularity is the - indicated direction, but the design is separate and unwritten. -- **It does not re-validate the benchmark fallback rate.** See §8. - ---- - -## 12. Addendum (2026-08-03) — the estimator unified; the real reduction on this corpus is zero - -§10 identified the `avgReduction: 7.8217%` as fabricated by an estimator mismatch and -declined to cite it. The mismatch is now fixed (`1b1e999`, DECISIONS.md §19): all -measurement routes through `estimateTokens` / `estimateBundleTokens` in -`src/core/hashing/tokenizer.ts`, and `countTokens` is called from exactly one place. - -**Every per-fixture figure in the §10 table is 0.00%.** Re-measured on the same ten -fixtures at `targetReductionRatio: 0.30`: - -| Fixture | in bytes | out bytes | identical | §10 reported | now | -|---|---|---|---|---|---| -| `HumanEval/0` | 348 | 348 | **yes** | 17.92% | **0.00%** | -| `HumanEval/1` | 504 | 504 | **yes** | 13.70% | **0.00%** | -| `HumanEval/2` | 328 | 328 | **yes** | 9.89% | **0.00%** | -| `HumanEval/3` | 446 | 446 | **yes** | 11.11% | **0.00%** | -| `HumanEval/4` | 386 | 386 | **yes** | 11.01% | **0.00%** | -| `CodeXGLUE/py/101` | 192 | 192 | **yes** | 0.00% | **0.00%** | -| `CodeXGLUE/py/102` | 164 | 164 | **yes** | 14.58% | **0.00%** | -| `CodeXGLUE/ts/201` | 130 | 130 | **yes** | 0.00% | **0.00%** | -| `CodeXGLUE/ts/202` | 49 | 49 | **yes** | 0.00% | **0.00%** | -| `CodeXGLUE/js/301` | 112 | 112 | **yes** | 0.00% | **0.00%** | - -| Metric | §10 (valve on) | now | -|---|---|---| -| `avgReduction` | 7.8217% (fabricated) | **0.0000%** | -| `fallbackRate` | 0.40 | **0.40** — unchanged | -| `totalValidationIssues` | 4 | **4** — unchanged | - -The two unchanged metrics are the point: nothing about the engine's *behavior* moved. Only -the arithmetic used to describe it did. §10's "actual token savings on this corpus remain -exactly zero" is now what the tooling reports rather than something a reader has to know. - -**§10's four already-zero rows were not clean either.** Those are the fallbacks, where the -*benchmark's* ratio compared two tokenizer-derived bundle summaries and correctly read 0%. -Their `trace.tokenAfter` was still the naive count, so the MCP adapter — which divides -`trace.tokenAfter` by `trace.tokenBefore` — reported a saving on them anyway. -`CodeXGLUE/py/101` read 53 → 48, a phantom 9.4% **on a pure fallback**, where the emitted -text is `request.rawInput` verbatim. It is now 53 → 53. - -**Accuracy is a separate, still-open question.** Scored against real `cl100k_base` over -these ten fixtures plus the four `tokendamper-benchmark/test_data` payloads, -`EnhancedHeuristicTokenizer` is the **less** accurate of the two estimators that were in -use — mean absolute error 24% against `ceil(len / 4)`'s 17%, max 56% against 44%. It was -adopted for the seam, not the numbers (DECISIONS.md §19). Recalibrating it, or landing -`createTiktokenAdapter` against a real encoder, is now a one-line change to -`DEFAULT_TOKENIZER`. - -**A weak test surfaced while checking this, not fixed here.** `test/integration/bench.test.ts` -Test 2 asserts `avgReductionRatio >= 0.40` and passes — genuinely, in bytes: 151 → 77 and -153 → 77 on two synthetic prose fixtures. But the 77 bytes are a bare `` -placeholder, and it passes the drift gate only because prose with no extractable symbols -takes `R_AST`'s `symbolsBefore.size === 0` default of 1.0 — the "passes by having nothing to -measure" case §6 already flagged. The threshold is being met by total content destruction on -inputs the metric cannot grade. Not an instance of the estimator bug; recorded so it is not -mistaken for evidence that the pipeline reduces anything. - -**§5 and §6 are unaffected**, for the same reason §10 gave: both are computed from -`DriftTracker` on before/after bundles and never touch a token estimate. The four remaining -fallbacks are still drift at exactly `0.60` plus the unclosed-bracket AST failure on -`CodeXGLUE/ts/202`. diff --git a/docs/phase-1d-granularity-design.md b/docs/phase-1d-granularity-design.md deleted file mode 100644 index a084e85..0000000 --- a/docs/phase-1d-granularity-design.md +++ /dev/null @@ -1,359 +0,0 @@ -# Phase 1d — Sub-Item Hashing Granularity (Design Proposal) - -> ## ⚠️ Status -> -> **Approved and implemented** — `7abb5b6` (mechanism), `e25b457` (wiring), `20ac438` (the -> SLA blocker). Written 2026-08-03 at `7dfc4f1`; every number below is measured against -> `dist/` at that commit. -> -> Precondition **(b)** was chosen (§8) and ships as the docstring guard in -> `selectElisionRegions`. **(a) — making `R_struct` do work for code — remains open**, and -> the guard defends the measured case, not the class. -> -> **Two corrections to this document, both found during implementation. Read them before -> citing §4 or §11.** -> -> 1. **§4's "depth-2 operating point" is wrong**, and it was this document's own -> recommendation. It was derived from measuring a single class-shaped file. Literal depth -> counting misses top-level function bodies entirely and is arbitrary wherever nesting -> differs. What shipped is **function-body selection at any depth**, which is what -> "depth-2" was standing in for. Measured over six real sources: 57.38% mean reduction -> against depth-2's 37.95% on the same usable set. DECISIONS.md §20. -> 2. **§11's open question 2 is therefore answered differently than asked.** The choice was -> "fixed vs budget-driven"; fixed was chosen, but the fixed rule is structural rather than -> a depth number. -> -> Real end-to-end result, 52 source files through the CLI: **22 reduce with no fallback, -> mean 52.99%**, byte-identical across fresh processes. The bundled bench corpus stays at -> 0.00% on purpose — see DECISIONS.md §20. - ---- - -## 1. The question - -Phase 1d established that `compression:token-hashing` can never succeed on a single-item -code bundle: it replaces an item's entire content, so every symbol dies at once, `R_AST` is -a boolean, and `S_k` is pinned at the formula constant `0.60` -(`docs/phase-1d-drift-investigation.md` §4, §6). - -The proposed remedy is **sub-item granularity**: elide *regions* within an item instead of -the whole item, so surviving regions keep their symbols and `R_AST` becomes fractional. - -The alternative — marking a hashed placeholder `recoverable: true` so `DriftTracker` exempts -it — is rejected up front and is not evaluated here. `DECISIONS.md` §16 and the -corresponding `NOTES-FOR-DOCS.md` entry established that `recoverable` is a claim about -**this** payload, verifiable only when an intact copy survives in it. Token-hashing has no -such copy. Asserting recoverability because rehydration machinery exists somewhere is the -error that produced the inflated 98.59% figure earlier in this phase. - -## 2. Answer - -**Sub-item granularity works, and the measurements are stronger than expected.** It also -changes the stage's failure mode from safe to unsafe, which is why §8 attaches a -precondition rather than a green light. - -Three properties hold simultaneously, measured, not argued: - -| Property | Result | -|---|---| -| Byte-identical round trip through the **existing** recovery valve | **12 / 12** | -| AST-valid after elision (of items that had an eligible region) | **8 / 8** | -| Drift gate discriminates instead of saturating | see §4 — it grades cleanly across depths | - -And the failure mode it introduces: - -> `HumanEval/0`: **55.66% reduction, `S_k = 0.0000`, AST-valid, round-trips exactly.** -> The elided region is the function's docstring — the entire specification of what the -> model is being asked to write. Every gate is green. The output is worthless. - -## 3. The gate, stated exactly - -For code, `R_struct` is pinned at `1.0` (`DECISIONS.md` §18 — the only marker is -`filepath:`, derived from `item.path`, which elision never touches). Substituting into - -``` -S_k = 1 − (0.60 · R_AST + 0.40 · R_struct) -``` - -gives, for code specifically: - -``` -S_k ≤ 0.40 ⟺ R_AST ≥ 1/3 -``` - -**A sub-item elision passes iff it retains more than a third of the bundle's symbols.** -That is the design's quantitative target, and it is the first time this stage has had one. - -## 4. Measured: the metric grades correctly once it has granularity - -Real 5,025-byte TypeScript file (`src/core/hashing/token-hasher.ts`), varying only which -nesting depth of brace-interior is elided: - -| Strategy | regions | bytes | reduction | AST valid | `R_AST` | `S_k` | drift gate | -|---|---|---|---|---|---|---|---| -| depth-1 — class bodies | 2 | 5025 → 673 | 87.79% | true | 0.2632 | 0.4421 | **falls back** | -| **depth-2 — method bodies** | 6 | 5025 → 2994 | **43.52%** | true | 0.7895 | 0.1263 | **passes** | -| depth-3 — inner blocks | 4 | 5025 → 4350 | 15.45% | true | 1.0000 | 0.0000 | passes | - -This is the result §6 of the investigation predicted. The metric is not broken; it was being -handed a boolean. Given fractional input it moves smoothly and the `0.40` threshold -discriminates sensibly — depth-1 destroys the method signatures along with the bodies and is -correctly refused; depth-2 keeps them and is correctly allowed. - -**Depth-2 — signature-preserving body elision — is the proposed operating point.** - -Across the full corpus at that granularity (Python by indented block, TS/JS by brace -interior): - -| Fixture | regions | bytes | reduction | AST valid | `R_AST` | `S_k` | drift FB | round trip | -|---|---|---|---|---|---|---|---|---| -| `HumanEval/0` | 1 | 348 → 179 | 55.66% | true | 1.0000 | 0.0000 | no | **yes** | -| `HumanEval/1` | 1 | 504 → 166 | 69.86% | true | 0.6667 | 0.2000 | no | **yes** | -| `HumanEval/2` | 1 | 328 → 126 | 63.74% | true | 1.0000 | 0.0000 | no | **yes** | -| `HumanEval/3` | 1 | 446 → 154 | 67.46% | true | 0.6667 | 0.2000 | no | **yes** | -| `HumanEval/4` | 1 | 386 → 167 | 59.63% | true | 1.0000 | 0.0000 | no | **yes** | -| `CodeXGLUE/py/101` | 1 | 192 → 154 | 28.30% | true | 0.7500 | 0.1500 | no | **yes** | -| `CodeXGLUE/py/102` | 0 | — | 0.00% | true | 1.0000 | 0.0000 | no | yes | -| `CodeXGLUE/ts/201` | 0 | — | 0.00% | **false¹** | 1.0000 | 0.0000 | no | yes | -| `CodeXGLUE/ts/202` | 0 | — | 0.00% | **false¹** | 1.0000 | 0.0000 | no | yes | -| `CodeXGLUE/js/301` | 0 | — | 0.00% | **false¹** | 1.0000 | 0.0000 | no | yes | -| `test_data/codebase.py` | 7 | 16937 → 1341 | 93.26% | true | 0.6667 | 0.2000 | no | **yes** | - -¹ **Invalid on input, before any transform.** These three are truncated CodeXGLUE -completion prompts — they end at an open brace by design. See §7. - -Read this table with §2's warning in force. The reduction figures are real bytes, and -several of them are deleting the wrong bytes. - -## 5. The design - -### 5.1 A new chokepoint, beside the existing one - -`core/elision.elideItem` replaces an item's **entire** content and cannot express a partial -elision. Sub-item elision needs a sibling in the same module — nothing may bypass the -chokepoint, which is the lesson Issue 2 paid for: - -```ts -export interface ElisionRegion { - readonly start: number; // inclusive index into item.content - readonly end: number; // exclusive -} - -export function elideRegions(params: { - readonly item: ContextItem; - readonly regions: ReadonlyArray; // disjoint, ascending - readonly markerFor: (regionText: string) => string; - readonly metadata: Readonly>; -}): ElisionOutcome; -``` - -It keeps `elideItem`'s contract: render, check savings, validate the candidate with the very -validator `selectValidator` picks, and on refusal **skip and let the stage continue**. - -### 5.2 Two hard rules, both discovered by measurement - -**Rule A — the hashed region must be exactly the bytes replaced.** -`TokenHasher.rehydrateText` substitutes the placeholder in place. Any whitespace the encoder -adds around it survives rehydration and the round trip is no longer byte-identical. A first -prototype emitted `indent + placeholder` and scored **0 / 7** on round-trip; removing the -added indent scored **7 / 7**. - -**Rule B — the placeholder must occupy a syntactically valid position.** -Rule A alone puts a Python placeholder at column 0, which the `PythonValidator` rejects -(`AST_INDENTATION_ERROR`). Applying Rule A without Rule B took AST validity from 8/8 to -**0/8**. - -A and B conflict unless the region boundary is chosen to satisfy both: **exclude the leading -indentation from the region.** The indent stays in the surrounding text, the placeholder -inherits its column, and the hashed bytes are exactly what was removed. With that boundary -both hold at once — 12/12 round trip and 8/8 AST valid. This is the single most important -implementation constraint in the design and it is invisible from the type signatures. - -### 5.3 Region selection - -Deterministic, derived from a single left-to-right scan of `item.content`. No parser is -introduced; the scanners already exist. - -- **TypeScript / JavaScript** — reuse the `TypeScriptValidator` bracket/quote/comment state - machine to find brace spans, and take **interiors at nesting depth 2**. Braces stay - outside the region, so balance is preserved by construction. -- **Python** — reuse the `PythonValidator` indent stack to find the block under a - `def`/`class` header, and take that block **minus the first line's indentation**. -- **Everything else** — no regions. Prose, logs, markdown and JSON are out of scope (§9). - -Both scanners are already written and already trusted for validation; this reads their -spans instead of only their verdicts. - -### 5.4 Eligibility - -The placeholder is exactly **77 bytes** (``). A region must -exceed that to save anything at all, and comfortably exceed it to be worth the risk. -Proposed floor: **101 bytes** (77 + 24), used for every measurement above. The existing -whole-item `minContentLength` of 40 stays for the whole-item path. - -`elideItem`'s savings check applies per region: if the rendered replacement is not smaller -than the region, skip that region — not the item. - -### 5.5 Determinism - -Region boundaries are a pure function of `item.content` and the language. Same input, same -regions, same bytes out. Invariant 1 holds. Verified across repeated runs of every probe. - -## 6. Interaction with the recovery valve — required accounting - -**The valve already handles this and needs no change.** `TokenHasher.rehydrateText` uses a -global regex replace, so N placeholders inside one item all resolve; measured byte-identical -on 12/12 samples including a 7-region file. `attemptAutomatedRehydration` triggers on -`content.includes('>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n False\n …" -``` - -55.66% reduction. `R_AST = 1.0000`. `S_k = 0.0000`. AST-valid. Round-trips exactly. **The -docstring is the task.** Drift scores it perfect because docstrings contain no symbols, and -`R_struct` — the half of the metric that is supposed to notice structural loss — is a -constant for code and notices nothing. - -This is the same shape as every finding in this phase: a check that passes because it never -looked. The difference is that here it would be *shipped as a feature*. - -**Precondition for approval.** One of these must land with the granularity change, not -after it: - -- **(a) Make `R_struct` do work for code.** The real fix, and already an open item in - `DECISIONS.md` §18: teach `extractMarkers` content-derived structural markers — comment and - docstring blocks, nesting depth, function and class boundaries. Then deleting a docstring - costs retention instead of being invisible. -- **(b) Refuse comment-and-docstring-only regions in the selector.** Cheap, targeted, and - ships in a day. It defends against the case measured here, **not against the class** — any - other high-information symbol-free content (a SQL literal, a config block, a worked - example) remains invisible to the metric. - -Recommendation: **(b) to unblock, (a) to actually close it**, with (a) tracked rather than -assumed. Shipping (b) alone and calling the problem solved would repeat the pattern this -phase exists to break. - -## 9. Explicitly out of scope - -- **JSON.** Sub-item JSON elision is **broken on the reverse path** and must not be - attempted in a first cut. `rehydrateText` checks `unwrapElisionContent` first, which only - fires when the *whole* item is a wrapped marker. A wrapper nested inside a larger document - falls through to the regex path and the raw content is substituted *inside* the wrapper: - - ``` - input : {"id":1,"payload":{"rows":[1,2,3],"note":"xxx…"},"tail":true} - output: {"id":1,"payload":{"__td_block__":"{"rows":[1,2,3],"note":"xxx… - ``` - - Not byte-identical, and not valid JSON. The format and its inverse are one contract; the - wrapper form is not composable at sub-item granularity. Whole-item JSON elision is - unaffected and keeps working. -- **The Gateway.** Invariant 8 stands. `token-hashing` still does not run there. -- **Prose, logs, markdown.** No region selector, no change. -- **Prompt-cache alignment.** Flagged, not fixed: `runTokenHashingStage` **never consults - cache pinning** — its only nod to invariant 6 is a comment about `role === 'system'`. - Sub-item elision mutates bytes in the *middle* of an item, which is worse for a cached - prefix than dropping a whole late item. This is a pre-existing gap that this change makes - material. It needs a decision before the stage is enabled anywhere prefix caching matters. - -## 10. Reproduction - -Four scratchpad probes against `dist/` at `7dfc4f1`; none are repo code. - -1. **Validator behaviour under sub-item placeholders** — TS/Python/JSON validators against - placeholders in balanced, unbalanced, in-string and mis-indented positions. -2. **Round-trip** — `TokenHasher.rehydrateText` over content with 1, 2 and 7 placeholders. -3. **Yield** — signature-preserving elision over the ten bench fixtures plus - `test_data/codebase.py` and a real 5 KB TS source file, scored with the real - `DriftTracker`, the real validators and the unified `estimateTokens`. -4. **Depth sweep** — the §4 table. - -Notable results the probes corrected mid-flight, recorded because the first answer was -wrong in each case: a "region straddling braces" test that removed one `{` and one `}` and -so proved nothing; a round-trip that failed 7/7 on added indentation; and an AST collapse -from fixing that round trip naively. - -## 11. Open questions for approval - -1. **Precondition (a) or (b)?** §8. Recommendation: (b) now, (a) tracked as the real fix. -2. **Depth-2 as the fixed operating point, or budget-driven?** The depth sweep suggests the - planner could choose depth from `targetReductionRatio` and back off on a drift failure. - That is more capable and less predictable. Recommendation: fix it at depth-2 for the - first cut; determinism is the product. -3. **Does the recovery valve get region-level partial un-hashing now, or later?** §6. - Recommendation: later — it is the Phase 1c bridge, not part of this change. -4. **What is the answer for AST-invalid input?** §7. Three of ten bundled fixtures, and the - product's own input shape. This blocks Phase 1c more than it blocks 1d, but it should be - decided rather than inherited. -5. **Cache pinning.** §9. Who owns the decision, and does it gate enabling the stage? diff --git a/docs/phase-1d-semantic-gate-disposition.md b/docs/phase-1d-semantic-gate-disposition.md deleted file mode 100644 index a6185ef..0000000 --- a/docs/phase-1d-semantic-gate-disposition.md +++ /dev/null @@ -1,313 +0,0 @@ -# Phase 1d — The Semantic Gate: Disposition of Precondition (a) - -> **Status:** investigation complete, **nothing implemented**. This document disposes of -> `docs/phase-1d-granularity-design.md` §8 precondition **(a)** — "make `R_struct` do work for -> code" — and of the same open item recorded as `DECISIONS.md` §18. -> -> **Recommendation in one line:** (a) as written is the wrong shape and the measurement says -> so; ship two narrow changes (a coverage fix and a *trim*, not a refusal), close Phase 1 with -> the hole documented, and take the real semantic term as its own phase — knowing that no -> zero-dependency term measures meaning. -> -> Measured 2026-08-04 against `dist/` at `a12e411`. - ---- - -## 1. Method, and why it is stated first - -This repository is its own corpus, and a previous session read its own edits as behavioural -movement. So: - -- **Corpus A (this repo), frozen.** 68 files — 64 `src/**/*.ts` plus the 4 tracked `*.py` — - copied to a scratch directory at commit `a12e411`, with a `sha256sum` manifest - (`e1d1b4e80b0bee23…`). Every run below reads the copy, never the working tree. -- **Corpus B (external), frozen.** 39 Python files from `pip` 25.0.1 in `.venv` - (`pip/_internal` and `pip/_vendor`, 2–40 KB, sorted, every 7th). Corpus A is ~94% - TypeScript, which is not a neutral sample for anything that depends on where a language puts - its documentation. Corpus B is real, heavily documented Python written by people who never - heard of this project. -- **Only the engine varies.** Counterfactuals are applied by patching the **compiled** - `dist/src/core/elision/regions.js` and restored afterwards by checksum - (`7c792196462ed86d…` before and after). The input bytes are identical across all runs. -- **Every figure is from the real CLI**, `optimize --target-reduction-ratio 0.3`, one - process per file. -- **Regions are recovered from the emitted output, not from an instrumented build.** The - marker carries its own byte count and hash, so the reconstruction is verified: for every one - of the 105 (A) and 124 (B) markers, the recovered span re-hashes to the digest in the marker - and the surrounding text matches the original byte-for-byte. **229/229 verified.** A silent - mis-alignment cannot pass. -- **Determinism re-checked.** The Corpus A sweep, re-run in fresh processes, produces a - byte-identical result file. - -Cross-check on the harness itself: an in-process run through `core/engine.optimize()` -reproduces the CLI's fallback set exactly (37/68 on A), which is how the fallback *reasons* -below were obtained — the CLI trace does not carry them. - -## 2. How widespread the docstring case is - -**Baseline, Corpus A:** 29 files reduce with no fallback, mean token reduction **48.62%** -(mean byte reduction 45.79%). 105 markers: **100 function-body regions + 5 whole-item -elisions**. 37 files fall back. - -**Baseline, Corpus B:** 20 files reduce, mean **39.69%**, 124 markers (123 regions + 1 -whole-item). 19 fall back. - -Regions whose removed span **begins with** a docstring or comment block: - -| | Corpus A (repo, 94% TS) | Corpus B (pip, Python) | -|---|---|---| -| regions leading with a doc block | **14 / 105 (13%)** | **75 / 124 (60.5%)** | -| — Python docstrings | 13 | 75 | -| — TS leading comments | 1 | 0 | -| reducing files affected | **2 / 29 (7%)** | **19 / 20 (95%)** | -| doc-block bytes / elided bytes | 1,466 / 80,715 (1.8%) | 10,324 / 73,487 (14.0%) | - -The asymmetry is structural, not a sampling accident. TypeScript puts its JSDoc **above** the -function header, which is outside the brace span `scanBraceSpans` selects, so the doc survives -by construction. Python's docstring is the **first statement inside the body**, which is -exactly where `scanPythonDefBodies` starts the region. Corpus A understates the problem by -roughly an order of magnitude because of what language it is written in. - -### What it costs to exclude them - -Two rules were implemented in the compiled engine and measured, both paired against the same -frozen bytes: - -- **trim** — advance the region start past a leading comment/docstring run to the first - executable character, then re-apply the size and substantiveness filters. The doc stays in - the output; the body below it is still elided. -- **refuse** — drop any region that begins with a doc block, as `selectElisionRegions` - currently drops a doc-*only* region. - -| | baseline | trim | refuse | -|---|---|---|---| -| **A** mean over the 29 files that reduce in all three | 48.62% | **48.17%** (−0.45pp) | 46.92% (−1.70pp) | -| **A** total tokens emitted over all 68 files | 102,800 | 103,244 (+0.43%) | 104,693 (+1.84%) | -| **A** `codebase.py` | 34.18% | 28.30% | **4.63%** | -| **A** `token-hasher.ts` | 45.87% | 38.68% | 26.09% | -| **B** mean over the 17 files that reduce in all three | 45.51% | **38.73%** (−6.78pp) | 23.76% (−21.75pp) | -| **B** total tokens emitted over all 39 files | 120,291 | **115,093 (−4.32%)** | 125,705 (+4.50%) | -| **B** files falling back | 19 | 12 | 11 | - -Read the two "total tokens emitted" rows before the means. **On real Python, trimming -docstrings out of the elided region makes the pipeline emit 4.3% fewer tokens overall than -eliding them.** Per surviving file it saves less, but it converts **7 fallbacks into -reductions**, and a fallback emits the entire input. - -The mechanism is measurable, not inferred. Fallback causes on Corpus B: **17 of 19 are -`Imperative constraint directive dropped`**, 1 drift, 1 AST — and the dropped directives are -docstring and comment sentences: `"If in a virtualenv…"`, `"Always return False…"`, -`"The digests must be…"`, `"# Do not trust on…"`. On Corpus A: 20 constraint-directive, 14 -drift, 3 AST. **On Python it is `cleanup:constraint-preservation`, not `DriftTracker`, that is -currently doing the work of noticing that documentation was destroyed** — and it notices only -when the prose happens to be phrased as an imperative. "Returns the number of open files" -passes; "Return the number of open files" does not. - -This also means the naive comparison is a trap: `refuse` shows 28 reducing files against the -baseline's 20 and a mean of 19.33% against 39.69%, which is three different denominators. Only -the paired rows and the totals are comparable. - -## 3. Whether the cheap fix suffices - -**No. It closes one instance of three, and the two it does not close are the larger ones.** - -First, a correction to the premise this task was scoped from. `HumanEval/0` **is already -caught** — the guard shipped with the granularity change: - -``` -$ node dist/src/cli/main.js optimize humaneval0.py --target-reduction-ratio 0.3 -from typing import List - - -def has_close_elements(numbers: List[float], threshold: float) -> bool: - """ Check if in given list of numbers, any two numbers are closer to each other than - … - """ ---- tokenBefore=106 tokenAfter=106 fallbackUsed=true driftScore=0.6 - -regions selected: [] -isSubstantiveRegion(docstring-only body): false -``` - -`isSubstantiveRegion` refuses the region, the item falls through to the whole-item path, `S_k` -pins at 0.60, and the input is echoed. The 55.66%-at-`S_k`-0.0000 measurement describes the -pre-`e25b457` engine. **The specific case is closed; the class is open, and its live instances -have nothing to do with docstrings.** - -### (i) Comments that are not leading — 14 of 29 files (A), 9 of 20 (B) - -19 regions on A and 15 on B remove comment bytes while *not* beginning with a comment. A -leading-block rule protects none of them. On Corpus A these are precisely the load-bearing -ones — the fallback reasons name them: `"// Rule 1: Never hash…"`, `"// DO NOT widen th…"`, -`"* A validator neve…"`. They are caught today only when they read as imperatives, by the -constraint checker, and a fallback is all-or-nothing. - -### (ii) Whole-item elision of a file the metric cannot see — 5 of 29 reducers (A) - -``` -$ node dist/src/cli/main.js optimize corpus/src/index.ts --target-reduction-ratio 0.3 -[TokenDamper: 14 code lines elided, 420 bytes, sha256:10a4b0eb949b] ---- tokenBefore=130 tokenAfter=18 fallbackUsed=false driftScore=0 astCoverage={"checked":1,"unchecked":0} -``` - -The package's entire public API surface, deleted, **86.15% reduction at `S_k = 0.0000`**, AST -coverage reporting a clean check. This is not a docstring and no region rule touches it. - -The cause is a **defaulted metric**, and it is the ninth instance of the invariant-10 pattern -(`src/core/ledger/drift-tracker.ts`): - -```ts -let astSymbolRetentionRatio = 1.0; -if (symbolsBefore.size > 0) { … } // otherwise the 1.0 stands -``` - -`extractSymbols` matches `import … from '…'` but not `export * from '…'`, so a barrel file -yields the empty set — and an empty *before* set is scored as **perfect retention** rather -than as *nothing measured*. Five files on Corpus A (`src/index.ts`, `src/bench/index.ts`, -`src/bench/fixtures/index.ts`, `src/core/ledger/index.ts`, `src/config/index.ts`) are elided -whole at `S_k = 0.0000` for this reason. `R_struct` cannot object: its marker set on all 29 -reducing files is exactly `{filepath:…}` — §18 confirmed on this corpus. - -The same default governs non-code. `README.md` → 0 symbols, `SECURITY.md` → 0, -`sample_logs.txt` → 0 symbols **and** 1 marker (`filepath:`). So the `logs:WHOLE passed=true -S_k=0.00 saved=97.5%` result in `DECISIONS.md` §24 is a **doubly-vacuous pass** — neither term -examined anything. That does not make §24's conclusion wrong (its rejection of a static -content-type gate stands, and the bytes saved are real), but "passed at `S_k = 0.00`" carries -no safety information there and should stop being cited as if it did. Markdown is the -exception: it has real markers (README: 20 headings/fences), so `R_struct` genuinely does work -on it. - -### (iii) Function bodies whose loss is free by construction - -For every recovered region, run the real `extractSymbols` over the removed span alone: - -| | Corpus A | Corpus B (Python) | -|---|---|---| -| function-body regions | 100 | 123 | -| contributing **zero** symbols | **42 (42.0%)** | **106 (86.2%)** | -| bytes removed by those regions | 15,562 / 79,901 (19.5%) | 39,673 / 63,278 (**62.7%**) | -| median symbols per region | 1 | **0** | - -On real Python, five sixths of the regions this engine removes are **invisible to `R_AST` -before the metric is even computed**, and they account for nearly two thirds of the deleted -bytes. The median region moves the metric by nothing at all. - -The exemplar is this project's own safety guarantee: - -``` -export function resolveFallback(request, validation, currentBundle): FallbackOutcome {[TokenDamper: 14 function-body lines elided, 263 bytes, sha256:98777770f6bd]} ---- tokenBefore=211 tokenAfter=150 fallbackUsed=false driftScore=0 -``` - -The body deleted is the `shouldFallback` branch — the implementation of invariant 3. No -docstring, no comment, `S_k = 0.0000`. `R_AST` scores it perfect because the function *name* -survived, which is precisely what the region selector was designed to guarantee. - -**That is the structural statement of the problem.** The selector keeps signatures *so that -the gate will pass*. The gate scores signatures. A rule engineered to preserve exactly what -the check measures cannot be checked by it. Excluding docstrings narrows the selector; it does -not restore the check. - -## 4. Whether a real semantic term is warranted — and what it could measure - -### §18's proposed markers are, measured, the wrong shape - -§18 names the remedy as teaching `extractMarkers` "nesting depth, function and class -boundaries, import blocks, brace balance". Each was computed on the original and on the **real -CLI output** of every reducing file: - -| proposed marker | differs before/after (A) | differs before/after (B) | -|---|---|---| -| brace balance | 1 / 29 | 0 / 20 | -| paren balance | 0 / 29 | 1 / 20 | -| function headers | 0 / 29 | 2 / 20 | -| class headers | 0 / 29 | 0 / 20 | -| import lines | 0 / 29 | 2 / 20 | -| max nesting depth | 20 / 29 | 13 / 20 | -| comment starts | **14 / 29** | **14 / 20** | -| docstring delimiters | 1 / 29 | **17 / 20** | - -Four of §18's five candidates are **near-constants under the shipped selector, for the same -reason `R_struct` already is**: the selector preserves balance and signatures by construction. -Adding them would replace one decorative constant with four. Nesting depth does vary — but it -varies on *every* successful body elision, which makes it a compression detector wearing a -loss detector's name; as a retention term it would penalise the transform for having worked. - -The only markers that actually move are **comments and docstrings**. So §18's own remedy, -followed honestly to its measurement, arrives at the same content class as the "cheap fix". -(a) and (b) are not "the real fix" versus "the stopgap" — **(a), done zero-dependency, is the -scored version of (b)**: it grades partial documentation loss and it also covers the -whole-item path, where a region rule does nothing. - -**Disposition of (a): reject as worded, retain the intent.** A term named "structural -integrity" that is computed from comment density would be a lie in the metric's own -vocabulary; if this is built, it belongs beside `R_AST` as an explicit information-class -retention term, not smuggled into `R_struct`. - -### Be skeptical of the term itself - -I am not proposing an embedding or a model call, and not because of cost. `DECISIONS.md` §9 -already settled it, and it is the product's whole differentiator: an embedding makes the gate -non-deterministic across versions, unauditable in the trace, and dependent on a network or a -several-hundred-megabyte artifact in a tool whose pitch is a deterministic local proxy. A -gate that cannot explain in one line why it refused is worse than no gate. **If the honest -conclusion is that meaning is not measurable here, the honest output is a refusal to certify, -not a heuristic that scores something adjacent and calls it semantics.** - -And that is the conclusion. Every zero-dependency candidate measures a **proxy for -information class** — is this a symbol, a comment, a literal, a heading — never information -*content*. Such a metric can say "you deleted 40% of the documentation and 3 of 11 symbols". -It cannot say whether the 263 bytes removed from `resolveFallback` mattered more than the 263 -bytes of a getter. Anyone who reads the next iteration of this metric as measuring semantic -value will be making the same mistake this phase has now catalogued nine times. - -## 5. What to do, and where it belongs - -**Close Phase 1 with the hole documented and two narrow changes. Take the term itself as its -own phase, if at all.** - -Recommended, in this order, each its own commit and each reproducible from the frozen corpora: - -1. **Stop defaulting `R_AST` and `R_struct` to 1.0 when the *before* side is empty.** This is - the invariant-10 instance, it is cheap, and it is the only one of these changes that is - about correctness rather than policy. Follow §23's precedent exactly: report that nothing - was measured (`measured: false` on `DriftReport`, surfaced on the trace) rather than - silently reporting perfect retention. Whether an unmeasurable drift on a *changed* item - should refuse is a policy decision that should be made explicitly and separately — - inverting it without deciding would fall the engine back on all prose. -2. **Trim leading doc blocks out of the region; do not refuse the region.** Measured cost - −0.45pp on A and −6.78pp per surviving file on B, against **−4.3% total emitted tokens on - B** because it converts 7 fallbacks. `refuse` costs 4× more and is strictly worse on both - corpora — it takes `codebase.py` from 34.18% to 4.63%. -3. **Record the residue as a known limitation, in `DECISIONS.md`, with the §3 numbers.** After - 1 and 2, an elided function body with no local declarations still scores `S_k = 0.0000`; - 86% of Python regions are in that class. Phase 1 should close saying so. - -Not in Phase 1, and not obviously worth doing at all: a comment/docstring **retention term** -beside `R_AST`. It is the only candidate the measurement supports, it would grade what items 2 -and 3 currently veto or ignore, and it needs its own weights derived from measurement rather -than inherited — which is a phase, not a change. - -One thing that should **not** happen under any of this: tuning the `0.40` threshold. Nothing -in this investigation moves the argument in §18 that a threshold cannot recover discriminating -power from terms that do not vary. - -## 6. Reproduction - -Five scratchpad probes against `dist/` at `a12e411`; none are repo code. The frozen corpora, -the manifest, the patcher and the probes are in the session scratchpad. - -1. `sweep.js` — CLI sweep + region recovery, verified against the marker's own hash. -2. `analyze.js` — leading-doc and comment/string byte attribution per region; cross-checked - against the shipped `isSubstantiveRegion` on every region (229/229 agree). -3. `patch.js trim|refuse|restore` — the counterfactual engines, applied to compiled output and - restored by checksum. -4. `symbols.js` / inline probe — `extractSymbols` and `extractMarkers` per file and per region. -5. `structprobe.js` — §18's proposed markers computed on original vs. real CLI output. - -Corrections this investigation made to its own first answers, recorded because each was -wrong before it was right: the first region reconstruction verified against a plain -`sha256(text)` and failed all 29 files, because `hashContent` serializes before hashing; and -the first in-process reason capture passed `cliOverrides: { targetReductionRatio }` instead of -`{ budget: { targetReductionRatio } }`, silently planned `pass_through`, and reported 7 -fallbacks where the CLI sees 37 — the documented no-budget trap, arrived at from inside. diff --git a/docs/phase-4b-lever-disposition.md b/docs/phase-4b-lever-disposition.md deleted file mode 100644 index 2694456..0000000 --- a/docs/phase-4b-lever-disposition.md +++ /dev/null @@ -1,287 +0,0 @@ -# Phase 4b — Disposition of the Three Levers Against DECISIONS §32 - -> **Date:** 2026-08-06 · **Status: measured; no remedy implemented or proposed.** (§0's -> finding — that the pin was nominal — *was* acted on; the defect itself remains open.) -> Engine: `dist` built from `73177bd`. Corpus frozen under `sha256` manifests: 144 files — -> 45 `pip` Python, 64 repo TypeScript, 9 shell scripts, 25 repo markdown, 1 log. Tokens are -> `cl100k_base` where a corpus aggregate is quoted and the **engine estimator** where a CLI -> trace is quoted verbatim; each figure says which. -> -> **Question:** §32 defers the hash-commented-code defect into drift/§28 on the grounds that -> the only identified fix is a `looksLikeMarkdown` change with blast radius over every prose -> item. Is that the only fix? -> -> **Answer: two of the three levers are dead, and the third terminates in the same deferral.** -> §32 stands, with better evidence than it had. Its claim that a classifier change is the *only* -> identified fix is now wrong in the letter — three are identified — and right in the substance. - ---- - -## 0. Reconciliation first: what the 4b.3 characterization test actually does - -The `73177bd` commit message says the `KNOWN DEFECT` block exists "so whoever fixes it removes -a failing assertion deliberately". **That description is wrong, and the discrepancy the -question points at is real.** - -`test/unit/markdown-marker-allowlist.test.ts` contains no `.skip`, no `.fails`, no `.todo`. Both -`KNOWN DEFECT` tests **pass**, and are counted in the 471. They assert current behaviour: - -```ts -expect(result.trace.astCoverage).toEqual({ checked: 0, unchecked: 1, uncheckedContentTypes: ['markdown'] }); -expect(result.validation.driftCoverage?.structMeasured).toBe(true); -expect(result.validation.driftCoverage?.measured).toBe(true); -expect(result.validation.driftCoverage?.unwitnessedItems).toEqual([]); -``` - -So: **a passing characterization test.** The mechanism I described is real but inverted — the -assertions do not fail now, they will *begin* failing the moment someone fixes the defect, which -is what forces the encounter. - -**Does that make the defect the de facto spec? Yes.** A green assertion is a specification; the -only thing marking these as not-a-spec is the `describe` name and a docblock, which are prose -and enforce nothing. A tool reading the suite — or a person skimming green output — sees -`structMeasured: true` on a 99%-deleted shell script asserted as correct. - -**Fixed after this note was first written.** The block now states the *contract* and carries -`it.fails`, which in this vitest inverts the verdict exactly as needed: green while the body -throws, red with `Expect test to fail` once it stops. Verified in both directions — green today, -and red under a simulated remedy (the allowlist temporarily emptied so `measured` goes false), -then reverted. - -My earlier dismissal of `it.fails` here — "it inverts to 'the body must throw', which these -bodies do not" — was wrong because it assumed the body keeps asserting the *defect*. Stating -the **contract** instead makes the inversion exact. - -Three guards, because `it.fails` is satisfied by any throw and would otherwise be invariant -10's shape inside the mechanism meant to fix invariant 10's shape: the pipeline result is -computed at describe scope so a crash is a collection error rather than a swallowed pass; the -preconditions live in a separate ordinary passing test that goes red if the fixture drifts; and -the `it.fails` body holds exactly one assertion. - -The contract is remedy-agnostic and stated over the *input*, not over trace fields — §2 of this -note showed `tclConfig.sh` and `CODE_OF_CONDUCT.md` are identical on every field the trace -carries, so a field-level contract would condemn real prose too. Any of the four available -remedies satisfies it: stop deleting the file, stop claiming measurement, stop classifying it -as markdown, or stop harvesting its comments. - ---- - -## 1. Lever 1 — prohibit whole-item elision when `astCoverage.checked == 0` - -### What it would change - -Computed against per-item traces, not simulated by patching: an item with `checked == 0` that -currently reduces would return its input. - -| corpus | n | reduce | `checked==0` | **lose** | tok saved now | after lever | -|---|---|---|---|---|---|---| -| pip | 45 | 19 | 6 | 1 | 8,832 | 8,815 | -| ts | 64 | 2 | 64 | 2 | 78 | 0 | -| shell | 9 | 4 | 9 | **4** | 3,670 | 0 | -| prose | 25 | 2 | 25 | **2** | 785 | 0 | -| logs | 1 | 0 | 1 | 0 | 0 | 0 | -| **stdin total** | **144** | 27 | | **9** | **13,365** | **8,815** | - -**4,550 of 13,365 cl100k tokens given back — 34% of everything the pathless route currently -saves.** On the file-argument route the cost is small: 46 files reduce, 2 at `checked == 0`, -785 tokens. - -### Does it kill the cases §24 established are not structurally doomed? - -**Prose: yes, both of them.** The two prose files that reduce over stdin are exactly the §24 -class, and the lever stops both (engine estimator, as the CLI reports it): - -``` -CODE_OF_CONDUCT.md 910 -> 19 tokens (97.9%) drift 0.400 fallbackUsed false -SECURITY.md 305 -> 19 tokens (93.8%) drift 0.400 fallbackUsed false -``` - -**Logs: the case does not arise on this corpus, and I am not claiming the lever kills it.** -`sample_logs.txt` reduces 0% on *both* routes, falling back on constraint-preservation — -`Imperative constraint directive dropped: "999Z [CRITICAL] com."`, which is Issue 4's planted -directive doing its job. §24's `logs:WHOLE saved=97.5%` was measured on a payload this corpus -does not contain. Unmeasured here, stated as unmeasured. - -### The Gateway, which is where the lever actually dies - -Invariant 8: `cleanup:session-dedup` is the only stage the proxy runs. Measured on the -cross-turn scenario (not `session.json`, whose turn 2 falls back for unrelated reasons and -would have hidden this): - -``` -turn 2 elided the repeated context: true - elided to: "[TokenDamper Elided: ref=82744641a800 bytes=349 kind=conversation]" - msg[0] contentType=text validator=NONE (checked=0) - msg[1] contentType=text validator=NONE (checked=0) - msg[2] contentType=text validator=NONE (checked=0) -turns: [{raw:87, opt:87, saved:0, fb:false}, {raw:101, opt:32, saved:69, fb:false}] -``` - -**Every dedup elision the Gateway performs is a `checked == 0` whole-item elision.** The lever -does not cost the Gateway some reduction; it makes the Gateway a pass-through. Conversational -messages have no validator and never will — that is what §17 settled. - -### Why it cannot be narrowed - -The two items the lever must tell apart are **identical on every field it could key on**: - -| trace field | `tclConfig.sh` (must stop) | `CODE_OF_CONDUCT.md` (must not) | -|---|---|---| -| `contentType` | markdown | markdown | -| `astCoverage` | `checked 0, unchecked 1` | `checked 0, unchecked 1` | -| `driftScore` | 0.400 | 0.400 | -| `structMeasured` / `astMeasured` / `measured` | true / false / true | true / false / true | -| `symbolsBefore` | 0 | 0 | -| `unwitnessedItems` | `[]` | `[]` | -| `fallbackUsed` | false | false | -| outcome (engine estimator) | 1,877 → 19 | 910 → 19 | - -The single differing field is `contentMarkersBefore`: **79 versus 12** — and it points the wrong -way. The shell script has *more* structural evidence than the real document, because its -evidence is fabricated. Any threshold on marker count protects the shell script less. - -**Verdict: dead.** Not because it costs 34% of pathless yield, but because at the moment the -gate would fire the harmed case and the intended case are the same item — coverage is precisely -what they have in common — and because it stops the Gateway product outright. - ---- - -## 2. Lever 2 — non-content discriminators (shebang, executable bit, extension) - -Question set: 9 shell scripts + the 4 `pip` files 4b.2's probe declines *and* that land in -`markdown` + 25 real documents = **38**. (Two further declined `pip` files land in `text` and are -not part of the question — they harvest nothing.) - -### Availability, before accuracy - -| discriminator | stdin | MCP | Gateway | file argument | -|---|---|---|---|---| -| path extension | **absent** | **absent** | **absent** | present — and already decides correctly | -| executable bit | **absent** (no file) | **absent** | **absent** | present in principle | -| shebang | present | present | present | present | - -Two of the three do not exist on the routes where the defect lives. That is Phase 4b's premise, -not an incidental gap. The extension is decisive on the one route where nothing is harmed: all -four destroyed shell scripts reduce **only** pathless. - -The executable bit is additionally non-informative where it can be read at all: at the source -install on NTFS, `gettext.sh` tests `-x` true and **`env.sh` — one of the four files being -destroyed — tests `-x` false.** - -### Shebang confusion matrix - -| group | want | has shebang | -|---|---|---| -| 9 shell scripts | not markdown | **5 / 9** | -| 4 undetected `pip` files | not markdown | **0 / 4** | -| 25 real documents | markdown | **0 / 25** | - -Zero false positives — no document in the negative set starts with `#!`. Recall is **5 of 13** -on the positives. And the distribution is adversarial in the way that matters: - -``` -MISS env.sh 376 -> 27 <-- deleted 99% -MISS tclConfig.sh 1898 -> 28 <-- deleted 99% -MISS tkConfig.sh 1052 -> 29 <-- deleted 99% -MISS git-prompt.sh 6454 -> 6454 -``` - -**Of the four files actually being destroyed, the shebang catches one** (`vimspell.sh`). The -three it misses are sourced-not-executed shell — `env.sh`, `tclConfig.sh`, `tkConfig.sh` are -`.`-included config fragments, which is exactly why they carry no shebang, and exactly the -shape that survives whole-item elision at `S_k = 0.400`. - -**Verdict: dead.** Perfect precision, useless recall on the harmed subset, and its two stronger -companions are unavailable by construction on every route that matters. - ---- - -## 3. Lever 3 — the exact-0.400 cluster - -### The population - -16 item/route pairs land on exactly `0.400`, in two structurally opposite configurations: - -| attractor | algebra | n | what it means | -|---|---|---|---| -| **A** | `R_AST = 1` (empty-set default), `R_struct = 0` | **14** | nothing was measured on one side; everything was destroyed on the other | -| **B** | `R_AST = 1/3`, `R_struct = 1` | **2** | exactly one third of symbols retained, structure intact — the code gate's own boundary (§18) | - -`S_k = 1 − (0.6·R_AST + 0.4·R_struct)` gives `0.400` for both. One scalar, two opposite -situations, and `>` versus `>=` decides them together. - -Not all 16 are decided by the boundary: `ARCHITECTURE.md` and `bench_evaluator.ts` sit at -`0.400` and fall back on **constraint-preservation**, not drift. The boundary actually decides -**7**. - -### What flipping the inequality would do (computed, not implemented) - -7 currently-passing items would fall back, giving back **5,080 cl100k tokens**: - -``` -shell env.sh stdin 376 -> 27 -shell tclConfig.sh stdin 1898 -> 28 -shell tkConfig.sh stdin 1052 -> 29 -shell vimspell.sh stdin 457 -> 29 -prose CODE_OF_CONDUCT.md stdin 621 -> 28 -prose SECURITY.md stdin 220 -> 28 -ts stages_cleanup_constraint-preservation.ts filearg 851 -> 226 -``` - -The last row is the cost: a **legitimate sub-item code elision on the file-argument route**, -at `R_AST` exactly `1/3`, killed to stop four shell scripts on a route it has nothing to do -with. Flipping is not free and is not proposed. - -### What an explicit code gate would have to state - -Four things, and the fourth is why the inequality is currently load-bearing: - -1. **That the comparison applies only to a measured quantity.** Attractor A reaches the - threshold carrying `R_AST = 1` as an *empty-set default*, not a measurement. §28 established - this for validator-covered items and stopped there; every one of the 14 is outside that - scope. -2. **That for code `R_struct` is pinned at `1.0`** (§18 — the only marker is `filepath:`, which - elision cannot destroy), so for code the gate reduces exactly to `R_AST ≥ 1/3`. -3. **That "one third" is a chosen retention policy, not an artifact.** It is currently the - arithmetic consequence of `w_AST = 0.6`, `w_struct = 0.4` and `threshold = 0.40`, none of - which was picked to mean "keep a third of the symbols". Whether retaining exactly one third - is acceptable is a real question that `>` answers "yes" by direction rather than by decision. -4. **That the two attractors need different verdicts** — which a single scalar cannot express. - An explicit gate is therefore *two* gates: a measurement gate and a retention gate. `>` vs - `>=` stops being a decision only once they are separated. - -### What it would change on the corpus - -**As a restatement, nothing — inert.** Points 2–4 are documentation of behaviour that already -holds. Point 1 is the only one with teeth, and enforcing it removes the 14 attractor-A items -from the comparison entirely, routing them to the unmeasurable branch. That is §28's rule -extended past validator-covered items — **the deferred question, reached from the arithmetic -instead of from the classifier.** - -**Verdict: not dead, and the only lever that survives.** But its terminus is the deferral, not -an alternative to it. - ---- - -## 4. Findings - -1. **§32 stands.** Two of three levers are dead on measurement — lever 1 because the harmed and - intended cases are trace-identical at the point of decision *and* it stops the Gateway - product; lever 2 because it misses three of the four destroyed files and its stronger - companions do not exist on pathless routes. The third confirms the deferral rather than - avoiding it. -2. **§32's wording needs one correction.** It says the only identified fix is a - `looksLikeMarkdown` change. Three fixes are identified. Two are worse than the deferral and - one *is* the deferral, so the conclusion holds — but "only identified fix" should read "only - fix that does not either destroy the Gateway or reduce to §28's open question". -3. **The 4b.3 stdin denominator was 132 and should be 144.** The A/B loop globbed - `corpus-prose/*` at top level, covering 13 of 25 markdown files. Re-run over all 25: - **0 changed.** The inert conclusion is unaffected; the number was understated. -4. **The characterization test was a passing spec, not a pending one** (§0). **Now fixed**: - inverted with `it.fails` so the suite records the violation instead of blessing it, verified - red under a simulated remedy. -5. **A new datum for whoever takes up §28's question.** The prose casualties of lever 1 — - `CODE_OF_CONDUCT.md` at 97.9% and `SECURITY.md` at 93.8%, both at `S_k = 0.400`, both - `fallbackUsed: false` — are not a separate population from the shell scripts. They are the - same trace. Any rule that saves `tclConfig.sh` and spares them has to distinguish them by - something no current field carries. diff --git a/docs/phase-4b-pathless-code-scope.md b/docs/phase-4b-pathless-code-scope.md deleted file mode 100644 index e921694..0000000 --- a/docs/phase-4b-pathless-code-scope.md +++ /dev/null @@ -1,546 +0,0 @@ -# Phase 4b — Pathless Code Is Invisible to the Validator Layer (Scope) - -> **Label:** this document was filed as `phase-1e-pathless-code-scope.md` while its contents -> called the work "4b". One label, and it is **4b** — filename, heading and steps now agree. -> Renamed 2026-08-04, before anything else came to reference the old name. -> -> **Status: 4b.0 landed 2026-08-04 (harness only). 4b.1 landed 2026-08-05 — see §8. 4b.2 -> landed 2026-08-06 — see §9, and note that its §6 risk 2 was disposed of by a mechanism this -> document does not contain. 4b.3 landed 2026-08-06 — see §10, and read it before quoting §5's -> description of it: the fabrication it names is not in the buckets it names. All four steps -> are now implemented.** This document exists to be argued with before any code is written. Measured 2026-08-04 against `dist/` at `04ce2a3`, -> over corpora frozen in a scratch directory (`CLAUDE.md`, Gotchas — freeze before measuring). -> -> **Headline:** the obvious narrow fix — resolve a language inside `selectValidator` and leave -> classification alone — is the wrong one, and the measurement says so. It fixes the yield -> defect and makes a second, currently-latent defect material. - ---- - -## 1. Reproduction - -Same bytes, two entry forms, current build: - -``` -$ node dist/src/cli/main.js optimize corpus/…/codebase.py --target-reduction-ratio 0.3 - output 11,328 bytes tokenBefore 5029 tokenAfter 3310 fallback false drift 0 - astCoverage {"checked":1,"unchecked":0,"uncheckedContentTypes":[]} - -$ node dist/src/cli/main.js optimize - --target-reduction-ratio 0.3 < corpus/…/codebase.py - output 16,937 bytes tokenBefore 5029 tokenAfter 5029 fallback true drift 0.6 - astCoverage {"checked":0,"unchecked":1,"uncheckedContentTypes":["text"]} - contentTypeCounts {"text":1, …} -``` - -`createContextBundle` derives `contentType` from content **and** `sourcePath`; stdin has no -path. With `contentType: 'text'`, `selectValidator` returns `null`, `selectElisionRegions` -returns nothing (it asks the same validator for a language), and the item falls to whole-item -hashing, where `S_k` pins at the formula constant `0.60` and the pipeline falls back. - -§23's coverage reporting is working: `astCoverage` says plainly that nothing looked. The hole -is **visible**; it is not closed. - -### Yield, both frozen corpora - -| | file argument | stdin | -|---|---|---| -| pip corpus (39 Python files) — files reducing | 20 | **1** | -| pip corpus — aggregate tokens saved | 14.11% | **2.42%** | -| repo corpus (64 TS + 4 py) — files reducing | 30 | **5** | -| repo corpus — aggregate tokens saved | 19.31% | **0.13%** | - -Two of the three entry modes are pathless by construction. `optimize -` has no path; the MCP -`optimize_context` schema accepts `rawInput` plus budget knobs and **no path or language -field**; Gateway messages are provider payloads and have neither. The CLI file argument is the -only route that works, and it is the one route a coding assistant does not use. - -> **Correction, 2026-08-06 (`docs/phase-0-measurement-baseline.md` §4).** "The CLI file argument -> is the only route that works" is true for Python and false in general. The file route works -> only for the **19 extensions in `isCodeExtension`**. `.pl` and `.tcl` are not among them, so -> Perl and Tcl classify `markdown` on the file route exactly as they do over stdin, and a -> 57,037-token Perl file passed **by name** is deleted whole at 100% with -> `astCoverage.checked: 0` and `fallbackUsed: false`. Read "pathless" in this document as -> "outside `isCodeExtension`" wherever it is used as the boundary of the defect. - -## 2. There are two defects here, not one - -### D1 — no validator, no regions (the known one) - -Described above. Costs yield, fails closed, visible on the trace. - -### D2 — a positive misclassification that fabricates drift markers (not previously recorded) - -Pathless content does not merely lack a type; it is assigned a wrong one with confidence: - -``` -pathless classification of the frozen corpora - pip python → text 20, markdown 19 - repo ts → text 63, markdown 4, html 1 -``` - -Python files classify as **markdown** because `looksLikeMarkdown` tests -`/(^|\n)#{1,6}\s+\S/`, and `# NOTE: ...` is a Python comment. And -`DriftTracker.extractMarkers` harvests markdown headings for every type on the -`MARKDOWN_MARKER_TYPES` allowlist — which contains `markdown` **and `text`**: - -``` -pathless classification → fabricated markdown headings - py/markdown 20 files 717 headings - py/text 23 files 308 headings - ts/* 64 files 0 headings -``` - -**1,025 fabricated structural markers across 43 Python files**, e.g. -`heading:# NOTE: Maybe use the optionx attribute to normalize keynames.` This is the defect -`57950bf` fixed ("stop reading Python comments as markdown headings") arriving by a different -route: that fix gated harvesting on `contentType`, and misclassification hands it a -`contentType` on the allowlist. - -Worth noting for whoever touches that allowlist next: its docblock says a new `ContentType` -"should default to *not* harvesting these — an invented marker actively inflates drift". Its -membership includes `text` and `unknown`, which are the two "we do not know" buckets. The -stated rule and the list disagree. - -**D2 is currently inert.** Nothing is elided on the pathless path, so before and after markers -are identical and `R_struct` stays 1.0. It becomes material the moment D1 is fixed. - -## 3. Why the narrow seam is the wrong one — measured - -The intuitive minimal change (call it **option D**) resolves a language inside -`selectValidator` when `path` and `language` are absent, leaving `classifyContent` untouched. -Blast radius looks smaller: validation and region selection change, `extractMarkers` does not. - -That is exactly the problem. Simulating option D — real CLI output with regions elided, scored -by the real `DriftTracker` with the pathless `contentType` the item would still carry — over -the 20 pip files that reduce: - -``` -drift higher under option D : 14 / 20 -newly pushed over the 0.40 gate : 1 - - pip/_internal/cli/main_parser.py text S_k 0.400 vs 0.000 markers 24 -> 0 - pip/_vendor/packaging/_elffile.py text S_k 0.400 vs 0.000 markers 3 -> 0 - pip/_vendor/pygments/…/__init__.py markdown S_k 0.514 vs 0.114 markers 4 -> 0 - pip/_internal/configuration.py markdown S_k 0.361 vs 0.015 markers 30 -> 4 -``` - -The right-hand column is the same elision scored with `contentType: 'code'`. Several files sit -at **exactly 0.400** — every fabricated heading destroyed, `R_struct = 0` — and pass only -because the gate is `> 0.40` rather than `>=`. Any real drift on top tips them over. - -So option D would deliver the yield and simultaneously convert a dormant metric defect into -spurious fallbacks on precisely the content it was enabling. **Whatever resolves the language -must also correct the content type**, and the two must move together. - -## 4. Is content-based detection viable at all? Python yes, TypeScript no - -§17 removed content-only code detection on purpose: its only signal was a markdown fence, and -the verdict flipped on apostrophe parity in surrounding prose. The question is whether a -majority-of-lines rule — the shape §22 and §27 already use for logs and YAML — has a usable -margin. Measured, the answer differs by language, and the difference is not an accident. - -**Python — separable, with a factor-of-seven margin.** Line predicates: `def`/`class` headers -ending in a colon, `from X import` / `import X` with no `from` clause, own-line decorators -(strong); block headers and statement keywords (weak); `#` lines neutral; and a disqualifying -count for shapes Python does not have (`;`/`{` line ends, `=>`, `function f(`, `const x =`). - -| set | ratio range | -|---|---| -| pip Python (positive, n=39) | 0.000 – 0.569 | -| repo Python (positive, n=4) | 0.066 – 0.313 | -| repo TypeScript (negative, n=64) | 0.000 – **0.021** | -| repo prose (negative, n=27) | 0.000 – **0.008** | - -Rule `strong ≥ 2 && ratio ≥ 0.15 && disqualified < 10%`: **38 of 43 Python files detected, 0 -false positives on 64 TypeScript sources and 27 prose documents.** The five misses are files -with almost no `def`/`class`/`import` density — `pip/_vendor/rich/_cell_widths.py` is a 452-line -list literal and scores 0.000, which is correct: it is not recognisable as Python by structure. -Misses fail to today's behaviour, which is the safe direction. - -**TypeScript — not separable on this corpus.** Using the same method, TypeScript positives span -0.283–1.000 and prose negatives reach **0.333**, with 34 strong-signal lines in -`docs/architecture/milestone_5_topology_knapsack_planner.md`. The ranges overlap; no threshold -orders them. - -The cause is worth stating because it will not go away: this repository's prose is -*documentation about TypeScript*, dense with fenced TypeScript. Nobody here writes design docs -full of Python. A whole-item classifier must choose one answer for a document that is genuinely -both — which is §17's finding, reached from the other end. **A TypeScript content probe is not -proposed, now or later, without a different kind of signal.** - -Real Gateway traffic behaves as the method predicts. Scoring the seven messages of -`test_data/session.json`: - -``` - session[0] user decisive 1 py 0.00 "Can you analyze our TokenDamper resilience pipel…" - session[2] tool decisive 5 py 1.00 "class CircuitBreaker:\n def __init__(self):…" - session[4] tool decisive 200 py 0.00 "2026-07-30T19:00:12.144Z [WARN] Stage 3 circuit…" - session[6] tool decisive 120 ts 0.67 "{\n \"stage_3\": {\n \"retry_count\": 5,…" -``` - -A five-line Python snippet is detected; prose messages score zero. Note `session[6]`: JSON -scores high on a brace-and-semicolon signal and is saved only by `looksLikeJson` running first. -Probe **order** is load-bearing, and a code probe must stay behind the JSON check. - -## 5. Proposed sequencing - -Four changes, deliberately separable, in this order. Each is independently useful and each has -its own evidence. - -**4b.0 — the benchmark harness passes a path (one line, no engine change). — LANDED -2026-08-04.** `run_benchmark.py` invoked `[cmd, "optimize", "-"]` and piped the text, which is -why it "saw no improvement from granularity". A harness defect, not an engine defect, and the -one that had been distorting published numbers. Landed alone, as proposed. - -Measured over the four bundled fixtures, engine frozen at `95056df` (all 64 `dist/**/*.js` -hashes identical across both runs), corpus frozen by `sha256` manifest. Tokens are -`cl100k_base` via the harness's own `tiktoken`: - -| fixture | before (stdin) | after (path) | outcome | -|---|---|---|---| -| `codebase.py` | 0.00% — fallback `S_k 0.60` | **27.61%** — no fallback | changed | -| `sample_logs.txt` | 0.00% — fallback, constraint directive | 0.00% — same | unchanged | -| `tool_output.json` | 0.00% — fallback `S_k 0.60` | 0.00% — same | unchanged | -| `session.json` | −1.39% — fallback `S_k 0.60` | −1.39% — same | unchanged | - -**One fixture of four moves.** That is the honest headline: this corpus is one Python file, -one log, and two JSON payloads, and only the Python file is reachable by the path route. Logs -and JSON fall back for reasons that have nothing to do with the path, and would not move under -4b.1–4b.3 either. - -Two figures in §1 and §7 of this document need reading in that light. §1 reports -`codebase.py` at 16,937 → 11,328 bytes, which the engine's own trace calls **34.18%** -(5,029 → 3,310). Scored with a real BPE tokenizer the same output is **27.61%**. The gap is -`EnhancedHeuristicTokenizer`, whose 24% mean absolute error CLAUDE.md already records; the -trace figure is a self-estimate and the `cl100k` figure is the one to publish. The §7 -projections (`2.42% → up to 14.11%`, `1 → up to 20 files`) are likewise engine-estimator -figures over the frozen pip corpus and should be re-derived against `cl100k` before they are -quoted as expected value. - -The −1.39% on `session.json` is **not** fixed by 4b.0 and is not meant to be. It is the -harness's *other* defect: `run_benchmark.py:75-77` sets `orig_tokens` from -`count_tokens(json.dumps(messages))`, a re-serialization that drops the file's pretty-printing, -while TokenDamper is handed the raw file and — correctly — echoes it back byte-identically on -fallback. The engine is behaving; the denominator is wrong. Separate concern, separate commit; -see CLAUDE.md's Issue 5 entry, which already documents it. - -**4b.1 — let the caller declare the language. — LANDED 2026-08-05, DECISIONS §29. See §8.** -`item.language` already exists on -`ContextItem`, is already **first** in `selectValidator`'s precedence, and **is never populated -by any adapter** — `constructors.ts:251` passes it through and nothing supplies it. Add the -declaration routes: a CLI `--language` / `--input-name` flag for stdin, a `path` or `language` -property on the MCP `optimize_context` schema, and an optional Gateway hint. Zero inference, -zero blast radius, and it is strictly better than a probe wherever the caller knows the answer -— which on the MCP path is always. - -**4b.2 — a Python-only content probe, setting language *and* content type together. — LANDED -2026-08-06, DECISIONS §31. See §9.** Runs -only when `path` and a declared `language` are both absent, and only behind the JSON check. -Sets `language: 'python'` **and** `contentType: 'code'` atomically. Both, because §3 shows the -language alone leaves the marker fabrication in place — and because -`CONTENT_TYPE_VALIDATORS.code` maps to the **TypeScript** validator, so a `code` tag without a -language sends Python to the wrong checker. That coupling deserves a test, not a comment. -**Superseded 2026-08-08 (Phase C): `CONTENT_TYPE_VALIDATORS.code` is now `null`.** The coupling -argument survives unchanged and the test still exists; only the failure mode moved, from *wrong* -validation to *absent* validation. See the note on item 3 of §6.3. - -**4b.3 — the `MARKDOWN_MARKER_TYPES` allowlist, separately. — LANDED 2026-08-06, DECISIONS -§32. The paragraph below is wrong about where the fabrication lives; see §10.** 4b.2 fixes the fabrication for -*detected* Python only. Undetected Python, and pathless code in any other language, still -harvest `#` and `- ` lines as structural markers from the `text` and `unknown` buckets. This is -its own decision with its own blast radius over every prose item in every bundle, and it should -not ride along. - -**Not proposed:** a TypeScript or JavaScript content probe (§4), and any change to §17's -removal of fence-based detection. - -## 6. Risks, and the measurement that must accompany 4b.2 - -1. **Turn-1 Gateway measurement is mandatory.** `validate()` runs `validateBundleAst` over - *every* item in the final bundle, so a classification change can fail an item nothing - touched. Measure turn 1 of a real session, where `cleanup:session-dedup` has no previous - hashes and cannot elide: any fallback there is a false positive by construction. That is how - §17 was found. -2. **New validation means new fallbacks are possible.** Enabling `PythonValidator` on pathless - fragments will reject some of them — the pip corpus already produces one - `Unexpected indent level 4` fallback on a *complete* file, and the Gateway carries - fragments, which are worse. Needs its own before/after count on the session corpus, and a - decision about whether a fragment that fails indentation should fall back or be reported as - uncheckable. -3. **`code` → `tsValidator` is a trap.** Any path that sets `contentType: 'code'` without a - language sends the item to the TypeScript validator. Pin it. - **Resolved 2026-08-08 (Phase C), and the pin was kept rather than deleted.** - `CONTENT_TYPE_VALIDATORS.code` is `null`: `code` is a *family*, not a language, and lexing - the family as TypeScript invented findings rather than weakening them (perl 39/40, tcl 30/40, - shell 22/40 false `AST_UNTERMINATED_STRING` / `AST_UNBALANCED_BRACKET`). A `code` tag without - a language is still not enough — it now yields `validated: false` and a row on - `trace.astCoverage` (DECISIONS §23) instead of a wrong verdict. The pin lives in - `test/unit/declared-language.test.ts` and `test/unit/bench/evaluator.test.ts`; both were - updated to assert `null` rather than `'typescript'`. -4. **User-visible marker text changes** for detected items — `[TokenDamper: N code lines - elided…]` instead of `text`/`markdown`. Correct per §24, but it is a CHANGELOG line. -5. **Corpus bias.** The negative set is 27 documents from one repository plus 7 synthetic - session messages. The zero-false-positive result is real but narrow; it should be re-run - against a wider prose corpus before the probe is trusted, and the probe should be the kind - that fails to today's behaviour rather than to a wrong answer. - -## 7. Expected value, stated honestly - -If 4b.1 and 4b.2 land, the pathless Python path should approach the file-argument path: on the -frozen pip corpus that is **1 → up to 20 files reducing** and **2.42% → up to 14.11%** aggregate -tokens saved, minus the five files the probe does not detect and minus whatever risk 2 costs. -TypeScript over stdin is **not** addressed and should not be claimed as addressed; 4b.1's -declaration route is the only thing on offer for it. - ---- - -## 8. 4b.1 as landed — what it measured, and three things this document got wrong - -**Date:** 2026-08-05. Implemented: DECISIONS §29, `test/unit/declared-language.test.ts`. -Shipped surface: CLI `--language` / `--input-name`, MCP `optimize_context.language` / -`.path`. **No Gateway hint** — see below. - -### The measurement - -Corpora re-frozen for this change (the 2026-08-04 scratch copies were gone): 64 TypeScript -sources extracted from `HEAD` with `git archive`, and 45 `pip/_internal` Python files, each -under a `sha256` manifest. Engine A/B'd as `dist-before` (built from a stashed tree at -`5b19394`) against `dist-after`. Tokens are real `cl100k_base`, not the engine's estimator. -`--target-reduction-ratio 0.3`. - -| corpus | bare stdin | `--language` | file argument | `--input-name` | -|---|---|---|---|---| -| repo TypeScript (64) | 0.07%, 2 files reduce, 57 fallbacks | **19.27%, 25 files** | 19.27% | 19.27% | -| `pip` Python (45) | 0.02%, 1 file, 43 fallbacks | **12.34%, 19 files** | 12.34% | 12.34% | - -Byte-identical to the file-argument route on **109/109 files**. AST coverage 0/109 → 109/109 -items checked. Deterministic across 6/6 fresh processes. Zero collateral: every undeclared -run is byte-identical before and after. - -### Correction 1 — §7's expected value is not comparable to what landed, in either direction - -§7 projected the pip corpus from `2.42% → up to 14.11%` and `1 → up to 20 files`. Those are -**engine-estimator** figures over a **different 39-file selection**. The landed measurement -is `0.02% → 12.34%` and `1 → 19 files` over 45 files in `cl100k`. Both differences push the -same way — a 24%-MAE estimator and a different file set — so the numbers are not evidence for -or against each other. Quote the §8 table, which states its tokenizer and its manifest. - -### Correction 2 — TypeScript over stdin *is* addressed, by the declaration route - -§7 closes: *"TypeScript over stdin is **not** addressed and should not be claimed as -addressed; 4b.1's declaration route is the only thing on offer for it."* That was written -about 4b.2's probe, which is Python-only and deliberately so. The declaration route landed -and delivers 19.27% on this repository's own sources. What remains unaddressed is -**undeclared** TypeScript over stdin, which is 4b.2's excluded scope (§4) and stays excluded. - -### Correction 3 — the risk register missed the effect that dominates - -§6 lists five risks, of which risk 2 (*"new validation means new fallbacks are possible"*) -was expected to cost a little yield through `PythonValidator` rejecting fragments. That is -not what happened. **Zero** fallbacks came from syntax. Six files across the two corpora -reduce under bare stdin and fall back once declared, and every one is `SEMANTIC_DRIFT_ -UNMEASURABLE` — five TypeScript barrels and `pip`'s `status_codes.py`. - -That is §28 reaching a route it had not reached. §28 refuses to certify an unwitnessed -elision only for items an AST validator **covers**, and nothing covers a pathless item, so -over stdin a symbol-free barrel was still being elided whole at `S_k = 0.0000` with no -fallback — the exact defect `5b19394` is recorded as closing: - -``` -index.ts, bare stdin: 135 -> 18 tokens fallbackUsed false - astCoverage {checked: 0, unchecked: 1, uncheckedContentTypes: ["text"]} - driftCoverage {symbolBearingItems: 0, unwitnessedItems: []} -index.ts, declared: refused fallbackUsed true - driftCoverage {symbolBearingItems: 1, unwitnessedItems: [7c2bce68…]} -``` - -The consequence for anyone reading the yield table: the declared route is not uniformly -additive. It gains 44 files and gives back 6, and the 6 were reducing only by being deleted -under a score that had measured nothing. It also means **a pathless item is not merely -unoptimized — it is unprotected**, which is a stronger argument for 4b.2 than the yield -figures §7 makes it on. - -### The Gateway hint, proposed in §5 and not built - -§5's 4b.1 includes "an optional Gateway hint". It is deliberately unbuilt. A provider payload -has no per-message language field, so the only shape available to a header or a config key is -a **whole-request** declaration — and §4's own scoring shows a real session is heterogeneous: -of seven `session.json` messages, one is a Python snippet, one is JSON, and the rest are -prose and logs. Declaring `python` for that request tags English as Python and hands it to -`PythonValidator`, whose indentation rule prose does not satisfy. That turns a declaration -route into a fallback generator on precisely the traffic invariant 8 exists to protect. A -per-message declaration would need a header format naming message indices; that is a design -with its own measurement, not a flag. - -### Addendum — the harness was a construction site too - -Filed after the initial 4b.1 commit, on the question "is anything left". There are exactly -three `createOptimizationRequest` call sites: CLI, MCP, and `src/bench/fixtures/loader.ts`. -The third passed `sourcePath` and dropped `fixture.language` — a **required** field on -`BenchmarkFixture` — so the benchmark harness re-derived a content type from a filename it had -sometimes synthesized itself. `codexglue.ts` writes `src/item_.txt` for a fixture with no -path, which classifies `text`, and that fixture then reached the engine with no validator and a -guaranteed fallback: `checked: 0`, 133 → 133 tokens. Declared: 133 → 59. - -This matters beyond the one fixture. §1 of this document contrasts "the file argument route" -with "the stdin route" as though the file route were sound. It is sound *when the filename -carries the answer*. The harness demonstrates the third case — a path that exists but does not -describe the content — and that case is invisible to the framing this document started with. - -Two consequences worth carrying into 4b.2: - -1. **A path is not a declaration.** `src/item_pathless-1.txt` is a real `sourcePath` and tells - the classifier something false. Any probe added in 4b.2 must not treat the presence of a - path as evidence that classification succeeded. -2. **A false declaration fails closed, and cheaply.** Believing `language` broke - `test/integration/bench.test.ts` Test 2, whose fixtures were English prose labelled - `python`. §28 refuses the elision (no symbols in English), the input returns verbatim, and - the only casualty is the reduction. That is the failure direction 4b.2's probe should also - aim for — and it is the measured answer to §6's risk 5, which asked that the probe "fail to - today's behaviour rather than to a wrong answer". - ---- - -## 9. 4b.2 as landed — the probe, and the step this document did not specify - -**Date:** 2026-08-06. Implemented: DECISIONS §31, `test/unit/python-content-probe.test.ts`. - -### What shipped - -`classifyContent` became a wrapper over `classifyContentShape`, which returns -`{ contentType, language? }`. The Python probe sits behind json/yaml/html/logs and ahead of -markdown, and sets both fields at once — §3's finding, implemented rather than argued. - -### The addition: the probe proposes, the parser confirms - -§6's risk 2 asked whether a fragment that fails the indentation rule should fall back or be -reported as uncheckable. The answer turned out to be neither, and it is a third option this -document did not consider: - -> **A probe may only claim content the validator for that language already accepts.** - -A declaration is the caller's assertion — failing on it is right, and §29 pins that case. A -detection is *our* guess, and content that does not parse is far likelier to mean the guess was -wrong than that the user's data is broken. Failing closed on our own guess is how a heuristic -becomes a fallback generator, which is precisely the trade §17 refused. - -`PythonValidator` imports types only, so the model layer can consult it with no cycle, and it -runs only for candidates the regex pass already accepted. - -Two measurements make this more than a nicety. The confirmation **fires**: a bad indent level, -an unterminated string and a call truncated mid-argument each clear the structural rule and are -each rejected by the parser. And it **costs nothing**: all six `pip` files the probe declines -parse fine, so the structural rule — not the validator — is what turned them down. - -### Measured - -| | detected | false positives | -|---|---|---| -| 45 `pip` Python (positive) | **39 (86.7%)** | — | -| 64 repo TypeScript | — | **0** | -| 25 repo markdown | — | **0** | -| repo YAML, `sample_logs.txt` | — | **0** | - -| route | before | after | -|---|---|---| -| `pip` over stdin, undeclared | 0.02%, 1 file, 0/45 checked | **12.27%, 19 files, 39/45 checked** | -| `pip` as a file argument | 12.34% | 12.34% — **0 collateral** | -| repo TS over stdin | 0.07% | 0.07% — **0 files changed** | - -99.4% of the filename route's yield, recovered without a filename. Gateway turn 1: no fallback, -byte-identical output. Turn 2: byte-identical before and after. Deterministic 6/6. - -### Correction to §7's expected value - -§7 projected `2.42% → up to 14.11%` for this corpus. The landed figure is `0.02% → 12.27%`, but -the two are not comparable in either direction: §7 used the engine's own estimator (24% MAE) -over a 39-file selection, and this uses `cl100k_base` over a 45-file selection re-frozen on -2026-08-05. The comparable pair is the one in the table above — the same corpus, the same -tokenizer, stdin against the filename route. - -### What 4b.2 does not close, and 4b.3 - -An **undetected** pathless Python file is not merely unoptimized — §8's addendum said a path is -not a declaration, and the same holds for a non-detection. `pip`'s `status_codes.py` is a -symbol-free constants file the probe declines; over stdin nothing covers it, §28's refusal -cannot fire, and it is elided whole and unwitnessed (44 → 27 tokens) while the file route -correctly refuses it. Detection narrows that population, it does not close it. - -**4b.3 is unchanged and still wanted.** Undetected Python and pathless code in every other -language still harvest `#` and `- ` lines as structural markers from the `text` and `unknown` -buckets. 4b.2 fixed the fabrication for *detected* Python only, exactly as §5 said it would. - ---- - -## 10. 4b.3 as landed — the allowlist was not where the fabrication was - -**Date:** 2026-08-06. Implemented: DECISIONS §32, -`test/unit/markdown-marker-allowlist.test.ts`. - -### What §5 said, and what is true - -§5: *"Undetected Python, and pathless code in any other language, still harvest `#` and `- ` -lines as structural markers from the `text` and `unknown` buckets."* - -Two errors, one small and one that changes what the step is. - -**Small: `- ` lines are not harvested at all.** `collectMarkers` gates exactly three kinds — -`heading:` (`#{1,6}\s+`), `fence:` (` ``` `) and `section:` (`---`/`===`/`System:`/`User:`/ -`Assistant:`/`[Context]`/`[Instructions]`). There is no bullet branch and never was. - -**Large: the fabrication is not in `text` or `unknown`.** Measured pathless across five frozen -corpora: - -| bucket | files | gated markers | -|---|---|---| -| `text` (60 TS + 2 py) | 62 | **0** | -| `html`, `logs` | 2 | **0** | -| `unknown` | — | only returned for empty content | -| `markdown` — **9 shell scripts** | 9 | **591**, every one a `#` comment | -| `markdown` — **4 undetected `pip` files** | 4 | **45**, every one a `#` comment | -| `markdown` — 25 real documents | 25 | 477 headings + 47 fences + 23 sections, genuine | - -`looksLikeMarkdown` fires on a single `#` heading, so a shell script's first `# Copyright …` -line makes the whole file markdown. §2's original table (`py/text 23 files 308 headings`) was -taken before §22's classifier fix and 4b.2's probe moved that population; what is left sits in -`markdown`, where no allowlist edit can reach it without gutting the 25 real documents. - -### So 4b.3 landed as scoped, and is inert - -The list is now `markdown` alone — `text`, `html`, `logs` and `unknown` removed, because the -docblock's own rule says a type should default to not harvesting and the two "we could not -tell" buckets are the worst possible exceptions to it. 132 files over stdin, 40 over the file -route and both Gateway turns are byte-identical before and after. A latent-trap fix, stated as -such. - -### The finding, which is worth more than the fix - -`tclConfig.sh`, frozen, through the real CLI over stdin: - -``` -1,877 -> 19 tokens (99.0% deleted) fallbackUsed false driftScore 0.4 -astCoverage {checked: 0, unchecked: 1, uncheckedContentTypes: ["markdown"]} -driftCoverage {structMeasured: true, measured: true, contentMarkersBefore: 79, …} -``` - -`S_k` lands on exactly `0.400` — `1 - (0.6·1 + 0.4·0)`, every fabricated marker destroyed — -and passes because the gate is `> 0.40` rather than `>=`. §3 predicted files sitting at exactly -0.400 in the abstract; this is one, and it is being deleted whole. - -The harm is not the score. It is that `structMeasured: true` and `measured: true` are reported -on 79 comment lines, so the `DriftCoverage` reporting §28 added **so that this class would be -visible** says the item was witnessed. The fabricated markers forge the evidence that anything -was measured. - -### Where it belongs, and how it reframes §28 - -Not in the allowlist (`# Copyright …` and `# A heading` are the same bytes). Not obviously in -`looksLikeMarkdown` either — that is a classifier change with blast radius over every prose -item, the gotcha CLAUDE.md states outright and the way §17 was found. It belongs in drift, in -the question §28 deferred: what does drift owe an item no validator covers? - -§28 deferred that as a product question about **prose**. It is not. The population is -**everything no validator covers**, and that includes real source code in every language the -AST-lite suite does not implement — shell, Ruby, Go, Rust, SQL. "May TokenDamper compress -prose" and "may TokenDamper delete 99% of a shell script under a forged `measured: true`" are -not the same question, and the second one does not need a product decision. diff --git a/docs/retired-documents.md b/docs/retired-documents.md new file mode 100644 index 0000000..1da634b --- /dev/null +++ b/docs/retired-documents.md @@ -0,0 +1,56 @@ +# Retired documents + +Audit **M11**. These files were removed from the working tree because their conclusions already +live in `DECISIONS.md`, `CHANGELOG.md` or `docs/audit-remediation-status.md`, and maintaining a +second copy of an argument is how the two drift apart. + +**Nothing is lost.** Git keeps every one of them. To read any file as it stood at retirement: + +```bash +git log --diff-filter=D --format=%H -1 -- docs/phase-1d-drift-investigation.md +``` + +then `git show ^:docs/phase-1d-drift-investigation.md`, or in one step: + +```bash +git show "$(git rev-list -1 HEAD -- docs/phase-1d-drift-investigation.md)^:docs/phase-1d-drift-investigation.md" +``` + +Source comments that cite a retired document keep the citation and mark it `(retired)`. The +citation is still meaningful — it names a document and section that existed, and the command +above retrieves it. + +--- + +## What each held, and where the conclusion lives now + +| Retired file | What it was | Conclusion now in | +|---|---|---| +| `docs/phase-0-measurement-baseline.md` | The frozen-corpus baseline and the Seam 2 measurement | DECISIONS §33–§34; `docs/audit-remediation-status.md` §2 | +| `docs/phase-1-stabilization-summary.md` | Phase 1.0 / Issue 2 summary report | DECISIONS §16, §22–§23; CLAUDE.md invariant 8 | +| `docs/phase-1d-drift-investigation.md` | Diagnostic record for the drift gate; the `S_k = 0.60` formula constant | DECISIONS §19, §28, §40; CLAUDE.md's Issue 3 entry | +| `docs/phase-1d-granularity-design.md` | Design proposal for sub-item hashing granularity | Implemented; DECISIONS §43 and `core/elision/regions.ts` | +| `docs/phase-1d-semantic-gate-disposition.md` | Disposition of the semantic gate's precondition (a) | DECISIONS §42; `core/constraints/directives.ts` | +| `docs/phase-4b-lever-disposition.md` | Measurement of three proposed levers against §32 | DECISIONS §33–§34, §40 | +| `docs/phase-4b-pathless-code-scope.md` | Scope of pathless code being invisible to validators | DECISIONS §29, §31; CLAUDE.md's Issue 2 entry | +| `docs/issue-2-content-type-contract-design.md` | The content-type contract design proposal | Implemented; DECISIONS §22–§23, §45 | +| `NOTES-FOR-DOCS.md` | Corrections to planning docs, recorded rather than edited in place | Folded into the documents they corrected | +| `tokendamper-headroom-known-issues.md` | The TokenDamper-vs-Headroom benchmark issue list | CLAUDE.md "Known bugs"; `docs/audit-remediation-status.md` | +| `study.md` | Onboarding guide for new contributors | `README.md` and `CLAUDE.md` | +| `purposed architecture changes.md` | Proposed architecture changes (pre-audit) | DECISIONS §22–§23, §35, §45 | + +--- + +## Why the ratio argument was weaker than it looked + +M11 was raised as a **4.1 : 1** documentation-to-code ratio (528 KB markdown against 127 KB of +`src/`). Measured immediately before this cleanup, it was **1.40 : 1** — and the improvement was +not real. Markdown had *grown* to 726 KB; `src/` had grown faster, to 518 KB. + +More to the point, **32.8% of `src/` is comment prose** (165 KB of 518 KB, 2,972 of 12,607 +non-blank lines). Counting that honestly, prose ran about **2.6 : 1** against code. The volume +did not shrink on its own; some of it moved into the source files, where it is at least adjacent +to what it describes. + +This retirement removes ~230 KB of narrative. The in-source commentary is deliberately left +alone: it is the part that sits next to the code it explains and is maintained with it. diff --git a/purposed architecture changes.md b/purposed architecture changes.md deleted file mode 100644 index bc2d9ce..0000000 --- a/purposed architecture changes.md +++ /dev/null @@ -1,100 +0,0 @@ -# Proposed Architecture Changes — TokenDamper - -## Context -Current architecture (MVP, linear pipeline, no DAGs): - -```text -Raw Input (JSON/Text) - -> Adapter (CLI / HTTP Gateway / MCP) - -> ContextBundle + OptimizationBudget - -> Stateless 0/1 Knapsack Planner - -> Linear Engine - -> Session Deduplication (TokenHasher) - -> Delta Compression (Myers Diff) - -> Workspace Topology Pruning - -> Validators (ConfidenceLedger, DebtTracker, DriftTracker) - -> Fallback (if safety thresholds violated) - -> Final Output (Optimized Context or Raw Input) - -> Explainability Trace (stderr / JSON) -``` - -This shape is a reasonable MVP skeleton. The bugs surfaced during benchmarking (see -`tokendamper-headroom-known-issues.md`) trace back to two specific architectural gaps in -this pipeline, not implementation typos. These changes are scoped fixes, not a rewrite — -the linear shape and knapsack planner should stay as-is. - ---- - -## Change 1 — Make content-type a first-class planning input, not a post-hoc validation discovery - -**Problem:** The planner allocates *how much* to compress but has no awareness of *what -kind* of content it's compressing. `compression:token-hashing` writes a -`` placeholder into JSON content, and only the downstream AST/JSON -validator discovers this broke syntax — after the transform has already run. This is the -root cause of the self-inflicted JSON corruption bug (Issue 2 in the known-issues file). - -**Proposed fix:** `ContextBundle` should carry a content-type tag (e.g. `json`, `code`, -`prose`, `logs`) determined at ingestion, and the Knapsack Planner should consult this tag -when selecting eligible stages — before any transform runs, not after. Concretely, either: -- Skip `compression:token-hashing` entirely for `json`-tagged bundles, or -- Change the hashing stage's placeholder format so it's syntactically valid for the - detected content type (e.g. a quoted string token instead of a bare `<...>` tag). - -Content-awareness belongs in the planning stage, not discovered downstream in validation. - ---- - -## Change 2 — Replace the single global validate→fallback gate with per-stage checkpointing - -**Problem:** `Validators` currently runs once, after the entire `Linear Engine` completes, -and `Fallback` is a single global action. This means if any one stage (e.g. stage 3 of 4, -token-hashing) produces invalid output, the *entire* pipeline's work is discarded — -including safe, valid reductions already achieved by earlier stages (Session Dedup, Delta -Compression). This is why `tool_output.json` and `session.json` landed at 0%/-1.39% -instead of partial reductions: every observed fallback discarded compute that was already -valid. - -**Proposed fix:** Validate incrementally after each stage in the Linear Engine, not just -once at the end. On a stage-level validation failure, roll back only that stage's -transform and keep the output of prior stages. This doesn't require a full DAG rewrite — -it's a checkpoint-and-partial-rollback mechanism within the existing linear sequence. This -alone would likely convert at least 2 of the current 0%-fallback cases into partial -reductions. - ---- - -## Change 3 — Guarantee the fallback path is byte-identical to raw input - -**Problem:** The -1.39% anomaly on `session.json` (Issue 5 in the known-issues file) -indicates "Fallback" currently means "re-render `currentBundle`" rather than "return the -untouched raw input bytes." Re-rendering from an internal bundle model is how a fallback -ends up a different size than the original even when nothing was supposed to change. - -**Proposed fix:** Split "fallback" into two distinct code paths: -1. **Raw passthrough** — bypasses the bundle/render model entirely and echoes the - original input verbatim. This should be the actual fallback path. -2. **Bundle rendering** — used only for genuinely successful (non-fallback) output. - -Byte-identical fallback should be a structural guarantee (impossible to violate by -construction), not something enforced only by testing. - ---- - -## What to keep unchanged -- The linear (non-DAG) pipeline shape — no need for a full rewrite. -- The Knapsack Planner's budget-allocation logic. -- The explainability trace (`planMode`, `stageCount`, `tokenBefore`/`tokenAfter`, - `fallbackReason`) — this should be **extended** to report per-stage status once - checkpointing (Change 2) is added, not redesigned. - ---- - -## Net effect of these three changes -Same overall architecture, but: -- Content-type flows into planning decisions instead of being discovered as a failure. -- A single bad stage no longer wipes out otherwise-valid upstream compression. -- Fallback output is guaranteed identical to raw input, closing the -1.39% class of bug. - -These are scoped, additive changes to the existing pipeline — appropriate to hand to -Claude Code alongside `tokendamper-headroom-known-issues.md` as follow-up implementation -work. diff --git a/src/core/constraints/directives.ts b/src/core/constraints/directives.ts index e9c9a26..197be42 100644 --- a/src/core/constraints/directives.ts +++ b/src/core/constraints/directives.ts @@ -25,7 +25,7 @@ const PROSE_CONTENT_TYPES: ReadonlySet = new Set([ * So neither "trust it everywhere" nor the audit's proposed "skip `code` entirely" is right: * the first keeps 51 false positives, the second discards 54 genuine constraints. What separates * them is not the content *type* but the region — an instruction to a reader lives in a comment - * or a docstring, never in an expression. `docs/phase-1d-semantic-gate-disposition.md` measured + * or a docstring, never in an expression. `docs/phase-1d-semantic-gate-disposition.md` measured [retired] * that this check is what catches Python docstring loss, and that is preserved here precisely * because docstrings stay in scope. * diff --git a/src/core/elision/index.ts b/src/core/elision/index.ts index 313a959..6a4bc6b 100644 --- a/src/core/elision/index.ts +++ b/src/core/elision/index.ts @@ -217,7 +217,7 @@ export interface ElideRegionsParams { * symbol in the bundle dies at once. `DriftTracker`'s `R_AST` is then a boolean and `S_k` * is pinned at the formula constant `0.60` — above the `0.40` gate, every time, structurally. * Whole-item hashing can never succeed on a single-item code bundle - * (`docs/phase-1d-drift-investigation.md` §6). Regions give the metric something fractional + * (`docs/phase-1d-drift-investigation.md` §6). Regions give the metric something fractional [retired] * to grade, and measured, it grades correctly. * * Two rules govern where a region may start and end, and both were found by measurement diff --git a/src/core/elision/regions.ts b/src/core/elision/regions.ts index 9ffc388..aee3a25 100644 --- a/src/core/elision/regions.ts +++ b/src/core/elision/regions.ts @@ -351,6 +351,41 @@ export interface SelectRegionsOptions { readonly minRegionBytes?: number; } +/** + * The languages `selectElisionRegions` can select sub-item regions for. + * + * Exported so the language-support report is derived from the gate rather than restating it. + * The two used to be the same fact written twice in different files, which is how audit M5b's + * marker formats drifted apart; this list and the check below must not repeat that. + */ +export type RegionElisionLanguage = 'typescript' | 'python'; + +export const REGION_ELISION_LANGUAGES: ReadonlyArray = Object.freeze([ + 'typescript', + 'python', +]); + +/** + * The region-selectable language for this item, or `undefined` if there is none. + * + * The first of the two gates behind audit H2, and the narrowing `selectElisionRegions` needs — + * one function rather than a predicate plus a second membership test that could disagree with it. + * An item that yields `undefined` can only be elided *whole*, which then has to survive the + * measurement gate; for a language whose symbols the drift tracker cannot see, that is refused + * by construction. + */ +export function regionElisionLanguage(item: ContextItem): RegionElisionLanguage | undefined { + const language = selectValidator(item)?.language; + return language !== undefined && (REGION_ELISION_LANGUAGES as ReadonlyArray).includes(language) + ? (language as RegionElisionLanguage) + : undefined; +} + +/** Whether sub-item elision is available for this item's language. */ +export function supportsRegionElision(item: ContextItem): boolean { + return regionElisionLanguage(item) !== undefined; +} + /** * Selects the sub-item regions of `item` that may be elided. * @@ -368,9 +403,8 @@ export function selectElisionRegions( item: ContextItem, options?: SelectRegionsOptions, ): ReadonlyArray { - const validator = selectValidator(item); - const language = validator?.language; - if (language !== 'typescript' && language !== 'python') { + const language = regionElisionLanguage(item); + if (language === undefined) { return []; } diff --git a/src/core/engine/index.ts b/src/core/engine/index.ts index cf62adc..6303524 100644 --- a/src/core/engine/index.ts +++ b/src/core/engine/index.ts @@ -242,6 +242,7 @@ export function optimize( ...(validation.driftReport ? { driftReport: validation.driftReport } : {}), ...(validation.astCoverage ? { astCoverage: validation.astCoverage } : {}), ...(validation.driftCoverage ? { driftCoverage: validation.driftCoverage } : {}), + ...(validation.languageSupport ? { languageSupport: validation.languageSupport } : {}), }); } @@ -263,6 +264,7 @@ export function optimize( ...(validation.driftReport ? { driftReport: validation.driftReport } : {}), ...(validation.astCoverage ? { astCoverage: validation.astCoverage } : {}), ...(validation.driftCoverage ? { driftCoverage: validation.driftCoverage } : {}), + ...(validation.languageSupport ? { languageSupport: validation.languageSupport } : {}), }); } @@ -287,6 +289,7 @@ export function optimize( ...(validation.driftReport ? { driftReport: validation.driftReport } : {}), ...(validation.astCoverage ? { astCoverage: validation.astCoverage } : {}), ...(validation.driftCoverage ? { driftCoverage: validation.driftCoverage } : {}), + ...(validation.languageSupport ? { languageSupport: validation.languageSupport } : {}), }); } diff --git a/src/core/ledger/drift-tracker.ts b/src/core/ledger/drift-tracker.ts index 977df8b..55e04f1 100644 --- a/src/core/ledger/drift-tracker.ts +++ b/src/core/ledger/drift-tracker.ts @@ -8,7 +8,7 @@ import type { ContentType, ContextBundle, ContextItem } from '../model'; * Deliberately an allowlist, not a denylist of `code`/`yaml`. A new `ContentType` should * default to *not* harvesting these — an absent marker costs a little discrimination, an * invented one actively inflates drift, and the second failure is the one that has - * actually bitten (DECISIONS.md §18, `docs/phase-1d-drift-investigation.md` §7). + * actually bitten (DECISIONS.md §18, `docs/phase-1d-drift-investigation.md` §7). [retired] * * **`markdown` alone, since Phase 4b.3 (DECISIONS §32).** The list used to also hold `text`, * `html`, `logs` and `unknown`, which contradicted the paragraph above: `text` and `unknown` @@ -316,7 +316,7 @@ export class DriftTracker { // `measured: false`, no fallback, because no validator covers `.pl` and the rule // therefore never looked. That is the defect, not prose. // - // See `docs/phase-0-measurement-baseline.md` §5 and DECISIONS §33. + // See `docs/phase-0-measurement-baseline.md` §5 and DECISIONS §33. [retired] const unwitnessedItemIds = this.findUnwitnessedItems(beforeBundle, effectiveAfter); // The two gates, decided separately. See `DriftReport.measurementGate`. diff --git a/src/core/model/constructors.ts b/src/core/model/constructors.ts index a2f6ad3..9402651 100644 --- a/src/core/model/constructors.ts +++ b/src/core/model/constructors.ts @@ -138,7 +138,7 @@ export function createOptimizationRequest( * modes are pathless by construction — `optimize -` has no filename, and an MCP * `optimize_context` call is a string in a JSON-RPC frame — so without a declaration the * only signal left is the content probe, and §17 removed content-only code detection on - * purpose. See `docs/phase-4b-pathless-code-scope.md`. + * purpose. See `docs/phase-4b-pathless-code-scope.md`. [retired] * * **It sets `language` and `contentType` together, never one without the other.** Setting * only `language` leaves `contentType` at whatever the probe guessed — `text` for most @@ -438,6 +438,7 @@ export function createValidationReport(report: ValidationReport): ValidationRepo ...(report.driftReport === undefined ? {} : { driftReport: report.driftReport }), ...(report.astCoverage === undefined ? {} : { astCoverage: report.astCoverage }), ...(report.driftCoverage === undefined ? {} : { driftCoverage: report.driftCoverage }), + ...(report.languageSupport === undefined ? {} : { languageSupport: report.languageSupport }), }); } @@ -463,6 +464,7 @@ export function createOptimizationTrace(trace: OptimizationTrace): OptimizationT ...(trace.driftScore === undefined ? {} : { driftScore: trace.driftScore }), ...(trace.astCoverage === undefined ? {} : { astCoverage: trace.astCoverage }), ...(trace.driftCoverage === undefined ? {} : { driftCoverage: trace.driftCoverage }), + ...(trace.languageSupport === undefined ? {} : { languageSupport: trace.languageSupport }), }); } @@ -512,7 +514,7 @@ export function freeze(value: T): Readonly { * `MARKDOWN_MARKER_TYPES`, so the file's `#` comment leaders would be harvested as markdown * headings and then "destroyed" by the very elision the detection just enabled — measured at * 1,025 fabricated markers across 43 pathless Python files - * (`docs/phase-4b-pathless-code-scope.md` §2, D2). + * (`docs/phase-4b-pathless-code-scope.md` §2, D2). [retired] * * An extension never sets `language`. It does not need to — `selectValidator` consults `path` * on its own — and doing so would put a `language` on every file-route item, moving every @@ -1009,7 +1011,7 @@ const pythonConfirmingValidator = new PythonValidator(); /** * Detects Python by structure, then **confirms it by parsing**. Phase 4b.2. * - * The structural half is `docs/phase-4b-pathless-code-scope.md` §4's measured rule: + * The structural half is `docs/phase-4b-pathless-code-scope.md` §4's measured rule: [retired] * `strong >= 2 && (strong + weak) / counted >= 0.15 && disqualified / counted < 0.10`, where * comment lines are neutral — excluded from the numerator *and* the denominator, since `#` is * a Python comment and a markdown heading and cannot be evidence either way. @@ -1090,11 +1092,11 @@ function isPython(text: string): boolean { * * **The discriminator is shape, not count.** A threshold on how many `#` lines a file has * points the wrong way — `tclConfig.sh` carries 79 to `CODE_OF_CONDUCT.md`'s 12, so any count - * rule protects the shell script *less* (`docs/phase-4b-lever-disposition.md` §1). What + * rule protects the shell script *less* (`docs/phase-4b-lever-disposition.md` §1). What [retired] * separates them is that a real document also has fences, lists or links, and a commented * config fragment has none. Measured over the 289-file Phase 0 corpus: code misclassified as * markdown falls from **114 of 264 files to 12**, while **all 25** real documents are - * retained. See `docs/phase-0-measurement-baseline.md` §6. + * retained. See `docs/phase-0-measurement-baseline.md` §6. [retired] * * The 12 residual leaks are honest rather than spurious — two shell scripts with `- ` lists, * three Tcl files whose `[...]` command syntax matches the link regex, four pip files, and diff --git a/src/core/model/types.ts b/src/core/model/types.ts index 5979134..a223d36 100644 --- a/src/core/model/types.ts +++ b/src/core/model/types.ts @@ -243,6 +243,51 @@ export interface DriftCoverage { readonly unwitnessedItems: ReadonlyArray; } +/** + * Whether this build's transforms can reduce an item's language **at all**. + * + * The third member of the same family as `AstCoverage` and `DriftCoverage`, and it answers the + * question those two leave open: they say whether anything *looked*, this says whether anything + * *could have acted*. + * + * Twelve of the nineteen extensions `isCodeExtension` recognises cannot produce a non-zero + * reduction under any flag combination (audit H2), and the reason is structural rather than + * tunable — `--max-drift 0.99` does not move it: + * + * 1. `selectElisionRegions` returns `[]` unless the selected validator's language is + * `typescript` or `python`, so everything else can only be elided whole. + * 2. A whole-item elision has to survive the measurement gate, which needs symbols or content + * markers. `DriftTracker.extractSymbols` is regexes over JS/TS declarations, Python + * `def`/`class`/`import`, and JSON keys — a Go `func`, a Rust `fn`, a C function, a shell + * function, a SQL statement and a CSS rule each yield **none**. + * + * So a Go file returns 0% and looks exactly like a Go file with nothing worth compressing. This + * report is what separates them, and it is the same correction M5a made for budgets: a 0% result + * has to say whether anything ran. + */ +export interface LanguageSupportReport { + /** Items whose language has at least one route to a surviving reduction. */ + readonly supported: number; + /** Items for which reduction is impossible in this build, whatever the budget says. */ + readonly unsupported: number; + /** + * The distinct declared languages behind `unsupported`, for the message. Falls back to the + * content type when an item carries no declared language. + */ + readonly unsupportedLanguages: ReadonlyArray; + /** Whether *every* item is unsupported — the case where 0% is guaranteed before any stage runs. */ + readonly noneSupported: boolean; + /** + * The explanation, in prose, present only when something is unsupported. + * + * It lives *inside* the report rather than being printed alongside it because the CLI writes + * this trace to stderr as a JSON document, and consumers — including this repository's own + * tests — parse the whole stream. A friendly line prepended to that stream is a breaking + * change to the channel's contract, which is exactly what a first attempt at this did. + */ + readonly reason?: string | undefined; +} + /** * The immutable validation outcome for an optimization attempt. */ @@ -255,6 +300,7 @@ export interface ValidationReport { readonly driftReport?: DriftReport | undefined; readonly astCoverage?: AstCoverage | undefined; readonly driftCoverage?: DriftCoverage | undefined; + readonly languageSupport?: LanguageSupportReport | undefined; } /** @@ -306,6 +352,7 @@ export interface OptimizationTrace { readonly driftScore?: number | undefined; readonly astCoverage?: AstCoverage | undefined; readonly driftCoverage?: DriftCoverage | undefined; + readonly languageSupport?: LanguageSupportReport | undefined; } /** diff --git a/src/core/trace/index.ts b/src/core/trace/index.ts index ad6a7d0..7a8ea1c 100644 --- a/src/core/trace/index.ts +++ b/src/core/trace/index.ts @@ -93,5 +93,9 @@ export function buildTrace( // nothing to measure" indistinguishably, and the CLI's stderr trace is the only place a // one-shot run can notice the difference. ...(validation.driftCoverage === undefined ? {} : { driftCoverage: validation.driftCoverage }), + // And the same again for language support: a 0% run cannot otherwise say whether this build + // has any transform for the input's language at all, which is a different problem from the + // input being incompressible and calls for a different response. Audit H2. + ...(validation.languageSupport === undefined ? {} : { languageSupport: validation.languageSupport }), }); } diff --git a/src/core/validation/index.ts b/src/core/validation/index.ts index 2e89746..61613d4 100644 --- a/src/core/validation/index.ts +++ b/src/core/validation/index.ts @@ -3,6 +3,7 @@ import type { ContextBundle, ContextItem, DriftCoverage, + LanguageSupportReport, OptimizationBudget, OptimizationPlan, ValidationIssue, @@ -11,8 +12,10 @@ import type { import { extractConstraintDirectives } from '../../stages/cleanup/constraint-preservation'; import { DriftTracker } from '../ledger/drift-tracker'; import { validateBundleAst } from './ast'; +import { describeLanguageSupport } from './language-support'; export * from './ast'; +export * from './language-support'; export interface ValidationOptions { readonly maxDriftThreshold?: number | undefined; @@ -96,6 +99,11 @@ export function validate( // population that was being deleted unwitnessed. `calculateDrift` no longer takes it. const symbolBearingItemIds = new Set(after.items.filter((item) => !unchecked.has(item.id)).map((item) => item.id)); + // Computed over `before`, not `after`: the question is what this build could have done to the + // input, which is a property of the input's languages and does not depend on what the stages + // managed to do (audit H2). + const languageSupport: LanguageSupportReport = describeLanguageSupport(before); + const driftReport = driftTracker.calculateDrift(before, after); const driftCoverage: DriftCoverage = { @@ -152,6 +160,23 @@ export function validate( }); } + // 6. Report language support, without voting on it either. + // + // An unsupported language is not an error — pass-through is byte-identical and correct. What + // it must not do is look like a supported language that happened to have nothing worth + // removing. Those two produce the identical `reductionRatio: 0` and only one of them is about + // the user's file (audit H2). This is the same correction M5a made for budgets. + if (languageSupport.unsupported > 0) { + const languages = languageSupport.unsupportedLanguages.join(', '); + issues.push({ + code: 'LANGUAGE_NOT_ELIDIBLE', + message: languageSupport.noneSupported + ? `No elision transform in this build can reduce ${languages}: there is no sub-item region selector for it, and whole-item elision cannot survive the drift gate. Elision reduces TypeScript/JavaScript and Python only. A 0% result here is structural, not a property of this input.` + : `${languageSupport.unsupported} of ${before.items.length} item(s) are in a language elision cannot reduce (${languages}); only whole-item pruning can affect them.`, + severity: 'info', + }); + } + // Verdicts are error-scoped. `issues.length === 0` was equivalent while every issue pushed // here was an error, but it makes the `severity` field decorative and turns any future // informational finding into a forced fallback. @@ -169,6 +194,7 @@ export function validate( driftReport, astCoverage, driftCoverage, + languageSupport, ...(reason ? { reason } : {}), }; } diff --git a/src/core/validation/language-support.ts b/src/core/validation/language-support.ts new file mode 100644 index 0000000..4b52672 --- /dev/null +++ b/src/core/validation/language-support.ts @@ -0,0 +1,82 @@ +import type { ContextBundle, ContextItem, LanguageSupportReport } from '../model'; +import { supportsRegionElision } from '../elision/regions'; + +/** + * Answers, before any stage runs, whether elision can reduce these items at all. + * + * Audit H2: twelve of the nineteen extensions `isCodeExtension` recognises cannot produce a + * non-zero reduction under any flag combination, and neither gate is threshold-controlled, so + * `--max-drift 0.99` does not move them. + */ + +/** + * Whether **elision** has any route to reducing this item. + * + * The answer is exactly `supportsRegionElision`, and the derivation matters because a looser + * predicate is tempting and wrong: + * + * - **Sub-item region elision** exists only for TypeScript/JavaScript and Python + * (`selectElisionRegions` returns `[]` for everything else). + * - **Whole-item elision of a symbol-bearing item** is refused outright by + * `compression:token-hashing` — since §40 a code item scores `S_k = 1 - R_AST`, and + * destroying every symbol makes that 1.0 against a gate that fires above 0.40. There is no + * threshold under which it survives (§43). + * - **Whole-item elision of a symbol-free item** is attempted, and then fails the same way one + * layer along: with no symbols, `R_AST` does not vote (§40), so `R_struct` decides — and + * eliding the whole item destroys every content marker, giving `R_struct = 0`. An item with + * neither symbols nor markers is refused by the measurement gate instead (§33). + * + * So for anything outside the two region-selectable languages, every elision route terminates in + * a refusal. That prediction matches the measured corpus exactly: python and typescript reduce; + * shell, perl, tcl, c, rust, css and prose are 0.00% on both routes. + * + * **A first attempt at this used "does the item yield symbols or markers?" and was wrong.** + * A trivial Go file yields exactly one symbol — `import:fmt`, an incidental match by the + * TypeScript import regex — which made Go read as supported while it still cannot reduce. The + * gate to ask about is the one that actually decides. + * + * Note this is scoped to elision. `pruning:topology-pruner` drops whole items and is + * language-agnostic, but it is selection rather than elision and needs a multi-item bundle; + * `noneSupported` is worded accordingly. + */ +export function isElisionReducible(item: ContextItem): boolean { + return supportsRegionElision(item); +} + +/** + * Builds the bundle-level report. Pure and cheap — a validator lookup per item, no content scan. + */ +export function describeLanguageSupport(bundle: ContextBundle): LanguageSupportReport { + const unsupportedLanguages = new Set(); + let supported = 0; + let unsupported = 0; + + for (const item of bundle.items) { + if (isElisionReducible(item)) { + supported += 1; + continue; + } + unsupported += 1; + // The declared language is the useful name when there is one — it is what the caller typed. + // Falling back to the content type keeps the message concrete for an undeclared item rather + // than printing `undefined`. + unsupportedLanguages.add(item.language ?? item.contentType); + } + + const languages = [...unsupportedLanguages].sort(); + const noneSupported = bundle.items.length > 0 && supported === 0; + + return { + supported, + unsupported, + unsupportedLanguages: Object.freeze(languages), + noneSupported, + ...(unsupported === 0 + ? {} + : { + reason: noneSupported + ? `Elision cannot reduce ${languages.join(', ')} in this build: there is no sub-item region selector for it, and whole-item elision cannot survive the drift gate. Elision reduces TypeScript/JavaScript and Python only, so 0% here is structural rather than a property of this input. Whole-item pruning is language-agnostic but needs a multi-item bundle.` + : `${unsupported} of ${bundle.items.length} item(s) are in a language elision cannot reduce (${languages.join(', ')}); only whole-item pruning can affect them.`, + }), + }; +} diff --git a/src/gateway/proxy.ts b/src/gateway/proxy.ts index f4d5327..59e1a41 100644 --- a/src/gateway/proxy.ts +++ b/src/gateway/proxy.ts @@ -435,7 +435,7 @@ function runGatewayOptimization( // // Note the drift exemption in DriftTracker covers `recoverable` (dedup) elisions only. // The lossy compression stages are still scored in full, so drift is a second, separate - // blocker on widening this list. See docs/phase-1-stabilization-summary.md (§5.3, §7). + // blocker on widening this list. See docs/phase-1-stabilization-summary.md (§5.3, §7). [retired] const config: ResolvedConfig = { ...baseConfig, planner: { ...baseConfig.planner, defaultMode: 'session_dedup' }, @@ -506,7 +506,7 @@ function runGatewayOptimization( * retention was vacuously 1.0 and drift vacuously 0.00. * * Both checks were reporting a pass they had never performed. See - * `docs/issue-2-content-type-contract-design.md` §2.2. + * `docs/issue-2-content-type-contract-design.md` §2.2. [retired] * * **This closed the JSON half only, and the record overstates it.** `classifyContent` does * see JSON as JSON, so the drift half above holds. It did *not* start selecting a validator diff --git a/src/stages/compression/token-hashing.ts b/src/stages/compression/token-hashing.ts index 8a04731..384f819 100644 --- a/src/stages/compression/token-hashing.ts +++ b/src/stages/compression/token-hashing.ts @@ -131,7 +131,7 @@ export function runTokenHashingStage( // Rule 5: prefer sub-item granularity. Whole-item hashing replaces every byte, so every // symbol in a single-item code bundle dies at once, `R_AST` is a boolean and `S_k` pins // at 0.60 — over the gate, every time, structurally - // (`docs/phase-1d-drift-investigation.md` §6). Eliding function bodies leaves the + // (`docs/phase-1d-drift-investigation.md` §6). Eliding function bodies leaves the [retired] // declarations that carry the symbols, so drift has something fractional to grade. // // `selectElisionRegions` returns nothing for content it cannot segment safely — JSON, diff --git a/study.md b/study.md deleted file mode 100644 index ad1a6ec..0000000 --- a/study.md +++ /dev/null @@ -1,101 +0,0 @@ -# TokenDamper Study Guide & Onboarding Document - -Welcome to **TokenDamper**! This document is designed for new developers joining the project. It provides the complete context of what the engine is, what has been happening in development up until now, how the architecture works, the mathematics powering it, and its current safety boundaries and vulnerabilities. - ---- - -## 1. What is TokenDamper? - -**TokenDamper** is a universal context optimization engine for AI coding assistants. When users send massive context bundles (prompts, entire codebases, file diffs, log files) to an LLM, it costs a lot of money and slows down response times. - -TokenDamper sits in the middle (as a CLI, local Gateway HTTP proxy, or MCP server) and aggressively but *safely* compresses this context before it hits the LLM. It guarantees correctness and semantics while reducing token usage and latency. - -### How it differs from other compressors (e.g., Headroom) -- **Headroom** largely relies on Machine Learning models (like the `kompress` model) to heuristically summarize and squash text, backed by SmartCrusher and CacheAligner algorithms. If the ML model isn't available, it falls back to basic structural dedup. -- **TokenDamper** is strictly **deterministic and mathematical**. It doesn't use an ML model to guess what to compress. Instead, it uses a Topology-Aware **0/1 Knapsack algorithm** to pack the most valuable context into a strict budget, reversible **SHA-256 Token Hashing** to deduplicate cross-turn sessions, and **Myers Diff** to send only changed lines. TokenDamper heavily prioritizes **safety** over aggressive compression. - ---- - -## 2. Context: What's Happening Right Now? - -We recently built and ran a full benchmark suite comparing TokenDamper against Headroom using realistic enterprise payloads (`sample_logs.txt`, `tool_output.json`, `codebase.py`, and a multi-turn `session.json`). Both engines were given a strict **30% token reduction target**. - -### The Benchmark Discovery -Initially, the benchmark harness had a bug where the budget wasn't explicitly passed down, causing both engines to default to 0% reduction (Pass-Through mode). We fixed the `isKnapsackMode` trigger in TokenDamper's `src/core/planner/index.ts` and rebuilt the engine. - -Once the budget was enforced, we discovered exactly how strict TokenDamper's safety boundaries are. In the benchmark, TokenDamper yielded a **0% reduction** on several files because it triggered **explicit safety fallbacks**: -1. **Constraint Loss**: While trying to compress `sample_logs.txt`, TokenDamper dropped a line containing a simulated secret key (`BLUE-PANDA-992`). The Validation stage caught this constraint loss and forced a fallback to the original payload to prevent data corruption. -2. **JSON AST Corruption**: For `tool_output.json` and `session.json`, the TokenHasher stage replaced large duplicate blocks with `` placeholders. Because these payloads were JSON, inserting ` Adapter (CLI / HTTP Gateway / MCP) - -> ContextBundle + OptimizationBudget - -> Stateless 0/1 Knapsack Planner - -> Linear Engine - -> Session Deduplication (TokenHasher) - -> Delta Compression (Myers Diff) - -> Workspace Topology Pruning - -> Validators (ConfidenceLedger, DebtTracker, DriftTracker) - -> Fallback (if safety thresholds violated) - -> Final Output (Optimized Context or Raw Input) - -> Explainability Trace (stderr / JSON) -``` - -### Core Subsystems -1. **Core Data Model (`src/core/model`)**: Frozen domain objects like `OptimizationRequest`, `ContextBundle`, `OptimizationBudget`, and `StageResult`. Immutability is preferred. -2. **Planner (`src/core/planner`)**: Pure and stateless. It looks at the budget and context and picks an `OptimizationPlan` (e.g., triggering `topology_knapsack` mode if a `targetReductionRatio` is present). -3. **Linear Engine (`src/core/engine`)**: Executes the chosen stages in order. -4. **Validation & Fallback (`src/core/validation`, `src/core/fallback`)**: Pure validators that compare the "before" and "after" state. If it detects a broken AST or dropped constraints, it tells the engine to completely fallback to the original raw input. -5. **Adapters (`src/adapters`)**: Allow TokenDamper to run as a CLI tool (`tokendamper optimize`), a transparent LLM proxy (`tokendamper exec`), or a Claude Desktop/Cursor server (`tokendamper mcp`). - ---- - -## 4. The Mathematics & Processes - -### 1. 0/1 Knapsack Planning -When a budget constraint is applied (`maxInputTokens` or `targetReductionRatio`), the planner treats context optimization as a **0/1 Knapsack Problem**. -- **Items**: Each file, function, or chat message is an item. -- **Weight**: The token count of the item. -- **Value**: A heuristic score assigned to the item based on its relevance (e.g., recent messages have high value, large unchanged files have lower value). -- **Goal**: Maximize the total value without exceeding the token weight budget. - -### 2. Myers Diff (Delta Compression) -Instead of sending an entire modified file to the LLM again, TokenDamper computes the difference between the previously cached file and the new file using the deterministic **Myers Diff** algorithm. It then sends only the changed lines (the diff), saving massive amounts of tokens. - -### 3. Token Hashing (Session Deduplication) -If a massive block of text has been sent to the LLM previously, the `TokenHasher` stage replaces that chunk with a tiny placeholder: ``. When the LLM responds, or if it needs to be restored, TokenDamper rehydrates the original content. - -### 4. Semantic Drift ($S_k$) & Optimization Debt ($D_k$) -- **Optimization Debt ($D_k$)**: Measures raw information loss. If you remove 50% of the tokens, debt increases. -- **Semantic Drift ($S_k$)**: Measures how much the *meaning* or *structure* of the code has deviated from the original. TokenDamper runs an AST (Abstract Syntax Tree) check. If structural nodes are missing or corrupted, $S_k$ spikes. If $S_k > 0.40$, the engine falls back. - ---- - -## 5. Current Safety Features & Vulnerabilities - -### Safety Features -- **Strict AST Validation**: The engine parses the final output to ensure code and JSON syntaxes are still valid. If they aren't, it aborts compression. -- **Constraint Ledger**: It tracks critical pieces of information (like secrets, explicit user prompts, or system instructions). If these are accidentally pruned by the Knapsack algorithm, it catches the loss and aborts. -- **Unconditional Fallback**: When validation fails, the engine does not try to guess or "repair" the output. It 100% falls back to the original raw input to prevent hallucinations or data loss. - -### Current Vulnerabilities / Areas for Improvement -1. **JSON AST Corruption via Hashing**: Right now, if TokenDamper processes a huge JSON payload (like `tool_output.json`) and tries to deduplicate a string inside it, it injects ``. Because it does this via raw string replacement rather than AST-aware node replacement, it breaks JSON parsing boundaries. The safety validator catches this and falls back, meaning **JSON payloads frequently see 0% compression**. -2. **Aggressive Semantic Drift Constraints**: The max drift threshold (0.40) is currently very sensitive. In long Python files, removing even redundant functions can push the score above 0.40, resulting in the planner failing open (0% compression) far too often. -3. **No ML Heuristics**: While being deterministic is a safety feature, it means TokenDamper lacks the semantic understanding that an ML model has (like knowing *which* logs are useless noise vs. important stack traces). - ---- - -## Next Steps for New Developers -1. Review `run_benchmark.py` in `tokendamper-benchmark/` to see how we test the engine constraints. -2. Look at `src/core/planner/index.ts` to understand how the Knapsack plan is triggered. -3. Investigate `src/core/validation/` to see how the AST Validator catches the JSON TokenHasher bugs. Your first major PR could be making the `TokenHasher` AST-aware so it doesn't break JSON structures! diff --git a/test/integration/bench.test.ts b/test/integration/bench.test.ts index bcbcef9..af6a72d 100644 --- a/test/integration/bench.test.ts +++ b/test/integration/bench.test.ts @@ -179,7 +179,7 @@ describe('TokenDamper Regression Test Suite & Performance Baseline (R5)', () => // `TokenHasher` reached the engine and `attemptAutomatedRehydration` returned immediately // on `if (!hasher && !ledger)` — the recovery path never ran. With the hasher supplied, the // engine rehydrates the placeholder, re-validates, and passes on every humaneval fixture. - // That remains true and is still asserted below. See docs/phase-1d-drift-investigation.md §10. + // That remains true and is still asserted below. See docs/phase-1d-drift-investigation.md §10. [retired] // // What was wrong was not the assertion but its **scope**. `maxFallbackRate: 0` was read as // a statement about the product; it was only ever a statement about humaneval, which is diff --git a/test/unit/bench/m1_reverification.test.ts b/test/unit/bench/m1_reverification.test.ts index e85d75d..5ff82f5 100644 --- a/test/unit/bench/m1_reverification.test.ts +++ b/test/unit/bench/m1_reverification.test.ts @@ -28,7 +28,7 @@ describe('M1 Challenger Re-verification Suite', () => { // Restored to 0, which is what the test name asserts. `aba84df` inverted this to // 1 and blamed Issue 3 (the drift threshold). The real cause was BenchmarkRunner // calling optimize() with no TokenHasher, which left the engine's rehydration - // recovery path switched off. See docs/phase-1d-drift-investigation.md §10. + // recovery path switched off. See docs/phase-1d-drift-investigation.md §10. [retired] expect(sweep.summary.fallbackRate).toBe(0); }); diff --git a/test/unit/bench/runner.test.ts b/test/unit/bench/runner.test.ts index 214fb87..f882c4f 100644 --- a/test/unit/bench/runner.test.ts +++ b/test/unit/bench/runner.test.ts @@ -112,7 +112,7 @@ describe('BenchmarkRunner Harness', () => { // Restored to 0, which is what the test name asserts. `aba84df` inverted this to 1 // and blamed the drift threshold; the real cause was the engine's rehydration // recovery path being switched off by BenchmarkRunner passing no TokenHasher. - // See docs/phase-1d-drift-investigation.md §10. + // See docs/phase-1d-drift-investigation.md §10. [retired] expect(sweep.summary.fallbackRate).toBe(0); }); }); diff --git a/test/unit/benchmark-harness-route.test.ts b/test/unit/benchmark-harness-route.test.ts index b6aa06e..eb8056a 100644 --- a/test/unit/benchmark-harness-route.test.ts +++ b/test/unit/benchmark-harness-route.test.ts @@ -17,7 +17,7 @@ import { join } from 'node:path'; * * This is a property of the harness, not of the engine, so it is pinned here as text rather * than by running the pipeline. The engine-side defect — that pathless input is invisible to - * the validator layer at all — is real, tracked in `docs/phase-4b-pathless-code-scope.md`, + * the validator layer at all — is real, tracked in `docs/phase-4b-pathless-code-scope.md`, [retired] * and deliberately NOT asserted here. Fixing it must not make this test pass by accident. */ describe('benchmark harness invocation route', () => { diff --git a/test/unit/constraint-prose-scope.test.ts b/test/unit/constraint-prose-scope.test.ts index b1858ef..37050ad 100644 --- a/test/unit/constraint-prose-scope.test.ts +++ b/test/unit/constraint-prose-scope.test.ts @@ -25,7 +25,7 @@ import { validate } from '../../src/core/validation'; * * That rules out both extremes. Trusting it everywhere keeps 51 false positives; the audit's * proposed "skip `code` entirely" discards 54 genuine constraints — including the Python - * docstring case `docs/phase-1d-semantic-gate-disposition.md` measured this check to be the only + * docstring case `docs/phase-1d-semantic-gate-disposition.md` measured this check to be the only [retired] * thing catching. The separator is the *region*, not the content type. * * Measured effect of scoping to prose regions and attributing per item: Python 14.98% -> 23.14%, diff --git a/test/unit/declared-language.test.ts b/test/unit/declared-language.test.ts index e8338ed..bb5e53a 100644 --- a/test/unit/declared-language.test.ts +++ b/test/unit/declared-language.test.ts @@ -118,7 +118,7 @@ describe('a declaration sets language and contentType together', () => { // // It used to be *wrong* validation: `CONTENT_TYPE_VALIDATORS.code` was the TypeScript // validator, so a `code` tag without a language was how Python got checked by the wrong - // checker — the trap `docs/phase-4b-pathless-code-scope.md` §6.3 named. That mapping is + // checker — the trap `docs/phase-4b-pathless-code-scope.md` §6.3 named. That mapping is [retired] // now `null`, because `code` is a *family*, not a language, and lexing the whole family // as TypeScript invented findings rather than weakening them (perl 39/40, tcl 30/40, // shell 22/40). diff --git a/test/unit/drift-unwitnessed-elision.test.ts b/test/unit/drift-unwitnessed-elision.test.ts index 48c1f7c..0b441c1 100644 --- a/test/unit/drift-unwitnessed-elision.test.ts +++ b/test/unit/drift-unwitnessed-elision.test.ts @@ -24,7 +24,7 @@ import { parse } from '../../src/adapters/cli'; * whole on the file-argument route at `S_k = 0`, `measured: false`, no fallback, because * nothing covers `.pl`. The Gateway keeps within-payload deduplication either way, because * `resolveRecoverableElisions` substitutes recoverable elisions back before the rule runs. - * See `docs/phase-0-measurement-baseline.md` §5 and DECISIONS §33. + * See `docs/phase-0-measurement-baseline.md` §5 and DECISIONS §33. [retired] */ describe('drift refuses to certify an elision it has no evidence for', () => { const tracker = new DriftTracker(); diff --git a/test/unit/gateway.test.ts b/test/unit/gateway.test.ts index aeccb86..2648233 100644 --- a/test/unit/gateway.test.ts +++ b/test/unit/gateway.test.ts @@ -269,7 +269,7 @@ describe('Gateway HTTP & Proxy Interceptor', () => { // because the consumer is a stateless provider API that never calls `rehydrate_context`. // The content is deleted, and conversational prose carries neither symbols nor content // markers — so the elision cannot be evidenced and drift refuses it. The §9 addendum in - // docs/phase-1-stabilization-summary.md already described this population as sending the + // docs/phase-1-stabilization-summary.md already described this population as sending the [retired] // model a marker it has no way to resolve. const sessionStore = server.getSessionStore(); const soleContext = 'A paragraph of ordinary conversational context that appears exactly once per turn'; @@ -491,7 +491,7 @@ describe('Gateway HTTP & Proxy Interceptor', () => { // only because `DriftTracker.extractSymbols` harvests `jsonkey:` symbols solely when // `contentType === 'json'`. Tagged `text`, a JSON document yielded zero symbols, // retention was vacuously 1.0 and drift vacuously 0.00 — a pass produced by not - // looking. See docs/issue-2-content-type-contract-design.md §2.2. + // looking. See docs/issue-2-content-type-contract-design.md §2.2. [retired] expect(res2.body).toBe(turn2); expect(res2.body).not.toContain('__td_block__'); diff --git a/test/unit/language-support.test.ts b/test/unit/language-support.test.ts new file mode 100644 index 0000000..465e8a3 --- /dev/null +++ b/test/unit/language-support.test.ts @@ -0,0 +1,140 @@ +import { describe, expect, it } from 'vitest'; +import { optimize } from '../../src/core/engine'; +import { + createContextBundle, + createOptimizationBudget, + createOptimizationRequest, +} from '../../src/core/model/constructors'; +import { DEFAULT_CONFIG } from '../../src/config/schema'; +import { validate } from '../../src/core/validation'; +import { describeLanguageSupport } from '../../src/core/validation/language-support'; +import type { OptimizationPlan } from '../../src/core/model/types'; + +/** + * Audit H2 — twelve of nineteen recognised languages cannot produce a non-zero reduction under + * any flag combination, and a user seeing 0% had no way to tell that apart from a file with + * nothing worth compressing. Those two call for different responses and looked identical. + * + * The decision taken was to keep accepting every language and **report why**, rather than narrow + * the accepted set: pass-through is byte-identical and harmless, so refusing it would remove a + * working behaviour to make a point. This is the same correction M5a made for budgets. + */ + +const JS_BODY = 'function alpha(a){ return a + 1; }\nfunction beta(b){ return b * 2; }\n'; +const PY_BODY = 'def alpha(a):\n return a + 1\n\ndef beta(b):\n return b * 2\n'; + +const bundleFor = (body: string, path: string, language?: string) => + createContextBundle(body, 'file', path, undefined, language); + +const PLAN: OptimizationPlan = { + planId: 'p', + mode: 'topology_knapsack', + stageIds: Object.freeze([]), + revalidationPoints: Object.freeze(['end']), + fallbackPolicy: 'original_input', +}; + +describe('which languages elision can reduce (H2)', () => { + // The three that can, and a representative spread of those that cannot. Kept as one table so + // the ratio is visible: this is the audit's "three of nineteen" headline, asserted. + const cases: ReadonlyArray = [ + ['typescript', 'x.ts', JS_BODY, true], + ['javascript', 'x.js', JS_BODY, true], + ['python', 'x.py', PY_BODY, true], + ['go', 'x.go', JS_BODY, false], + ['rust', 'x.rs', JS_BODY, false], + ['c', 'x.c', JS_BODY, false], + ['java', 'x.java', JS_BODY, false], + ['shell', 'x.sh', JS_BODY, false], + ['sql', 'x.sql', JS_BODY, false], + ['css', 'x.css', JS_BODY, false], + ['json', 'x.json', '{"alpha":1,"beta":2}', false], + ['markdown', 'x.md', '# Title\n\nSome prose that goes on for a little while.', false], + ['yaml', 'x.yaml', 'alpha: 1\nbeta: 2\n', false], + ]; + + for (const [language, path, body, expected] of cases) { + it(`${language} is ${expected ? '' : 'not '}reducible by elision`, () => { + const report = describeLanguageSupport(bundleFor(body, path, language)); + expect(report.supported).toBe(expected ? 1 : 0); + expect(report.unsupported).toBe(expected ? 0 : 1); + expect(report.noneSupported).toBe(!expected); + }); + } + + it('is three of the thirteen probed, matching the corpus', () => { + // The corpus agrees independently: python and typescript reduce; shell, perl, tcl, c, rust + // and css are 0.00% on both routes. A predicate that disagreed with that would be wrong + // however reasonable it looked. + const supported = cases.filter(([lang, path, body]) => describeLanguageSupport(bundleFor(body, path, lang)).supported > 0); + expect(supported.map(([lang]) => lang)).toEqual(['typescript', 'javascript', 'python']); + }); + + it('does not mistake an incidental symbol match for support', () => { + // The first version of this predicate asked "does the item yield symbols or markers?" and + // called Go supported, because a trivial Go file yields exactly one symbol: `import:fmt`, + // matched by the TypeScript import regex. It witnesses nothing about the function bodies, + // and Go still cannot reduce. The gate to ask about is the one that actually decides. + const go = 'package main\n\nimport "fmt"\n\nfunc compute(items []int) int {\n\ttotal := 0\n\treturn total\n}\n'; + expect(describeLanguageSupport(bundleFor(go, 'a.go', 'go')).noneSupported).toBe(true); + }); +}); + +describe('the report reaches the caller (H2)', () => { + it('validate() reports an info issue naming the language, and does not fail the run', () => { + const bundle = bundleFor(JS_BODY, 'a.go', 'go'); + const report = validate(bundle, bundle, PLAN, createOptimizationBudget({})); + + const issue = report.issues.find((i) => i.code === 'LANGUAGE_NOT_ELIDIBLE'); + expect(issue).toBeDefined(); + expect(issue!.severity).toBe('info'); + expect(issue!.message).toContain('go'); + // Informational, not a verdict — an unsupported language passes through correctly. + expect(report.passed).toBe(true); + expect(report.shouldFallback).toBe(false); + }); + + it('says nothing for a language elision can reduce', () => { + const bundle = bundleFor(JS_BODY, 'a.ts', 'typescript'); + const report = validate(bundle, bundle, PLAN, createOptimizationBudget({})); + expect(report.issues.find((i) => i.code === 'LANGUAGE_NOT_ELIDIBLE')).toBeUndefined(); + expect(report.languageSupport?.noneSupported).toBe(false); + }); + + it('survives every whitelist between validate() and the trace', () => { + // **This is the regression guard, and it is not hypothetical.** The field has to pass through + // four separate object literals that each enumerate their keys — `validate`'s return, + // `createValidationReport`, `buildTrace` and `createOptimizationTrace`. Three of them dropped + // it silently while it was being developed, and the symptom each time was + // `trace.languageSupport: undefined` with everything else correct. Asserting on the trace, + // rather than on `validate()`, is what catches that. + const config = { ...DEFAULT_CONFIG, budget: { ...DEFAULT_CONFIG.budget, targetReductionRatio: 0.5 } }; + const request = createOptimizationRequest('package main\n\nfunc main() {}\n', config, { + requestId: 'h2-trace', + adapterName: 'test', + adapterVersion: '1', + source: 'file', + sourcePath: 'a.go', + language: 'go', + }); + + const result = optimize(request, {}); + expect(result.trace.languageSupport).toBeDefined(); + expect(result.trace.languageSupport!.noneSupported).toBe(true); + expect(result.trace.languageSupport!.unsupportedLanguages).toContain('go'); + }); + + it('carries its explanation inside the report, not alongside it', () => { + // The CLI writes the trace to stderr as a JSON document and consumers parse the whole + // stream — `test/integration/cli.test.ts` and `byte-identity-fallback.test.ts` both do. + // A first attempt at H2 printed a friendly notice line *before* the JSON and broke four of + // them: stderr stopped being parseable. The explanation belongs in a field. + const bundle = bundleFor(JS_BODY, 'a.go', 'go'); + const report = describeLanguageSupport(bundle); + expect(report.reason).toMatch(/Elision cannot reduce go/); + + // And nothing is emitted for a supported language, so the field stays absent rather than + // carrying an empty string. + expect(describeLanguageSupport(bundleFor(JS_BODY, 'a.ts', 'typescript')).reason).toBeUndefined(); + }); +}); diff --git a/test/unit/markdown-marker-allowlist.test.ts b/test/unit/markdown-marker-allowlist.test.ts index 68502be..3f7a003 100644 --- a/test/unit/markdown-marker-allowlist.test.ts +++ b/test/unit/markdown-marker-allowlist.test.ts @@ -126,7 +126,7 @@ describe('only markdown yields markdown markers', () => { * `text`, harvests zero headings, and lands in the measurement gate's reach. * * §32 deferred seam 2 on the belief that it meant "require more than one `#` line" — a *count* - * threshold, which `docs/phase-4b-lever-disposition.md` had already shown points the wrong way, + * threshold, which `docs/phase-4b-lever-disposition.md` had already shown points the wrong way, [retired] * since `tclConfig.sh` carries 79 markers to `CODE_OF_CONDUCT.md`'s 12. The discriminator that * works is *shape*, and its measured cost to prose is zero files. * diff --git a/test/unit/python-content-probe.test.ts b/test/unit/python-content-probe.test.ts index 4a5b2d1..b5a4109 100644 --- a/test/unit/python-content-probe.test.ts +++ b/test/unit/python-content-probe.test.ts @@ -21,7 +21,7 @@ import { selectValidator, validateItemAst } from '../../src/core/validation/ast' * message is a provider payload with no language field anywhere in its schema, so for the * traffic the proxy actually carries a probe is the only route that exists. * - * Scoped to Python deliberately. `docs/phase-4b-pathless-code-scope.md` §4 measured + * Scoped to Python deliberately. `docs/phase-4b-pathless-code-scope.md` §4 measured [retired] * TypeScript positives at 0.283–1.000 against prose negatives reaching 0.333 — overlapping * ranges, no threshold orders them — because this repository's prose is documentation *about* * TypeScript, dense with fenced TypeScript. A TypeScript probe is not proposed, now or later. @@ -59,7 +59,7 @@ describe('the probe identifies Python and says so in both fields', () => { }); it('stops the marker fabrication that made D2 worth fixing before enabling elision', () => { - // `docs/phase-4b-pathless-code-scope.md` §2: pathless Python classified `text` or + // `docs/phase-4b-pathless-code-scope.md` §2: pathless Python classified `text` or [retired] // `markdown`, both on `MARKDOWN_MARKER_TYPES`, so `# NOTE: …` comment leaders were // harvested as structural markers — 1,025 of them across 43 files — and the next elision // then "destroyed" them. §3 measured that fixing the language *without* the content type diff --git a/test/unit/validator-guarantee.test.ts b/test/unit/validator-guarantee.test.ts new file mode 100644 index 0000000..4875412 --- /dev/null +++ b/test/unit/validator-guarantee.test.ts @@ -0,0 +1,93 @@ +import { describe, expect, it } from 'vitest'; +import { validateItemAst } from '../../src/core/validation/ast'; +import type { ContextItem } from '../../src/core/model/types'; + +/** + * Audit M1 — what the "AST-lite validators" actually guarantee, pinned as executable fact. + * + * The TypeScript validator builds no AST. It is a lexer that tracks strings, template + * interpolation, comments and regex literals, and detects exactly two error classes: unbalanced + * brackets and unterminated strings/comments. The product's headline property therefore means + * **bracket and quote integrity**, not syntax validity — which is now what `README.md` says. + * + * This file exists so that sentence cannot quietly stop being true. Every row below is also a + * row in the README table, and each was reproduced against the shipped validator rather than + * copied from the audit — three other audit claims in this project turned out to be wrong when + * measured (DECISIONS §40, §42, §45). + * + * **These are characterization tests, not aspirations.** A `PASS` on nonsense is the documented + * behaviour. If someone wires the real compiler API, these will fail — that is the signal to + * update the README table in the same change, not to weaken the test. + */ + +const item = (content: string, language: string, path: string): ContextItem => + ({ + id: 'i', + itemId: 'i', + kind: 'file', + contentType: 'code', + content, + origin: 'test', + contentHash: 'h', + language, + path, + metadata: {}, + }) as ContextItem; + +describe('the TypeScript validator checks balance, not syntax (M1)', () => { + const accepted: ReadonlyArray = [ + ['an assignment with no right-hand side', 'const x = ;'], + ['a parameter with no type after its colon', 'function f(a: , b) { return 1; }'], + ['an import with no binding', 'import from "x";'], + ['an identifier that starts with a digit', 'let 123abc = 5;'], + ['an operator pile-up', 'const a = 1 +++++ 2;'], + ['plain English prose', 'ceci nest pas du code'], + ]; + + for (const [label, source] of accepted) { + it(`accepts ${label} — balanced, and that is all it asks`, () => { + const result = validateItemAst(item(source, 'typescript', 'a.ts')); + expect(result.valid).toBe(true); + // `validated: true` matters as much as `valid: true`: this is a real check returning a + // pass, not the absence of a check reading as one (DECISIONS §23). + expect(result.validated).toBe(true); + }); + } + + it('rejects unbalanced brackets, which is the guarantee it does make', () => { + const result = validateItemAst(item('super(; }', 'typescript', 'a.ts')); + expect(result.valid).toBe(false); + }); +}); + +describe('the Python validator is meaningfully stronger, but not a parser (M1)', () => { + it('catches a missing colon after a def', () => { + expect(validateItemAst(item('def f()\n return 1', 'python', 'a.py')).valid).toBe(false); + }); + + it('catches stray leading indentation', () => { + expect(validateItemAst(item(' leading indent', 'python', 'a.py')).valid).toBe(false); + }); + + it('accepts well-formed Python', () => { + expect(validateItemAst(item('def f():\n return 1', 'python', 'a.py')).valid).toBe(true); + }); + + it('still accepts plain English prose', () => { + expect(validateItemAst(item('ceci nest pas du code', 'python', 'a.py')).valid).toBe(true); + }); +}); + +describe('the JSON validator is a real parser (M1)', () => { + it('accepts valid JSON', () => { + expect(validateItemAst(item('{"a":1}', 'json', 'a.json')).valid).toBe(true); + }); + + it('rejects malformed JSON', () => { + expect(validateItemAst(item('{"a":}', 'json', 'a.json')).valid).toBe(false); + }); + + it('rejects prose', () => { + expect(validateItemAst(item('not json at all', 'json', 'a.json')).valid).toBe(false); + }); +}); diff --git a/tokendamper-headroom-known-issues.md b/tokendamper-headroom-known-issues.md deleted file mode 100644 index 6d22dbe..0000000 --- a/tokendamper-headroom-known-issues.md +++ /dev/null @@ -1,213 +0,0 @@ -# TokenDamper vs Headroom Benchmark — Known Issues (context for Claude Code) - -## Project summary -Benchmarking two LLM context/token compression engines — **TokenDamper** (this repo, -TypeScript, CLI) and **Headroom** (third-party Python package) — using a harness at -`tokendamper-benchmark/run_benchmark.py`. The goal is a fair, apples-to-apples comparison -of reduction % and latency across static single-file payloads and a synthetic multi-turn -agent session. - -Repo entry points relevant to this work: -- `tokendamper-benchmark/run_benchmark.py` — the benchmark harness -- `src/core/planner/index.ts` — TokenDamper's planner (decides `pass_through` vs - `topology_knapsack` mode based on whether a budget/ratio is supplied) -- `src/core/validation/index.ts` — runs `DriftTracker`, enforces semantic drift ≤ 0.40, - issues `SEMANTIC_DRIFT_EXCEEDED` and forces fallback if exceeded -- `docs/v1_deployment_audit.md` — documents a now-fixed bug where `emittedOutput` used to - always return raw input regardless of what the engine computed (confirmed already fixed - in current `fallback/index.ts`, which now returns the rendered `currentBundle`) -- `test_data/session.json` — new 7-turn synthetic agent session fixture (added to test - cross-turn dedup / delta compression, which single-file payloads don't exercise) - ---- - -## Issue 1 — Harness gave neither engine a compression target (FIXED) -**Symptom:** Both TokenDamper and Headroom showed 0% reduction on every payload. - -**Root cause:** -- `run_tokendamper()` called `subprocess.run([cmd, "optimize", "-"], ...)` with no - `--max-input-tokens` / `--target-reduction-ratio` flag. Per `src/core/planner/index.ts`, - no budget → `isKnapsackMode = false` → mode = `pass_through` → `stageIds` stays an empty - array → zero optimization stages ever run. Guaranteed 0% regardless of input. -- `run_headroom()` called `headroom_compress(messages)` with no target/budget argument, and - wrapped every payload as a bare `{"role": "user", "content": text}` turn regardless of - what the payload actually represented (tool output vs. live user prompt). - -**Fix applied:** -- TokenDamper now called with `--target-reduction-ratio 0.3`. **Correction (re-verified):** - this flag alone was not sufficient to fix the harness. `isKnapsackMode` in - `src/core/planner/index.ts` originally only tripped on `budget.maxInputTokens`, so a - `--target-reduction-ratio`-only budget was silently inert — it still resolved to - `pass_through` mode with zero stages, regardless of the flag. A planner code change - (adding `targetReductionRatio > 0` as a second trigger for knapsack mode) was also - required before this flag had any effect. See `CLAUDE.md` Known bugs. -- Headroom now called with `target_ratio=0.7` (keep 70%), `protect_recent=0`, - `compress_user_messages=True`. -- Tool-output-shaped payloads (`tool_output.json`, `codebase.py`, etc.) are now wrapped as - `{"role": "tool", "tool_call_id": "...", "content": raw_text}` for Headroom instead of - `role: "user"`. -- `target_tokens` is explicitly computed and printed per payload so both engines are held - to the same bar. -- TokenDamper's stderr trace (`result.trace`) is now parsed and logged: `planMode`, - `stageCount`, `tokenBefore`, `tokenAfter`, `fallbackUsed`, `fallbackReason`. -- Added `test_data/session.json`, a multi-turn fixture, since single static files don't - exercise either engine's actual differentiators (cross-turn dedup, delta compression, - live-zone/session compression). -- Latency column now flagged as non-equivalent: TokenDamper is timed via - `subprocess.run()` (Node process spawn), Headroom via an in-process Python call. - -**Status:** Harness fix confirmed working — it now surfaces real engine behavior instead -of a config artifact. The issues below are what showed up *after* this fix. - ---- - -## Issue 2 — TokenDamper's own hash placeholders break its own JSON/AST validator (BUG, unresolved) -**Symptom:** On `tool_output.json` and `session.json`, TokenDamper falls back to 0% (or -worse) reduction with trace reason: -``` -AST Error... JSON Syntax Error: Unexpected token '<', "`-style placeholders. The validation pipeline then runs an -AST/JSON syntax check on the *compressed* output and correctly finds `` is -not valid JSON — because token-hashing wrote a non-JSON-safe placeholder into JSON content. -The pipeline then aborts and falls back to raw input. - -**Why this matters:** This is not a legitimate safety abort (unlike Issue 3 below) — it's -TokenDamper's own compression stage producing output that its own validation stage -necessarily rejects, for structured-data payloads. This will 0%-fail on **any** JSON-shaped -payload run through token-hashing. - -**Questions for Claude Code to investigate:** -- Does `compression:token-hashing` have any content-type awareness (JSON/code vs prose)? - If not, should it skip structured-data payloads, or use a placeholder format that - round-trips as valid JSON (e.g. a quoted string token instead of a bare `<...>` tag)? -- Is there a config flag to make the hashing stage JSON-safe, or does this need a code fix - in the hashing stage itself? - -**Update (re-verified) — co-occurs with Issue 3:** On `tool_output.json` and `session.json`, -the actual `fallbackReason` is not the JSON-AST error in isolation. The trace reports the -JSON-AST error *and* the Issue 3 semantic-drift breach (`0.60 > 0.40`) together, concatenated -in a single fallback reason. These are two failures co-occurring on the same payload, not two -independent single-cause failures on separate payloads. See Issue 3. - ---- - -## Issue 3 — Semantic drift fallback on code (plausibly legitimate, needs confirmation) -**Symptom:** On `codebase.py`, TokenDamper falls back with: -``` -Semantic drift metric (0.60) exceeds maximum threshold (0.40). -``` - -**Context:** `src/core/validation/index.ts` enforces drift ≤ 0.40 (see also lines ~2660, -2770, 2875, 2908 in the consolidated source). Headroom independently chose `router:noop` -on the same file (0% reduction), so both engines agree this file shouldn't be aggressively -compressed — this is weaker evidence of a bug and more likely correct conservative -behavior on source code. Worth confirming intent rather than treating as broken. - -**Update (re-verified) — the Headroom corroboration above is no longer supported:** On -re-run, Headroom did not independently choose `router:noop` on `codebase.py`. It instead hit -a 20-second `ContentRouter` single-cache-miss timeout and failed open to passthrough (the -`kompress` ML model was not downloaded/ready; `HEADROOM_DETECT_BACKEND=rust` was active for -this run, skipping the ML path entirely — see Issue 6). Same 0% output, different mechanism. -That is not an independent second opinion, and the reasoning above should not be cited as -settled on that basis. The drift abort on `codebase.py` may still be correct behavior — the -evidence for it just isn't there anymore. Needs a re-run with Headroom's ML backend actually -available (not `HEADROOM_DETECT_BACKEND=rust`) before this corroboration can be trusted again. - -**Update (re-verified) — not isolated to `codebase.py`:** the same `0.60 > 0.40` drift breach -also fires on `tool_output.json` and `session.json`, co-occurring there with the Issue 2 -JSON-AST error in a single combined `fallbackReason`. See Issue 2. - -**Update 2026-08-04 (4b.0) — on `codebase.py` this symptom was the harness, not the engine, -and it is gone.** The drift abort above was reproduced through `run_benchmark.py`, which piped -the file to `optimize -`. With no path the engine resolves no language, selects no elision -regions, falls to whole-item hashing, and `S_k` pins at the formula constant `0.60` — the -ceiling for code, not a measurement of what was lost. Handed the same bytes as a **file -argument**, `codebase.py` reduces **27.61%** (`cl100k`) with **no fallback** and drift `0`. - -So the question this issue was holding open — "is the drift abort on `codebase.py` correct -conservative behavior?" — was the wrong question. There was nothing conservative happening: the -engine had not looked at the file. **Issue 3 is closed for `codebase.py`.** - -It is **not** closed for `tool_output.json` and `session.json`. Both still fall back at `0.60` -on the path route, and the paragraph above about Headroom's timeout still stands — there is -still no independent corroboration for those two. Do not read this update as retiring Issue 3. - -**Update 2026-08-05 (4b.1) — the harness was not the only caller taking that route.** 4b.0 -fixed the benchmark's invocation; it did not give the engine a way to know what pathless -content is, and two of the three entry modes are pathless by construction. A caller can now -declare: `--language` / `--input-name` on the CLI, `language` / `path` on the MCP -`optimize_context` tool. Measured over frozen corpora in `cl100k`, the declared route matches -the file-argument route **byte-for-byte on all 109 files** (repo TypeScript 0.07% → 19.27%, -`pip` Python 0.02% → 12.34%). DECISIONS §29 and `docs/phase-4b-pathless-code-scope.md` §8. -This changes nothing for `tool_output.json` or `session.json`: both are JSON, both are already -classified as JSON without a declaration, and both fall back for reasons a language -declaration does not touch. - ---- - -## Issue 4 — Constraint-preservation correctly protected a planted "secret" (not a bug, confirms feature works) -**Symptom:** On `sample_logs.txt`, TokenDamper aborted (0% reduction) with a -`constraint-preservation` fallback after detecting an imperative-tagged line (a synthetic -`Secret Key: "BLUE-PANDA-992"` line planted specifically to test this). - -**Status:** This is TokenDamper working as intended — it refused to silently drop a line -it flagged as a hard constraint. Not a bug. (Note: `BLUE-PANDA-992` is a fake string -generated purely for this benchmark, not a real credential — no leak occurred.) - ---- - -## Issue 5 — Fallback path returns a different byte size than the original (-1.39%, unresolved) -**Symptom:** On `session.json`, TokenDamper's fallback shows **-1.39%** reduction (i.e. the -"compressed" output is *larger* than the original), not 0%. - -**Why this is odd:** A fallback is supposed to return the original payload unchanged -(0% reduction), not a modified/larger one. Something in the fallback path is -re-serializing, reformatting, or otherwise mutating the payload before emitting it. - -**Questions for Claude Code to investigate:** -- Diff the exact bytes: original `session.json` vs. the emitted fallback output. -- Check whether `fallback/index.ts` (or whatever renders `currentBundle` on fallback) is - re-stringifying JSON with different whitespace/key order, or appending metadata/trace - info to the emitted payload instead of returning the untouched raw input. - ---- - -## Issue 6 — Headroom's `target_ratio` is a soft hint, not an enforced budget -**Symptom:** Against a 30% target reduction, Headroom actually produced: -- `sample_logs.txt`: 18.15% -- `tool_output.json`: 34.36% -- `codebase.py`: 0.00% (`router:noop`) -- `session.json`: 87.92% - -None of these hit the 30% target — Headroom's heuristic engines (SmartCrusher, -CacheAligner, its `router:mixed:*` transforms) treat `target_ratio` as guidance, not a -hard constraint the way TokenDamper's `--target-reduction-ratio` is meant to be. - -**Why this matters for the benchmark:** Any headline claim like "Headroom achieves X% -reduction" needs the caveat that it isn't reliably steerable to a specified target — this -is the same kind of disclosure gap the original 0%-reduction investigation was trying to -surface for TokenDamper. - -**Note:** This run also used `HEADROOM_DETECT_BACKEND=rust` because of a Windows/ONNX -backend issue for Headroom's ML components — Headroom skipped its ML-based `kompress` -compression and only ran heuristic engines. Any future run should confirm whether the ML -backend changes these numbers, and note which backend was active in the report. - ---- - -## Open items / suggested next steps for Claude Code -1. Fix or work around the `` vs JSON-validator conflict (Issue 2) — this is - the highest-impact bug since it silently defeats TokenDamper on all structured-data - payloads. -2. Root-cause the -1.39% fallback size anomaly (Issue 5) with a byte-level diff. -3. Decide whether the `codebase.py` semantic-drift abort (Issue 3) is correct behavior or - an overly conservative threshold — may need a code-specific drift threshold rather than - one shared with prose/logs. -4. Re-test Headroom with the ONNX/ML backend working (not `HEADROOM_DETECT_BACKEND=rust`) - to see if reduction numbers or target-adherence change. -5. Consider whether Headroom exposes any *hard* budget/enforcement parameter (vs. the - soft `target_ratio` hint) and re-run with it if so, to get a genuinely comparable - "does it hit the target" number against TokenDamper. diff --git a/tools/corpus-harness/README.md b/tools/corpus-harness/README.md index 6d69858..5d4fb13 100644 --- a/tools/corpus-harness/README.md +++ b/tools/corpus-harness/README.md @@ -6,7 +6,7 @@ This exists because every reduction figure in this project is measured over file may also be editing, and because two separate measurements have already been wrong in ways nothing caught: the repo moved under a measurement (CLAUDE.md, Gotchas), and a 4b.3 A/B loop globbed one directory level and measured 132 of 144 files without noticing -(`docs/phase-4b-lever-disposition.md`, finding 3). +(`docs/phase-4b-lever-disposition.md`, finding 3). [retired] ## Use diff --git a/tools/corpus-harness/recipe.json b/tools/corpus-harness/recipe.json index 40e76fb..6eb64fa 100644 --- a/tools/corpus-harness/recipe.json +++ b/tools/corpus-harness/recipe.json @@ -1,5 +1,5 @@ { - "$comment": "Declarative corpus spec. `collect.js` reads this, walks each root recursively, and,selects deterministically (sort by relative path, take the first `limit`).,,Roots are machine-specific by nature — this corpus is third-party source found on a,developer machine, not vendored bytes. That is deliberate: vendoring GPL/BSD source into,this repo to measure a classifier would be a licensing problem for a measurement.,The recipe is the reproducible artifact; `manifest.json` records what a given machine,actually produced, so two runs on the same machine are comparable and a run on a,different machine is visibly a different corpus rather than silently one.,,`expect` is asserted after selection. It exists because the 4b.3 A/B loop globbed one,directory level and measured 132 of 144 files without noticing (see,docs/phase-4b-lever-disposition.md finding 3). A count that is not asserted is a count,that will be wrong eventually. | 2026-08-09: typescript 56->57->59 and prose 25->28. These buckets count this repository's own files, which grow; the harness refusing on the mismatch is it working. The 57->59 step is src/cli/ingest.ts and src/core/render/index.ts, added by audit H5. | 2026-08-10: typescript 59->60 and prose 28->29, during audit Wave 2. The typescript file is src/bench/fixtures/bundled-path.ts (audit M10); the prose file is docs/audit-remediation-status.md, which landed in 7a1b5a7 AFTER the dd540fe baseline was recorded, so that one was already outstanding. Both denominators moved, so Wave 2 aggregates are NOT directly comparable to the dd540fe table - compare per-file rows over the shared set instead.", + "$comment": "Declarative corpus spec. `collect.js` reads this, walks each root recursively, and,selects deterministically (sort by relative path, take the first `limit`).,,Roots are machine-specific by nature — this corpus is third-party source found on a,developer machine, not vendored bytes. That is deliberate: vendoring GPL/BSD source into,this repo to measure a classifier would be a licensing problem for a measurement.,The recipe is the reproducible artifact; `manifest.json` records what a given machine,actually produced, so two runs on the same machine are comparable and a run on a,different machine is visibly a different corpus rather than silently one.,,`expect` is asserted after selection. It exists because the 4b.3 A/B loop globbed one,directory level and measured 132 of 144 files without noticing (see,docs/phase-4b-lever-disposition.md finding 3). A count that is not asserted is a count,that will be wrong eventually. | 2026-08-09: typescript 56->57->59 and prose 25->28. These buckets count this repository's own files, which grow; the harness refusing on the mismatch is it working. The 57->59 step is src/cli/ingest.ts and src/core/render/index.ts, added by audit H5. | 2026-08-10: typescript 59->60 and prose 28->29, during audit Wave 2. The typescript file is src/bench/fixtures/bundled-path.ts (audit M10); the prose file is docs/audit-remediation-status.md, which landed in 7a1b5a7 AFTER the dd540fe baseline was recorded, so that one was already outstanding. Both denominators moved, so Wave 2 aggregates are NOT directly comparable to the dd540fe table - compare per-file rows over the shared set instead. | 2026-08-10, audit decisions H2/M1/M11: typescript 60->61 (src/core/validation/language-support.ts, H2) and prose 29->18 (M11 retired twelve narrative documents and added docs/retired-documents.md). The prose bucket is now 18 real documents rather than 29 — a smaller negative set for looksLikeMarkdown, and deliberately so: what it lost was superseded phase narrative, not a different kind of prose.", "buckets": [ { "name": "shell", @@ -96,7 +96,7 @@ "ts" ], "limit": 64, - "expect": 60, + "expect": 61, "why": "AST-lite COVERS this - the positive control, and the repo's own corpus. NOTE: 56, not the 64 quoted in docs/phase-4b-*.md - 8 of the 64 sources are under minBytes and are excluded here. Aggregates are not directly comparable to those documents." }, { @@ -109,7 +109,7 @@ "md" ], "limit": 40, - "expect": 29, + "expect": 18, "why": "real markdown - the negative set any looksLikeMarkdown change must not break. `.agents` is excluded below: it holds prior agent scratchpads, and because sort-then-take is alphabetical they filled all 40 slots on the first run and pushed out every hand-written document. Deterministic is not the same as representative." } ], diff --git a/tools/corpus-harness/seam2.js b/tools/corpus-harness/seam2.js index e289d14..9879bb9 100644 --- a/tools/corpus-harness/seam2.js +++ b/tools/corpus-harness/seam2.js @@ -7,7 +7,7 @@ * Usage: node tools/corpus-harness/seam2.js * * DECISIONS §32 named three seams for the hash-commented-code defect and - * docs/phase-4b-lever-disposition.md measured two of them dead. Seam 2 — tightening the + * docs/phase-4b-lever-disposition.md measured two of them dead. Seam 2 — tightening the [retired] * classifier — was never measured. This measures it. * * The known trap, from the disposition (§1): a *count* threshold on markers points the wrong