Phase C, the Node 18 CI failure, and an audit-gated roadmap - #1
Merged
Conversation
CONTENT_TYPE_VALIDATORS.code mapped to the TypeScript validator as a historical default for
the ~19 extensions isCodeExtension covers, of which the AST-lite suite implements three. That
meant every language outside the TS family was lexed by a scanner written for a different
grammar. It is not a weaker check; it is a check that invents findings.
Measured false AST_UNTERMINATED_STRING / AST_UNBALANCED_BRACKET verdicts: perl 39 of 40, tcl
30 of 40, shell 22 of 40, powershell 7 of 40. c and css score 0 of 50, because the C-family
scanner happens to fit. Shell and PowerShell are in isCodeExtension today, so those were live
false verdicts on the file route - $'...' quoting, ' inside comments, ${...} expansion and
[[ ... ]] tests are ordinary syntax a TS lexer reads as unterminated or unbalanced. This is
DECISIONS §17's finding, that a verdict decided by apostrophe parity is not validating
anything, reached from the extension side instead of the fence side. Removed for the same
reason rather than tuned.
Nothing genuinely covered loses coverage. TypeScript, JavaScript, Python and JSON all reach
their validator through the language and path branches of selectValidator, which run first
and are unchanged. The remainder now report validated:false and appear on trace.astCoverage,
which is §23's distinction - an unexamined item is not a passing one.
Two tests asserted the old mapping deliberately, as recorded traps. The trap did not
disappear; it changed shape, from *wrong* validation to *absent* validation, so both pins were
re-aimed rather than deleted. declared-language and bench/evaluator now assert null where they
asserted 'typescript', and evaluator additionally pins item.path as undefined - those fixtures
pass the path as origin, which is why the extension arm never rescues them and why `language`
remains the only thing standing between a benchmark fixture and an unchecked item.
Five stale claims moved with it, not the three the audit listed: two doc comments in
constructors.ts, phase-4b-pathless-code-scope §4b.2 and §6.3, CHANGELOG's Phase 4b.1 entry,
and DECISIONS §29. The three historical records are annotated as superseded rather than
rewritten. max_audit.md quotes the old comment too and is left alone, because it is correct as
history.
typecheck, lint, build clean; 481/481 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ting CI runs the matrix [18.x, 20.x, 22.x] and package.json claims engines >=18.0.0, but 13 packages in the installed tree require ^20.19.0 || ^22.13.0 || >=24. The 18.x leg has been failing for two independent reasons, neither of them a regression in any commit that ran it. Lint dies on the eslint 10 stack: eslint@10.8.0, @eslint/core, @eslint/config-array, @eslint/config-helpers, @eslint/object-schema, @eslint/plugin-kit, espree@11, eslint-scope@9, eslint-visitor-keys@5, brace-expansion@5. Test dies separately on vite@7.3.6, reached through vitest. vitest's own engines field advertises ^18.0.0 and is misleading - its dependency tree does not honour it, so the leg would fail even with lint removed. Scanned per leg: 18.20.8 has 13 incompatible packages, 20.19.5 has 0, 22.21.1 has 0, 24.14.0 has 0, 26.4.0 has 0. So only the 18 leg was ever red, and the fix is to stop testing a version nothing supports rather than to pin the toolchain backwards. Node 18 reached end of life on 2025-04-30. engines is set to ^20.19.0 || ^22.13.0 || >=24 rather than >=20.19.0, because that is the actual intersection: eslint excludes 21 and 23, and brace-expansion excludes 21. The lockfile carries the same string; the refresh touched one line and zero version/resolved/integrity entries. .npmrc sets engine-strict so this fails at npm ci with EBADENGINE instead of three steps later at Lint. That is the part that made the diagnosis expensive: npm only warns by default, the install succeeds, and the error surfaces somewhere that does not name the cause. npm never publishes .npmrc, so it constrains contributors and not consumers. Verified end to end on a clean f93c385 worktree with the fix applied: npm ci, typecheck, lint, build all clean, 481/481 tests. Incidentally closes a latent runtime bug the old floor was already wrong about: src/gateway/proxy.ts calls AbortSignal.any, which needs >=18.17.0, so >=18.0.0 was never true. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… to its directory CLAUDE.md was restating things a reader can get faster from source: the npm script list, the full optimize flag list, six architecture table rows naming directories by their contents, the MCP tool and resource names, and the TypeScript/Prettier/ESLint conventions line. All of that drifts silently against package.json, SUPPORTED_FLAGS and TOOL_DEFINITIONS, and none of it is a decision - it is a directory listing with extra steps. 33 lines out, 1 in. The two benchmark gotchas are not derivable and were not deleted. They move to a new directory-scoped tokendamper-benchmark/CLAUDE.md, which loads when working in that directory: that TokenDamper is timed through a Node process spawn while Headroom is an in-process Python call, so the latency numbers are not apples-to-apples, and that Headroom's target_ratio is a soft hint rather than an enforced budget. Both are traps for someone reading BENCHMARK_RESULTS.md, and both belong next to the harness rather than in a root file that a TypeScript-only session also pays for. This is the M11 direction from the audit applied to one file - the invariants and measured baselines earn their place, the restatements do not - not a full docs retirement. These edits predate this session and were sitting uncommitted in the working tree; they are committed here as their own unit so the Phase C and CI changes stay separable from them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s their preconditions ROADMAP.md scheduled BM25 scoring, MMR, AST folding and Prometheus metrics on top of a pipeline the 2026-08-07 audit measured at 14.04% on its own corpus, 0.0% on its own benchmark, and 0 bytes on the Gateway. CLAUDE.md has said "do this before roadmap feature work" for some time without the roadmap reflecting it. It does now. The gate is not a priority note, it is a precondition. v1.2.0's BM25 scorer and MMR pass, v1.3.0's cache_control placement and Milestone 8 all build on solve01Knapsack and scoreBundleTopology, and those are unreachable on every shipping path: createContextBundle emits a one-item bundle for CLI, MCP and bench, prefix locking pins item 0, and the solver always selects it. Building on them produces correct code with no observable effect, which is the condition the audit already found in ~1,000 existing lines. Inserted a blocking v1.1.x track - v1.1.1 green tree, v1.1.2 data loss, v1.1.3 honest instruments - and a Scope Decision Gate carrying the four questions whose answers determine whether the later releases are buildable at all. Numbering of v1.2.0 through v2.0.0 is deliberately unchanged so existing cross-references stay valid; the remediation work is versioned as patch releases instead of renumbering the chain. Three items absent from the audit's own recommended order are promoted into the track: M8+M9, a credential echo one env var from being live; M5a, roughly ten lines of schema that convert the whole MCP mode from guaranteed no-op to functional; and M10, bench throwing for every installed user. M2 moves from the audit's fourth position to first, because a red tree makes every later fix unverifiable. v1.3.0 is re-scoped rather than gated, and that correction came from reading source rather than from the audit. Its "Declaration Boundary Detector" is scheduled as new work, but FUNCTION_HEADER and CONTROL_FLOW_HEADER in elision/regions.ts already implement that discriminator and selectElisionRegions already folds bodies on TS, JS and Python. The roadmap was scheduling something that shipped - the same class of error its own appendix records once already. Milestone 9 is flagged for re-derivation: its proposed w_atom term would add a third weight to a formula whose R_struct term is a pinned constant for code, and its TD_PRESERVE tracking formalizes CONSTRAINT_DIRECTIVE_LOST, currently the largest single cause of 0% reduction. Added a measured starting position table, because every benchmark target in the document was stated without the number it improves on - per the audit's own §3.3, a target with no baseline has the same shape as a green check that never ran. max_audit.md is committed with it. ROADMAP.md now cites it as the evidence base throughout, and a roadmap citing a file that is not in the repository is the failure mode this project cares most about. Drop it with git rm if it was meant to stay a scratch artifact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…true
Closes H3, M3 and M4a from `max_audit.md`. No source code changes: tests,
fixtures, docs and `package.json` only, so `dist/` is unaffected.
H3 — the regression suite guarded a set the product does not ship.
`tokendamper bench` printed 0.0% reduction at a 40% fallback rate while the
suite was green. The two facts never met because no test ran the combined
fixtures: Tests 1-5 loaded `loadBenchmarkFixtures('humaneval')` — the half
that structurally cannot fall back — and Test 2 built a private two-fixture
set inline under an artificial `maxInputTokens: 50` and asserted 40% against
that. codexglue, at 0.80 fallback, was never run at all.
All five now load the shipped combined set, and `baseline.json` records
measured truth rather than a target: `minTokenReductionRatio` 0.40 -> 0.0,
`maxFallbackRate` 0.0 -> 0.40. The old values move to an `aspirational` block
so the gap stays visible instead of being deleted.
The assertions use equality, not `>=`. A `>=` check against a measured floor
of 0.0 is vacuously true and can never fail, which is the same defect one
level up — invariant 10 applied to the guardrail rather than to the engine.
Equality means an improvement breaks the suite too and has to be recorded
deliberately. Verified able to fail: mutating the recorded numbers turns 3 of
6 tests red.
Two things the measurement surfaced that were not written down anywhere.
Every non-fallback fixture also reduces exactly 0%, because `BenchmarkRunner`
supplies a `TokenHasher` and the engine rehydrates what it elided — so the
shipped set produces no reduction by two independent mechanisms, not one. And
`syntaxPassRate: 1.0` is now asserted alongside the 40% fallback rate, since
syntax is evaluated on emitted output and emitted output on fallback is the
input; a metric that reads 1.0 whenever the engine does nothing cannot
distinguish success from inaction, and pairing them keeps that legible.
Test 3 keeps the `aba84df` finding intact, rescoped to humaneval. Its claim is
still true; what was wrong was its scope, so it is narrowed rather than
removed.
M3 — `package.json` declared MIT while `LICENSE` grants MPL-2.0 and the README
said so. `package.json` is publishable (`"private": false`, `files`,
`prepublishOnly`) and npm treats its `license` field as authoritative, so
scanners read MIT — permissive — and receive MPL-2.0 copyleft. That error runs
in the harmful direction: it understates the obligations a consumer takes on.
`package.json` and `CLAUDE.md` now match `LICENSE`. "All rights reserved" is
dropped from the README copyright notice, where it sat directly above an
open-source grant and asserted its opposite. Recorded as DECISIONS §36, which
also notes the MIT -> MPL migration itself was never recorded.
M4a — the README's Gateway warning outlived the fix it described. It claimed
the Gateway "bypasses TokenDamper's validation pipeline" and that
`fallbackUsed` is "hardcoded `false`"; both have been untrue since Phase 1.0b
(`proxy.ts` imports and calls `core/engine.optimize`, and `fallbackUsed` is
`result.fallbackUsed`). It is replaced rather than deleted: removing a false
warning while leaving "0/1 Knapsack Planning", "Cross-turn Session
Deduplication" and "Reversible Token Hashing" uncaveated would have made the
page less accurate, not more. The notice now carries the measured status —
cross-turn dedup saves 0 bytes on ordinary traffic (H1), `tokendamper exec`
returns 401 to its own child (C3), non-ASCII bodies can be corrupted at the
socket (C2). Qualifying those feature claims properly is Wave 3, after the
Gateway question is decided.
Typecheck clean, lint clean, 481/481 tests passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four independent changes on this branch. They are separable commits on purpose — the CI fix can be reverted without losing Phase C.
2cbdb45— fix(phase-C):codeis a family, not a languageCONTENT_TYPE_VALIDATORS.codemapped to the TypeScript validator as a historical default for the ~19 extensionsisCodeExtensioncovers, of which the AST-lite suite implements three. Every language outside the TS family was lexed by a scanner written for a different grammar — which is not a weaker check, it is a check that invents findings:AST_UNTERMINATED_STRING/AST_UNBALANCED_BRACKETShell and PowerShell are in
isCodeExtensiontoday, so those were live false verdicts on the file route. This is DECISIONS §17's finding — a verdict decided by apostrophe parity is not validating anything — reached from the extension side instead of the fence side.Nothing genuinely covered loses coverage: TS, JS, Python and JSON all reach their validator through the
languageandpathbranches ofselectValidator, which run first and are unchanged. The remainder now reportvalidated: falseand appear ontrace.astCoverage— §23's distinction that an unexamined item is not a passing one.Two hazard-pinning tests were re-aimed, not deleted. They asserted the old mapping deliberately, as recorded traps. The trap did not disappear; it changed shape from wrong validation to absent validation, so both now assert
nullwhere they asserted'typescript'.Five stale claims moved with it — not the three previously identified. Two doc comments in
constructors.ts,phase-4b-pathless-code-scope§4b.2 and §6.3,CHANGELOG's Phase 4b.1 entry, andDECISIONS§29. Historical records are annotated as superseded rather than rewritten.46b5ff9— fix(ci): the Node 18 leg tests a version the toolchain stopped supportingThis is the CI failure, and it is not a regression in any commit that ran it.
package.jsonclaimedengines >=18.0.0and the matrix tested18.x, but 13 packages in the tree require^20.19.0 || ^22.13.0 || >=24. The leg failed at two independent steps:eslint@10.8.0,@eslint/core,espree@11,eslint-scope@9,eslint-visitor-keys@5,@eslint/config-array,@eslint/plugin-kit,brace-expansion@5)vitest@3.2.7→vite@7.3.6.vitest's ownenginesadvertises^18.0.0and is misleading; its dependency tree does not honour it, so the leg failed even with lint removed.Scanned per leg: 18.20.8 → 13 incompatible; 20.19.5 → 0; 22.21.1 → 0; 24.14.0 → 0; 26.4.0 → 0. Only the 18 leg was ever red. Node 18 reached EOL 2025-04-30, so the fix is to stop testing a version nothing supports rather than pin the toolchain backwards.
enginesis set to the actual intersection (^20.19.0 || ^22.13.0 || >=24, not>=20.19.0) because eslint excludes 21 and 23 andbrace-expansionexcludes 21. The lockfile refresh touched one line, zeroversion/resolved/integrityentries..npmrcaddsengine-strictso this fails atnpm ciwithEBADENGINEinstead of three steps later at Lint — that delay is what made the diagnosis expensive. npm never publishes.npmrc, so it constrains contributors, not consumers.Incidentally closes a latent bug the old floor was already wrong about:
gateway/proxy.tscallsAbortSignal.any, which needs>=18.17.0, so>=18.0.0was never true.929b3d2— docs(claude): trim CLAUDE.md, scope the rest to its directory33 lines out, 1 in. Removed restatements that drift silently against
package.json,SUPPORTED_FLAGSandTOOL_DEFINITIONS. The two benchmark gotchas were moved, not deleted, into a new directory-scopedtokendamper-benchmark/CLAUDE.md. These edits predate the session and are committed as their own unit so the code changes stay separable.0f7dbb9— docs(roadmap): gate feature releases on the auditThe roadmap scheduled BM25, MMR, AST folding and Prometheus metrics on top of a pipeline measured at 14.04% on its own corpus, 0.0% on its own benchmark, and 0 bytes on the Gateway. The gate is a precondition, not a priority note: v1.2.0's scorer/MMR, v1.3.0's
cache_controland Milestone 8 all build onsolve01Knapsack, which is unreachable on every shipping path (one-item bundles, prefix locking pins item 0, solver always selects it).Adds a blocking v1.1.x track plus a Scope Decision Gate. Release numbering v1.2.0–v2.0.0 is unchanged so cross-references stay valid.
One correction came from reading source, not the audit: v1.3.0's "Declaration Boundary Detector" is scheduled as new work, but
FUNCTION_HEADER+CONTROL_FLOW_HEADERinelision/regions.tsalready implement it. Re-scoped rather than gated.Verification
npm ci→typecheck→lint→build→test, run on a cleanf93c385worktree with the fix applied, and again against this branch's HEAD:[20.x, 22.x, 24.x]matrix. The branch push did not trigger CI (ci.ymlfires only on push tomain/ PR tomain), so the matrix change proves itself here for the first time. The per-leg dependency scan above is deterministic, but it is not the same as a green run.Reviewer notes
max_audit.mdis now tracked (812 lines). Included becauseROADMAP.mdcites it as its evidence base throughout, and a roadmap referencing a file absent from the repo is the exact failure mode this project guards against.git rm --cachedif it was meant to stay scratch.DECISIONS.mdentry of its own — recorded as a supersession note on §29 and a CHANGELOG bullet. A §36 is the remaining piece.🤖 Generated with Claude Code