Audit H5: the knapsack gets something to solve

[ojassug]

Stacked on #6 (→ #5 → #4 → #3). Base is audit-h6, so this diff shows only the H5 commit. Merge the stack bottom-up; each retargets automatically.

Closes the ingestion half of audit H5. optimize now accepts multiple paths and directories.

The centerpiece could not run

ARCHITECTURE.md, README.md and CLAUDE.md all put a "Stateless 0/1 Knapsack Planner" at the centre of the design. None of it could execute:

• createContextBundle produces exactly one item
• applyCacheAwarePrefixLocking pins everything inside the first 1,024 tokens
• solve01Knapsack places pinned items outside the candidate set and always selects them

So item 0 was always pinned, itemsPruned was always 0, and planner/knapsack.ts, planner/cache-aware.ts, topology/topology-scorer.ts, topology/dependency-graph.ts and topology/git-inspector.ts could not affect any output the product was able to produce.

Measured after the change, on src/core at maxInputTokens: 4000: 31 items ingested, 15 pruned, 20,540 tokens saved by the planner alone.

Output format

test/unit/fallback-render.test.ts pinned the success path's items.join('\n') as a latent defect and said, in as many words, that whoever made an emittedOutput consumer multi-item "should stop and read it". This is that change, so the defect is fixed rather than inherited.

• One item renders as its content and nothing else — CLI, MCP and bench stay byte-identical.
• More than one renders with a ==> path <== header. That's head/tail's convention, chosen because it's one a reader already knows. It is not collision-proof and nothing escapes it: the consumer is a model being given context, legibility beats round-trip parseability, and anything needing to machine-parse should read finalBundle from the trace, which carries items structurally.
• Fail-open is per file — each file's original bytes, never a re-encoding of the decoded string, so DECISIONS §35 holds per item. What is not byte-identical is the stream as a whole, because the headers are TokenDamper's and were in no input file.

Directory walks are sorted (order is load-bearing: prefix locking pins the first ~1,024 tokens, so it decides what bypasses the knapsack) and skip node_modules, dist, .git and similar.

Three defects this exposed, fixed here

Each was invisible while every shipping bundle held one item.

#	defect	why it mattered
1	Pruning was scored as drift	findUnwitnessedItems always exempted pruned items — selection is not elision — but the ratios compared whole bundles, so a pruned item's symbols vanished and R_AST read the planner doing its job as semantic loss
2	Doomed whole-item elision was attempted	Since §40, S_k = 1 - R_AST for code, so destroying every symbol scores 1.0 against a gate firing above 0.40. No threshold or flag lets it through. Two pure-types.ts files (interfaces to lose, no bodies to elide) were sinking a 16-file batch
3	TD_PRESERVE: matched its own implementation	drift-tracker.ts (the regex literal) and cli/html-reporter.ts (the highlighter for that directive) each acquired a phantom content marker. R_struct is a bundle-scoped set, so one such marker being elided drove it to 0 and took a 16-file batch to S_k = 0.4053 — on a run whose real symbol retention was 99.1%

(3) also retires the html-reporter.ts regression #4 recorded as "left in deliberately".

Also fixed: envelope headers were counted on the output side only, so a multi-item fallback reported 72,973 → 73,667 tokens — a negative reduction, the same shape as the phantom −1.39% already diagnosed once in the Python bench harness (Issue 5).

Reviewer note — a defect I introduced and caught

Scoping drift to retained items made it silently blind when item ids don't correspond. id is content-derived at construction and preserved by the transforms, so it's safe in the pipeline — but a caller rebuilding its after bundle independently would get fresh ids, leaving nothing to compare and reporting S_k = 0 for a gutted bundle. That's precisely the "check that never ran" hazard invariant 10 exists for. It broke 5 tests immediately.

The guard: the exemption applies only when at least one id survives, which is the evidence ids are being carried rather than regenerated. With no correspondence the whole bundle is compared, as before — failing open to more measurement rather than less. Worth an eye.

Measured

Frozen 293-file corpus, 586 rows: 1 row changed, 0 regressions. TypeScript 27.33% → 29.55%.

Fallback counts drop sharply (prose 28 → 9, TypeScript over stdin 57 → 0) because items whose elision was doomed are no longer transformed-then-reverted. Same emitted bytes, less wasted work.

Not done — §3.1 is now the binding constraint

Multi-file runs still fall back on real corpora, and not for any reason this change can fix. On the 45-file Python corpus: drift 0.0359, AST clean, 169 KB elided — and it falls back, because 26 constraint failures across 14 items revert all 45. Validation is bundle-scoped and fallback is all-or-nothing (audit §3.1, Phase 1c, unstarted).

This PR delivers the mechanism; §3.1 stands between it and the outcome.

Phase 1c's stated prerequisite was attribution, and that now exists for the classes that matter: constraint failures name their item (#6), unwitnessed items name theirs (#3), AST issues carry itemId. Drift remains bundle-scoped and would need its own rule.

Verification

• npm run typecheck, npm run lint — clean
• npx vitest run — 525 passing, 60 files (up from 518)
• 5 new integration tests covering delimiters, single-file byte-identity, directory walking and ordering, per-file fail-open with a non-UTF-8 file, and the negative-reduction guard
• Four pinned tests asserted behaviour this changes and were updated with their findings preserved — three had comments explicitly describing what is now different

ARCHITECTURE.md is unchanged: multi-item bundles were always in the model (createBundleFromItems predates this) and nothing about the linear pipeline moved. What changed is that a shipping adapter finally builds one.

🤖 Generated with Claude Code

[ojassug]

Superseded: audit-h5 was merged into main directly (dd540fe), which brought in all three commits this PR carried — the gateway fix (6acaf30), H6 (579d503) and H5 (05bff65). GitHub rejected retargeting to main for exactly that reason: there are no new commits left between them.

Nothing is lost — all three verified as ancestors of main, and main is green: build, typecheck and lint clean, 525/525 tests passing.