Audit Wave 1: silent data loss, Gateway byte corruption, and a trace that explains

[ojassug]

Supersedes #2, which was auto-closed when its head branch was renamed phase-c-code-is-not-typescript → audit-wave-1. Same three commits, same diff; no review had happened on #2.

Wave 1 of the max_audit.md remediation, following Wave 0 in #1. Three independent fixes, each verified against the unfixed code before landing.

C1a — a document whose witnesses were all destroyed is no longer certified

ab24b96 · audit C1 (measurement half) · DECISIONS §37

DriftTracker.findUnwitnessedItems built its probe bundle from the before item, so it asked "did evidence exist?" rather than "did any of it survive?". An item whose witnesses were entirely destroyed was exempt on the grounds that they had once been there.

Measured on this repository's own files, on both the file and stdin routes:

file	before	after	fallbackUsed	S_k
CODE_OF_CONDUCT.md	3,542 B	72 B	false	0.369 / 0.400
SECURITY.md	1,154 B	72 B	false	0.333 / 0.400

validation.passed: true, both gates reporting pass, and unrecoverable on the CLI — no TokenHasher is supplied, so the removed bytes exist nowhere.

Neither gate could fire, and the arithmetic is closed-form. Prose yields no symbols, so R_AST = 1.0 as an empty-set default contributes a free 0.60. filepath: is derived from item.path and no content transform can destroy it, so R_struct = 1/(N+1) for N headings. Therefore S_k = 0.4·N/(N+1) — which approaches 0.40 from below and never reaches it, for any N, against a gate firing on > 0.40. The two stdin rows landed on exactly 0.400 and were admitted by the strict comparison: the supremum of the expression being waved through by the operator, not a near miss.

An item that changed is now refused when it yields no symbols and no content-derived markers survive in the after item. Two properties keep this safe:

• Scoped to symbol-free items. Whole-item elision of code still refuses as SEMANTIC_DRIFT_EXCEEDED — the accurate reason, since R_AST measured that loss exactly — rather than being relabelled unmeasurable.
• Strictly additive. Refusing on the surviving set is stronger than refusing on the before set, so every §33 refusal still refuses. Nothing caught is now let through.

Measured cost over a frozen 293-file corpus (586 rows, both routes): 4 rows changed, all four this defect. Everything else byte-identical — TypeScript 14.00%, Python 14.98%, uncovered buckets 0.00%, all unchanged. Prose goes 0.67% → 0.00%, which was the loss.

C2 — the Gateway reads bytes, not string fragments

99672af · audit C2, L3 · DECISIONS §38

GatewayServer.onRequest accumulated its body with body += chunk, calling toString('utf8') on each chunk independently. A multi-byte UTF-8 sequence straddling a chunk boundary became U+FFFD on both sides — at the socket, before the pipeline exists, on the bytes forwarded upstream.

Measured against the unfixed server, each body written as two req.write() calls split on a continuation byte:

body	sent	forwarded
héllo — ünïcode ✓ 日本語 😀	94 B	98 B — h��llo …
こんにちは世界	76 B	82 B
┌─┐│ build ok │└─┘	89 B	95 B

A corrupted body is always longer, because U+FFFD re-encodes to three bytes.

This is DECISIONS §35 at a different seam — Phase B applied that reasoning to the adapter reading from disk and not the one reading from a socket, where it's worse because the bytes reach a provider rather than a terminal. MCP was never affected, instructively: setEncoding('utf8') installs a StringDecoder which holds partial sequences across chunks; manual concatenation is exactly what bypasses it.

Now collects Buffer[], concatenates on end, decodes once. Bodies failing a UTF-8 round trip — which correct concatenation does not fix — are forwarded verbatim rather than optimized (every stage would reason about content the caller never sent) or rejected (not a transparent proxy's call). Also removes the O(n²) body-size check (L3).

M6 — the trace carries what the stages computed

b59722a · audit M6 · DECISIONS §39

buildTrace projected every StageResult down to { stageId, status, durationMs: 0, changed }, discarding metrics and notes and hardcoding the duration. A CLI trace now shows regionsHashed: 4, bytesSaved: 14509, irreversibleElisions: 1 and the note explaining that no token hasher was supplied so the removed content is retained nowhere.

durationMs is measured by the engine, not the stage — a stage that read a clock would stop being a pure function of its input (invariant 1), whereas timing an opaque call from outside cannot change what it returns. performance.now() because most stages finish inside a millisecond.

The pruner's note was not vague, it was false. pruning:topology-pruner returned "All items fit within token budget; no pruning required." unconditionally whenever itemsPruned === 0. Measured, a 5,405-token file at maxInputTokens: 10 reported that all items fit.

The mechanism is also H5: prefix locking pins everything in the first 1,024 tokens, the knapsack always selects pinned items, and createContextBundle builds a one-item bundle — so item 0 is always pinned and itemsPruned is always 0. The note announced pruning was unnecessary where it was impossible. It now names the mechanism, with bundleTokens and maxTokens in the metrics so the claim is checkable. This does not fix H5, but it stops the trace concealing it.

Verification

• npm run typecheck, npm run lint — clean
• npx vitest run — 493 passing, 56 files (up from 481/54)
• Every new test run against the unfixed code first: 2/2, 4/4, 4/4 red
• C1a's suite also contains three tests pinning behavior the fix must not change; those pass in both arms
• C1a's corpus delta produced via tools/corpus-harness — frozen, hash-verified, both routes

Also updates tools/corpus-harness/recipe.json (typescript 56→57, prose 25→28). Those buckets count this repo's own files, which grow; the harness refusing to run on the mismatch is it working as designed.

Deliberately deferred

C1b — filepath: is still counted in R_struct. That is why R_struct is pinned at 1.0 for code and contributes a free 0.40, and therefore why a code file can still lose 66.7% of its symbols and pass. It re-baselines every published reduction figure in the project and wants its own measurement pass.

Still open from the audit: C3 (exec 401s its own child), C4 (structured content flattened to a string), H1 (Gateway saves 0 bytes), H2, H4, H5, H6, and the remaining M-tier findings.

🤖 Generated with Claude Code