Skip to content

Audit C1b: a ratio that measured nothing does not vote, and a local variable is not a symbol - #4

Merged
ojassug merged 1 commit into
audit-wave-1from
audit-c1b
Aug 10, 2026
Merged

Audit C1b: a ratio that measured nothing does not vote, and a local variable is not a symbol#4
ojassug merged 1 commit into
audit-wave-1from
audit-c1b

Conversation

@ojassug

@ojassug ojassug commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #3. Base is audit-wave-1, so this diff shows only the C1b commit. Merge #3 first; this will retarget to main automatically.

Closes audit C1b and §3.2 — the second half of the R_struct defect, deferred from Wave 1 because it re-baselines published reduction figures and needed its own measurement pass.

It went differently than the audit specified, and the difference is the interesting part.

The audit's proposed fix is inert

max_audit.md §3.2 proposed computing R_struct over extractContentMarkers, excluding the filepath: marker that no content transform can destroy, and stated this "would fix both cases at once."

It fixes neither. Removing the only marker an item had leaves the before-set empty, and an empty set defaults R_struct back to 1.0 — the identical free 0.40, arriving by a different route. Measured: that change alone was byte-identical across all 586 rows of a frozen 293-file corpus.

The free 0.40 comes from the empty-set default, not from filepath:. That is DECISIONS §33's own argument — "0.0000 means 'retained everything' and 'found nothing to look at' indistinguishably" — applied to the score rather than to the gate.

Part 1: an unmeasured ratio is excluded, not defaulted

R_struct is computed over content markers, and the weight of a ratio whose before-set is empty is redistributed to the ratio that did measure something.

For code this makes S_k = 1 − R_AST, so the maximum symbol loss that can pass the 0.40 gate falls from 66.7% → 40%.

When neither ratio measured, retention returns 1.0 and stays silent — that case belongs to the measurement gate (§37, in #3). Having both refuse would attribute the refusal to the wrong question.

Part 2: extractSymbols counts semantic surface, not locals

Part 1 applied on its own costs 14 TypeScript files and 11.75pp (14.00% → 2.25%). The loss it was guarding against turned out to be almost entirely fictitious:

file symbols "lost" of which function-local
src/core/engine/index.ts 42 of 63 (66.7%) 41
src/core/hashing/tokenizer.ts 9 of 17 (52.9%) 9

Not one exported function, type or interface was lost in either case — selectElisionRegions retains signatures by construction. The rule matched const|let|var anywhere, so every const i, const result, const msg inside a function body counted as a semantic symbol on par with an exported function, and body elision is precisely the transform that removes them.

So the audit's "you can destroy two-thirds of every symbol in a file and pass" was measuring temporaries inside bodies the caller asked to have elided.

Python is the control, and the asymmetry is what identified the defect. Its extractor never had a locals rule, its measured symbol loss under the same elision is 0.0%, and it is untouched by either half of this change. A safety metric should not depend on which language's extractor happens to harvest block-scoped bindings.

Anchored with ^…/gm: in TS and JS a top-level declaration is a column-0 declaration; indented ones are body content.

Measured

Frozen 293-file corpus, 586 rows, both routes, via tools/corpus-harness:

arm TS files reducing TS saved Python saved
before (§37 only) 22 14.00% 14.98%
C1b alone 8 2.25% 14.98%
symbol fix alone 30 25.59% 14.98%
both (shipped) 29 23.38% 14.98%

The gate is stricter and reduction is higher (+9.38pp), because it now measures semantic loss instead of noise. Python, prose and every uncovered-language bucket are unchanged.

The one file it costs, and why it stays

src/cli/html-reporter.ts goes from reducing to falling back. Its sole content marker is directive:TD_PRESERVE:[^\s&]+)/g, — harvested from a regex literal in the file that implements syntax highlighting for that directive. Its own pattern source is its only structural evidence; eliding it drives R_struct to 0.

Left in deliberately: it fails conservatively (fallback, byte-identical output, no data loss), it is 1 file in 57, and special-casing it would over-fit the metric to one file in this repository. Recorded rather than patched.

Two hazard-pinning tests were updated

Both asserted properties this change deliberately abolishes. Findings preserved and supersession stated, not overwritten:

  • drift-tracker.test.ts pinned S_k === 0.60 as "the ceiling for code". That ceiling was the symptom, not a safety property — it existed precisely because R_struct voted 1.0 on evidence it never examined. Now 1.0, and the property is stronger: structMeasured is false, so no number of comments can move the score at all.
  • drift-unwitnessed-elision.test.ts (added in Audit Wave 1: silent data loss, Gateway byte corruption, and a trace that explains #3 for C1a) recorded that the retention gate could never fire for markdown. C1b makes that false — it now fires too. The measurement gate remains the load-bearing guard for the case retention cannot reach: a document with no content markers at all, which is now asserted separately.

Verification

  • npm run typecheck, npm run lint — clean
  • npx vitest run499 passing, 56 files (up from 493)
  • 6 new tests covering both halves, the 66.7% → 40% ceiling shift, and the weighted formula still applying when both ratios measured

Known residue, recorded not fixed

extractSymbols still harvests from comments — it is regex over raw content with no lexer. Measured artifacts: fn:of from the prose "pure function of its input", type:of from "that class of bug". After the locals fix this is the entire remaining symbol loss on src/core/engine/index.ts (1 of 22, 4.5%). Making extraction comment-aware is a per-language lexing problem and the measured cost does not yet justify it.

🤖 Generated with Claude Code

…variable is not a symbol

Closes audit C1b and §3.2. Two changes that only work together — landing
either alone is measurably worse than landing both.

The audit's proposed fix is inert. §3.2 proposed computing `R_struct` over
`extractContentMarkers`, excluding the `filepath:` marker that no content
transform can destroy, and stated it "would fix both cases at once". Measured, it
fixes neither: removing the only marker an item had leaves the before-set empty,
and an empty set defaults `R_struct` back to 1.0 — the identical free 0.40 by
another route. That change alone was byte-identical across all 586 rows of a
frozen 293-file corpus.

The free 0.40 comes from the empty-set default, not from `filepath:`. So an
unmeasured ratio is now excluded from the score rather than defaulted, with its
weight redistributed to the ratio that did measure something. For code
`S_k = 1 - R_AST`, and the maximum symbol loss that can pass the 0.40 gate falls
from 66.7% to 40%. When neither ratio measured, retention returns 1.0 and stays
silent — that case belongs to the measurement gate (§37), and having both refuse
would attribute the refusal to the wrong question. This is §33's argument applied
to the score rather than to the gate.

Applied on its own that cost 14 TypeScript files and 11.75pp, and the loss it was
guarding against turned out to be almost entirely fictitious. `extractSymbols`
matched `const|let|var` anywhere, so every `const i`, `const result`, `const msg`
inside a function body counted as a semantic symbol on par with an exported
function — and body elision is precisely the transform that removes them.
Measured at `targetReductionRatio: 0.5`:

  src/core/engine/index.ts       42 of 63 symbols "lost", 41 function-local
  src/core/hashing/tokenizer.ts   9 of 17 symbols "lost", all 9 function-local

Not one exported function, type or interface was lost in either case, because
`selectElisionRegions` retains signatures by construction. The audit's "you can
destroy two-thirds of every symbol in a file and pass" was measuring temporaries
inside bodies the caller asked to have elided.

Python is the control, and the asymmetry is what identified the defect: its
extractor never had a locals rule, its measured symbol loss under the same
elision is 0.0%, and it is untouched by either half. A safety metric should not
depend on which language's extractor happens to harvest block-scoped bindings.

Measured over the frozen corpus, 586 rows, both routes:

  arm                  TS files   TS saved   Python
  before (§37 only)          22     14.00%   14.98%
  C1b alone                   8      2.25%   14.98%
  symbol fix alone           30     25.59%   14.98%
  both (shipped)             29     23.38%   14.98%

The gate is stricter and reduction is higher, because it now measures semantic
loss instead of noise. Python, prose and every uncovered-language bucket are
unchanged.

One file regresses: `src/cli/html-reporter.ts`, whose sole content marker is
`TD_PRESERVE:` harvested from a regex literal in the file that implements syntax
highlighting for that directive. Its own pattern source is its only structural
evidence. Left in deliberately — it fails conservatively (byte-identical
fallback, no data loss), it is 1 file in 57, and special-casing it would over-fit
the metric to one file in this repository.

Two hazard-pinning tests asserted properties this abolishes, and were updated
with their findings preserved rather than overwritten: the 0.60 "ceiling for
code" (which was the symptom, not a safety property) and C1a's note that the
retention gate could never fire for markdown (it now does, and the measurement
gate remains the guard for the no-markers case).

Known residue, recorded not fixed: `extractSymbols` still harvests from comments,
because it is regex over raw content with no lexer — `fn:of` from "pure function
of its input", `type:of` from "that class of bug". After the locals fix that is
the entire remaining symbol loss on `src/core/engine/index.ts`, 1 of 22.

See DECISIONS §40.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant