Noise-robust overview: structured-noise stripping + homogeneous-sibling collapse

[Michaelliv]

Problem

napkin overview exists to prime an agent with the vault's actual vocabulary so lexical search works. On vaults containing large imported/converted document subtrees (OCR'd PDFs, DocuSign exports, HTML conversions), the overview drowns:

1. ID shrapnel wins TF-IDF. Dashed GUIDs (AB12CD34-5678-4EFA-…) slip past the existing [a-f0-9]{8,} filter because hyphens split them into short fragments ("dab", "bcf"). Each fragment is note-unique, so TF-IDF scores it as highly distinctive — the algorithm structurally rewards garbage.
2. Folder fans dominate the map. Hundreds of near-identical converted-document folders render one line each, burying the curated structure (on the worst benchmarked vault, ~230 of 242 lines were import folders).

Noise keywords aren't just wasted tokens — they prime the agent to search in noise terms.

Fix

Structured-noise stripping (always on). stripNoise now removes GUID/dashed-hex runs, digit-mixed ID blobs, HTML tags/entities, and long hex-letter runs before tokenization. A token-level "wordness" filter was tried and rejected — it killed real vocabulary (pdf, xlsx, fee). You can't judge a 3-letter token; you can recognize a GUID in context.

Homogeneous-sibling collapse (overview.collapse config, default true). Parents with ≥5 children whose mean pairwise term-distribution cosine ≥ 0.15 render as one aggregate row: contracts/ (+153 similar subfolders). Similarity is computed over body+heading terms only — shared naming conventions can't fake content homogeneity — and the vault root is never a collapse target (top-level folders are the taxonomy). --no-collapse restores the full listing; JSON gains an additive collapsedFolders field.

Warnings → stderr. Previously a single malformed-frontmatter note corrupted overview --json stdout.

Results (benchmark on 8 real-world agent vaults)

	worst-polluted vault	large mixed vault	4 clean vaults
overview lines	242 → 11	396 → 24	unchanged
noise keywords leaked	778 → 6	884 → 11	0 → 0
curated-keyword robustness*	0.71 → 0.82	0.78 → 0.92	1.0 → 1.0

* recall of the keywords the same algorithm produces when import subtrees are excluded — i.e. how well pollution is prevented from corrupting curated folders' keywords.

Interactions that shaped the design: collapse is useless without noise-stripping (ID shrapnel makes dump folders look different from each other, blocking collapse); note-level IDF was evaluated and rejected (marginal residual gain, measurable recall cost). The 0.15 threshold sits on a plateau — 0.125–0.175 produce identical collapse sets, and no curated folder collapses even at 0.10.

Notes

• No folder-name blacklists anywhere: anything statistically shaped like a repetitive document dump collapses; anything curated is untouchable by construction.
• CLI contract preserved: JSON shape extended additively; human output adds a (+N similar subfolders) marker.
• Tests: 358 pass, including new coverage for GUID stripping, collapse, root guard, heterogeneous-sibling preservation, and --no-collapse.