feat(verify): stream suspects to disk via an external-sort sink (#156)

[elfensky]

Stream verify suspects to disk via an external-sort sink (#156)

verify accumulated every suspect in a list before writing, so peak RSS scaled with the finding count — the last part of the verify path whose footprint grew with the corpus. Issue #156 measured this at ~4 GB under high suspect count.

The change

SuspectSink (in verify/report.py) replaces the in-memory list with an external merge-sort — each full chunk is sorted and spilled to a temp file, write k-way-merges the runs, and a running Counter/hard/total tally feeds the summary. run_verify and run_orbit_pass stream into it instead of building lists (orbit outliers were the main offender). Peak memory is now one chunk regardless of suspect count. Mirrors the existing grouping.ExternalSorter.

Byte-identity preserved by construction

suspects.jsonl / summary.{json,md} bytes are unchanged:

• the per-suspect serialization is extracted into one shared primitive (_suspect_line) used by both the sink and the retained list renderers;
• summary.json stays sort_keys=True, summary.md sorts its tally, so both are input-order-independent;
• heapq.merge's key-tie ordering (earlier run first, then the in-memory tail) reproduces the exact stable add-order that sorted(key=_sort_key) gave the list path.

tests/test_verify_stream.py locks this: TestByteEquivalence asserts the streamed output equals the list renderers across the empty / no-spill / multi-spill paths (including a tie-key pair), and TestConstantMemory forces 10 spills at chunk_size=10 and asserts the buffer never exceeds one chunk.

Validation

• 787 tests pass; ruff check + ruff format --check clean.
• Capstone at scale: verify --orbit over the full cleaned corpus (232,281,894 records / ~30 GB) held peak RSS flat at 181 MB (dedup, already streamed, 144 MB). 88,041 suspects found, byte-deterministic output. (This run's suspect count sat under the 200K chunk, so the in-memory tail path was exercised; the spill/merge path is covered by the unit tests above — a --all orbit run would exercise spills on real data.)

Closes #156.

🤖 Generated with Claude Code