[draft] hnsw: URL→node index, durable zombie reclaim, persist-then-swap, async hnsw-compact

[andreimarinescu]

URL→node index (O(k) reclaim). HNSW keeps byURL map[string][]int32 of live node ids
(full-string keys sharing the node's string bytes; ~1 GB at 16M URLs). MarkURLPassagesInvalid
is O(k) instead of an O(N) scan under the write lock, LookupVectorByURL is O(1) and returns the
first live passage (previously the first match, which after an invalidation was a zombie's
empty vector — serve_search.go's callers only check ok), and valid/zombie counts are
maintained rather than walked (/stats no longer scans 73M nodes every 5 s).

Durable incremental persist. The 60 s checkpoint only wrote nodes[lastN:]; invalidations
and back-links on older nodes were lost on every restart until a full persist — so reclaim would
have silently undone itself at each restart (reconcile can't catch duplicates, the URL is still
live). Every node whose vector was cleared or whose neighbor list gained an entry is tracked in a
dirty set and rewritten by the next checkpoint (+~10 blobs per insert). Shutdown is one more
incremental checkpoint instead of the full persist that has never completed at production scale
(TimeoutStopSec=30). Persist encodes per ~1 GiB window under the read lock and writes outside
it, so writers stall for one window instead of the whole persist (which, via RWMutex writer
preference, would have stalled every new search for the duration of a full persist under live
crawl).

Persist-then-swap. Node blobs live in one of two slots ('v'+0x01, 'v'+0x02); a full persist
(hnsw-compact, hnsw-rebuild) writes the next generation into the inactive slot, points meta at
it (HSW2 = HSW1 + slot byte; HSW1 is still emitted while the graph sits in slot 0x01, so a store
that has never swapped stays readable by v0.2.x), then clears the old slot. This removes both the
no-graph-on-disk window (Op 1 wiped disk before writing) and the tombstone storm that made Op 1's
persist run at 12 MB/s for 5h49m: fresh writes never land under the DeleteRange. A crash mid-swap
leaves the old graph loadable; the stale slot is cleared after the next load.

Zombie-transit traversal + entry-point relocation. searchLayer traverses zombies but never
admits them to the ef window (they used to fill it with +Inf entries — the 08-26 "dense returns 0
candidates" failure), and invalidating the entry point relocates it to the highest-level live
node. Recall on an 80 %-zombie graph is now equal to clean.

Async /admin/hnsw-compact. Returns 202 and runs CompactPersist in the background (409 while
in flight, ?wait=1 for the old synchronous shape); phase / node counts / persist progress / ETA
in /stats.hnsw_compact and cosift_hnsw_compact_running. The weekly script polls that instead
of holding a connection (Op 1's response died on the ~6 h connection); threshold 30→15 %; the
box-only timer unit is now in deploy/systemd/. /admin/checkpoint and /admin/frontier-clear
lift the 60 s WriteTimeout like hnsw-compact already did.

COSIFT_PEBBLE_COMPACTIONS exposes Pebble MaxConcurrentCompactions (default 1, unchanged;
the single slot is the 128K-SSTable and persist-throughput cause on the box).

COSIFT_ZOMBIE_RECLAIM defaults to on (0/false/off disables); /stats.hnsw_reclaimed_total,
cosift_hnsw_reclaimed_total, cosift_hnsw_zombie_nodes.

[codecov]

Codecov Report

❌ Patch coverage is 69.86900% with 138 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/index/hnsw_compact.go	16.94%	49 Missing ⚠️
cmd/cosift/serve_setup.go	11.36%	37 Missing and 2 partials ⚠️
cmd/cosift/serve_admin.go	86.23%	12 Missing and 3 partials ⚠️
internal/store/pebble.go	71.87%	4 Missing and 5 partials ⚠️
cmd/cosift/hnsw_rebuild.go	0.00%	8 Missing ⚠️
internal/index/hnsw.go	91.04%	6 Missing ⚠️
internal/index/hnsw_persist.go	94.54%	4 Missing and 2 partials ⚠️
cmd/cosift/serve_crawl.go	0.00%	2 Missing ⚠️
cmd/cosift/serve_stats.go	88.88%	1 Missing and 1 partial ⚠️
internal/crawler/wet.go	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!