persist: stream HNSW node blobs in bounded windows; compact/persist progress logs - #46
Merged
Merged
Conversation
…rogress logs Full persist materialized every encoded node blob before writing — ~240 GB of transient heap at the production graph size, an OOM before the first byte lands. hnsw-compact wipes the on-disk graph before persisting, so that OOM would leave no graph on disk. PersistFrom now encodes and flushes in ~1 GiB windows (meta still written last, preserving the meta-lags-nodes crash invariant) and logs progress with rate and ETA on multi-window persists. Compact's write-locked phases log every 10M nodes so a supervised run is observable while /stats is blocked on the lock.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PersistFrommaterialized every encoded node blob before writing the firstbyte (
hnsw_persist.go:84-90) — the 1 GiB chunking insidePutVectorNodesBatchbounds the Pebble batch, not the caller's entriesslice. On the production graph that is ~240 GB of transient heap on top of
the ~250 GB live graph: a full persist cannot complete on the box, it
OOM-kills first. That makes
/admin/hnsw-compactdestructive as shipped —the handler durably wipes the
vfamily before persisting, so the OOMwould land in the window where the RAM copy is the only copy. (Full persist
has never completed at production scale; the on-disk graph was built by the
60 s incremental checkpoints. The shutdown final persist hits the same wall,
silently.)
Fix.
PersistFromnow encodes and flushes in ~1 GiB windows, releasingeach window before the next; peak extra heap is ~2 windows regardless of
graph size. Meta is still written once, after all windows — the
meta-lags-nodes crash invariant is unchanged.
Observability. Multi-window persists log progress (
hnsw persist: N/M nodes (X GiB, rate, eta)); single-window persists (the incrementalcheckpoints) stay silent, so no journal noise at the 60 s cadence.
Compact()'s write-locked scan/rewire phases log every 10M nodes — duringthat phase
/statsblocks on the same lock, so these lines are the onlyliveness signal a supervising operator has.