docs: record the two vector invariants - #211
Merged
Merged
Conversation
Both are paired-update traps of the kind this section already lists, and both were learned the expensive way this week. ChunkEmbedText and chunkReuseKey must stay in sync. The key lets an unchanged chunk keep its vector across re-ingest, and it is sound only while it covers everything the embedder sees. Add a field to the embed header without adding it to the key and re-ingest hands back a vector for text the model never saw in that form -- no error, just a wrong vector that looks right. The reverse mistake only stops reuse working, which is merely wasteful. A new vector column means updating both clear paths, clearVectors and ResetEmbeddings. Missing one leaves that corpus vectorized under the previous embedder while the rest is rebuilt: two vector spaces in one store, and a ranking that degrades silently. That is not hypothetical -- document chunks got vectors in #208 and neither path was updated until #210 caught it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two paired-update traps, both learned the expensive way this week, recorded alongside the ones already in that section.
ChunkEmbedTextandchunkReuseKeymust stay in sync. The reuse key added in #210 lets an unchanged chunk keep its vector across re-ingest, and it is sound only while it covers everything the embedder actually sees. The asymmetry is why it is worth writing down: adding a field to the embed header without adding it to the key produces a wrong vector that looks right -- no error, no failing test, retrieval quietly keyed on text the model never saw in that form. The reverse mistake only stops reuse working, which is merely wasteful. One direction is a silent correctness bug and the other is a performance regression, and whoever edits the header next should know which is which.A new vector column means updating both clear paths,
clearVectorsinpgstore/store.goandResetEmbeddingsinpgstore/reset.go. Missing one leaves that corpus vectorized under the previous embedder while the rest is rebuilt: two vector spaces in one store, and a ranking that degrades without announcing itself. Not hypothetical -- document chunks got vectors in #208 and neither path was updated until #210 caught it.ResetEmbeddingsalso reports a count, so the count has to cover the new column or it under-reports what it did.Docs only.
🤖 Generated with Claude Code