Reduce allocations in TypesenseVectorStore vector queries - #6707
Open
chabinhwang wants to merge 1 commit into
Open
Reduce allocations in TypesenseVectorStore vector queries#6707chabinhwang wants to merge 1 commit into
TypesenseVectorStore vector queries#6707chabinhwang wants to merge 1 commit into
Conversation
Build the Typesense vector query with a single pre-sized StringBuilder instead of boxing every embedding value into a Stream<Float>, mapping each one to a String and joining an intermediate List. The generated query string is unchanged, and new unit tests lock its format. For a 1536-dimension embedding this lowers the allocation of building one similarity search request from about 217 KB to about 36 KB. Signed-off-by: chabinhwang <7chabin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TypesenseVectorStore.doSimilaritySearchbuilt the vector query by boxing every embedding value into aStream<Float>, mapping each one to aStringand joining an intermediateList.This replaces that with a single pre-sized
StringBuilder, so the query is written straight from thefloat[].The generated query string is byte-identical to the previous one, so Typesense request behavior is unchanged.
Why
Query construction allocates proportionally to the embedding dimension, once per similarity search.
Measured allocated bytes per constructed query (
ThreadMXBean.getThreadAllocatedBytes, values warmed up, embedding values uniform in[-1, 1]):The removed garbage is one
Floatbox and oneStringper dimension, plus the intermediateListand theStringJoinerbuffer.The buffer is pre-sized at 12 characters per value, which is what a typical embedding value plus its separator needs, so the builder does not have to grow. Under-sizing it (for example at 8) costs a resize and lands at roughly 54 KB for 1536 dimensions instead of 36 KB.
This is an allocation measurement, not a JMH benchmark, and the win is small relative to the HTTP round trip that follows.
Testing
TypesenseVectorStoreTestsis new and pins the generated query format:-0.0f,Float.MIN_VALUEandFloat.MAX_VALUE, which exercise the sign and scientific-notation formattingValidation run locally:
./mvnw -pl vector-stores/spring-ai-typesense-store clean package—Tests run: 33, Failures: 0, Errors: 0, checkstyle andspring-javaformatpassNaN,±Infinity,-0.0f, subnormals,Float.MIN_VALUE/MAX_VALUE, random bit patterns andtopK/threshold combinations) — identical output in every caseTypesenseVectorStoreITneeds a running Typesense instance and was not executed.Notes
Supersedes #6151, which was opened against a much older
mainand never triaged. That branch is now stale; this one is rebased on currentmain, adds the extra format tests, and fixes the buffer capacity estimate that was too small to actually avoid the resize.No associated issue.