Skip to content

Reduce allocations in TypesenseVectorStore vector queries - #6707

Open
chabinhwang wants to merge 1 commit into
spring-projects:mainfrom
chabinhwang:typesense-vector-query-allocations
Open

Reduce allocations in TypesenseVectorStore vector queries#6707
chabinhwang wants to merge 1 commit into
spring-projects:mainfrom
chabinhwang:typesense-vector-query-allocations

Conversation

@chabinhwang

Copy link
Copy Markdown
Contributor

Summary

TypesenseVectorStore.doSimilaritySearch built the vector query by boxing every embedding value into a Stream<Float>, mapping each one to a String and joining an intermediate List.
This replaces that with a single pre-sized StringBuilder, so the query is written straight from the float[].

The generated query string is byte-identical to the previous one, so Typesense request behavior is unchanged.

Why

Query construction allocates proportionally to the embedding dimension, once per similarity search.
Measured allocated bytes per constructed query (ThreadMXBean.getThreadAllocatedBytes, values warmed up, embedding values uniform in [-1, 1]):

dimensions before after reduction
384 54,680 B 9,080 B 83%
768 108,744 B 17,944 B 83%
1536 216,864 B 35,728 B 84%
3072 433,144 B 71,256 B 84%

The removed garbage is one Float box and one String per dimension, plus the intermediate List and the StringJoiner buffer.
The buffer is pre-sized at 12 characters per value, which is what a typical embedding value plus its separator needs, so the builder does not have to grow. Under-sizing it (for example at 8) costs a resize and lands at roughly 54 KB for 1536 dimensions instead of 36 KB.

This is an allocation measurement, not a JMH benchmark, and the win is small relative to the HTTP round trip that follows.

Testing

TypesenseVectorStoreTests is new and pins the generated query format:

  • the documented format for ordinary values
  • the empty-embedding edge case
  • -0.0f, Float.MIN_VALUE and Float.MAX_VALUE, which exercise the sign and scientific-notation formatting
  • a 1536-dimension embedding, asserting that every value and separator is present

Validation run locally:

  • ./mvnw -pl vector-stores/spring-ai-typesense-store clean packageTests run: 33, Failures: 0, Errors: 0, checkstyle and spring-javaformat pass
  • Old and new construction compared over 20,106 inputs (including NaN, ±Infinity, -0.0f, subnormals, Float.MIN_VALUE/MAX_VALUE, random bit patterns and topK/threshold combinations) — identical output in every case

TypesenseVectorStoreIT needs a running Typesense instance and was not executed.

Notes

Supersedes #6151, which was opened against a much older main and never triaged. That branch is now stale; this one is rebased on current main, adds the extra format tests, and fixes the buffer capacity estimate that was too small to actually avoid the resize.

No associated issue.

Build the Typesense vector query with a single pre-sized StringBuilder
instead of boxing every embedding value into a Stream<Float>, mapping
each one to a String and joining an intermediate List. The generated
query string is unchanged, and new unit tests lock its format.

For a 1536-dimension embedding this lowers the allocation of building
one similarity search request from about 217 KB to about 36 KB.

Signed-off-by: chabinhwang <7chabin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants