Skip to content

Direct mapped visited set imp - #63

Merged
gouyt13 merged 6 commits into
VectorDB-NTU:mainfrom
TamimEhsan:direct-map-visited-set
Aug 22, 2026
Merged

Direct mapped visited set imp#63
gouyt13 merged 6 commits into
VectorDB-NTU:mainfrom
TamimEhsan:direct-map-visited-set

Conversation

@TamimEhsan

@TamimEhsan TamimEhsan commented Aug 19, 2026

Copy link
Copy Markdown
Member

This PR adds a new epoch based visited set implementation and uses compile time abstraction for the visited set. The default implementation is still the HashBasedBooleanSet and the implementation stays the same exactly. So, no regression introduced.

Why the new Epoch Based Set

The hash based set due to its use of unordered set needs dynamic memory allocation and deallocation. So, that contributes to the whole QPS. For higher topk or EF this impacts the overall runtime. So, an alternative implementation is added. The choice is still hardcoded in visited_set.hpp. That's a task for the future.

Performance

Dataset: GIST10M, Topk = 10, quantization = 9 bit

EF Recall HashBased QPS (Old) EpochBased QPS (New) Multiplier (New/Old)
10 0.4913 12735.1 12259.8 0.96x
20 0.6230 9776.5 9515.3 0.97x
40 0.7564 6387.2 6603.5 1.03x
80 0.8495 4469.7 4559.3 1.02x
120 0.8920 3690.2 4161.5 1.13x
200 0.9338 2179.1 2674.8 1.23x
400 0.9671 1238.6 1667.0 1.35x
600 0.9781 870.1 1255.4 1.44x
800 0.9830 665.8 1016.4 1.53x
1000 0.9858 531.0 833.4 1.57x
1500 0.9892 367.2 614.4 1.67x
2000 0.9899 272.5 459.2 1.69x

@TamimEhsan
TamimEhsan marked this pull request as draft August 19, 2026 04:36
@TamimEhsan
TamimEhsan marked this pull request as ready for review August 19, 2026 04:50
@gouyt13
gouyt13 merged commit 37df21d into VectorDB-NTU:main Aug 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants