Skip to content

Fix sign flip during quantization - #85

Merged
gouyt13 merged 2 commits into
VectorDB-NTU:mainfrom
TamimEhsan:fix-sign-flip
Sep 2, 2026
Merged

Fix sign flip during quantization#85
gouyt13 merged 2 commits into
VectorDB-NTU:mainfrom
TamimEhsan:fix-sign-flip

Conversation

@TamimEhsan

Copy link
Copy Markdown
Member

The 1 bit code is quantized comparing positive vs negative or zero

x_u = (residual_arr > 0).template cast<int>();

However, during ex bit quantization, the signs are flipped under condition negative vs positive or zero.

// revert codes for negative dims
int32_t mask = (1 << ex_bits) - 1;
for (size_t j = 0; j < dim; ++j) {
if (res_arr.data()[j] < 0) {
TP tmp = ex_code[j];
ex_code[j] = (~tmp) & mask;
}
}

There's no change in the recall on the indices.

@gouyt13
gouyt13 merged commit 65b9533 into VectorDB-NTU:main Sep 2, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants