MB-68044: Limiting Training Size of Faiss Indexes

[Likith101]

• Added nlist to params. This is only populated during index time
• Set the number of training vectors to number of centroids * 40

[Likith101]

Tested out recall of latency vs latency with limited training using the recall option as the ground truth

Latency => Recall@3: 93.40% Recall@10: 90.32% Recall@100: 85.22%
Latency LT => Recall@3: 92.52% Recall@10: 89.00% Recall@100: 84.03%

[Thejas-bhat]

please resolve the merge conflicts. can you just double check the recall numbers once? also it would be better to make this tunable via bleve/cbft maybe?

[Likith101]

please resolve the merge conflicts. can you just double check the recall numbers once? also it would be better to make this tunable via bleve/cbft maybe?

I am not sure if making it tunable helps much. Since a user needs to be very familiar with how this setting can affect things.

When I initially tested, recall numbers seemed to drop by 0.5 with accuracy being the same. Do you want a rerun of the test once again?

[Likith101]

Metric	With change	Without change
Recall@3	88.1%	87.7%
Accuracy@3	89.3%	89.3%
Index Build Time	1170.6	1581.7