Apply the per-query filter conditions a dataset query set carries

[generall]

The bug

A tar dataset's tests.jsonl pairs each query with the filter it was answered under:

{"query": [...], "conditions": {"and": [{"similarity": {"range": {"gt": 0.349}}}]},
 "closest_ids": [...], "closest_scores": [...]}

Those closest_ids are the ground truth for the filtered query. bfb read the query vector and the ids but dropped conditions — the field was explicitly excluded from QueryRow — so it searched unfiltered and scored the result against the answers to a different question.

On laion-small-clip, half of whose 5000 queries are conditioned, that reads as recall 0.63 where vector-db-benchmark reports 0.99. It affects every filtered dataset in the catalog: arxiv-titles-*-filters, h-and-m-*-filters, the random-*-filters family, random-768-100-tenants.

Exact search over the corpus isolates it — the approximation is not involved:

Query group	Recall@10
2470 unconditioned queries, unfiltered	0.994
2530 conditioned queries, condition ignored	0.296
2530 conditioned queries, condition applied	0.998

The change

Parse conditions into a Filter and apply each query's own. The dialect is the one vector-db-benchmark defines in engine/base_client/parser.py, so numbers stay comparable with its results:

conditions := { "and": [entry, ..], "or": [entry, ..] }   both optional
entry      := { field_name: { condition_type: criteria } }
condition  := "match" { "value": string|number|bool }
            | "range" { "lt"?, "gt"?, "lte"?, "gte"? }
            | "geo"   { "lat", "lon", "radius" }

and → must, or → should. One entry may name several fields and a field several condition types; each becomes its own condition, as in create_condition_subfilters.

Unknown operators, unknown condition types, an empty range and a geo missing a coordinate are errors, not silent drops — silently dropping a condition is exactly the bug being fixed, and it fails quietly as a plausible-looking recall number.

Conditions are parsed once when the query set is opened, so a malformed set fails at startup and the timed request path only clones a ready-made filter. When a dataset supplies a filter it wins over a filters: block on the same request; bfb prints how many queries carry conditions:

Dataset "laion-small-clip": 2530 of 5000 queries carry filter conditions; applying them (their ground truth assumes it).

Verification

A/B on one collection, same server, same hnsw_ef=128, only the binary differs:

Dataset	qdrant/bfb:dev	this branch
laion-small-clip (half conditioned)	0.632	0.995
laion-small-clip-no-filters-1 (conditions: {})	0.9969591	0.9969591

0.995 lands where the exact-search analysis predicted (0.996); the remainder is HNSW on filtered search. The unfiltered twin is identical to seven decimals, and no "carry filter conditions" line is printed for it — sets without conditions take the same path as before.

6 new unit tests cover the laion shape, and/or → must/should, geo, multi-bound ranges, multi-condition entries, and that each malformed shape errors. 161 tests pass; cargo fmt --check and clippy --all-targets clean.

Follow-up

benchmarks/dataset-search in benchmark-skills currently reports accuracy: null for filtered datasets because bfb could not measure them. That guard should be dropped once this lands and qdrant/bfb:dev is rebuilt — I will send that separately rather than have it go stale against the published image.

🤖 Generated with Claude Code