Verify shard registration of external indexes on legacy binaries - #11
Merged
Conversation
added 2 commits
August 7, 2026 12:21
On the legacy /api/v1 line, POST /indexes can return 2xx from the metadata
layer while shard-level registration fails asynchronously; the reconciler
then retries the rejected config forever and every vector write fails with
"index not found". The two known legacy behaviours are opposites: older
builds require the field ("field or template must be specified"), v0.1.3
rejects it ("external embeddings index config cannot specify field or
template") -- so no fixed variant ordering can be correct.
After each 2xx on the legacy path, poll the index status until every shard
reports a non-null registration; if it stays broken within the bounded
window, delete the index and try the next variant/type. The non-legacy path
accepts on 2xx exactly as before.
Verified end to end against the published v0.1.3 Darwin arm64 release on the
50K x 1536d performance case (drop-old, fresh table): the field-less variant
registers on the first attempt, zero shard errors, recall 0.9974 / nDCG
0.9981 -- identical to a manually pre-created index. Found during the
2026-08-06 antfly-circus main smokeout.
…f __init__ CI's make lint runs black --check plus ruff. Both were already failing on main (serial_runner.py unformatted for the line-length-120 config; antfly.py over the 50-statement limit for __init__), and the index-verification code added here pushed __init__ past the branch limit as well. - Move external-index creation into _ensure_external_index, which also lets the variant loop use self._legacy_api instead of threading api_root through, and collapses the now-identical field-variant branches. - Reformat antfly.py and serial_runner.py with black. Re-validated end to end against the published v0.1.3 Darwin arm64 release on the 50K x 1536d case after the refactor: field-less variant registers on the first attempt, recall 0.9974 / nDCG 0.9981.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On the legacy
/api/v1line, index creation can return 2xx from the metadata layer while shard-level registration fails asynchronously — the reconciler then retries the rejected config forever and every vector write fails withindex not found. The two known legacy behaviours are opposites:field or template must be specified)external embeddings index config cannot specify field or template)so no fixed variant ordering can be correct (the previous field-first ordering for legacy 201s on v0.1.3 and then every insert 500s).
Fix: on the legacy path, after each 2xx, poll the index status until every shard reports a non-null registration; if it stays broken within a bounded window (~16 s), delete the index and try the next variant/index-type. Field-less is tried first. The non-legacy path accepts on 2xx exactly as before.
Verified end to end against the published v0.1.3 Darwin arm64 release on
Performance1536D50K(50K × 1536d, drop-old, fresh table): field-less variant registers first attempt, zeroindex not foundshard errors, recall 0.9974 / nDCG 0.9981 — identical to a manually pre-created index. Found during the 2026-08-06 antfly-circus main smokeout; a narrower campaign-local variant of this patch ran that lane's original comparison.