feat(indexing): replace the no-op re-index with a confirmed retry of failed ingestions#72
Merged
Merged
Conversation
…failed ingestions The per-collection re-index button called a no-op task: BM25 is the auto-maintained chunks.text_tsv generated column, so there is nothing to rebuild. A coverage gap is an unfinished ingestion, so the useful action is re-ingesting the failed documents (which re-embeds them) behind a confirmation. The button now shows only when a collection has failed documents and, on confirm, runs the existing bulk retry scoped to that collection. Scoping needs an exact id, not the queue filter's name substring (which would sweep a sibling whose name merely contains the same text), so add JobListQuery.collection_id and a CollectionIdSpec, with a test that pins the overlapping-name case. Also invalidate the index overview after a retry, correct the page header, and remove the now-orphaned useIndexCollection hook and client method.
jaymeklein
pushed a commit
that referenced
this pull request
Jul 21, 2026
Integrate the live MCP rate-limit (#70), sticky pagination (#71), and indexing retry-failed (#72) work that landed on main. Resolved two conflicts in the MCP layer: build_mcp_middleware keeps #70's live rate_limit_rpm callable alongside this branch's per-user resolve_principal, dropping the now-obsolete master_key param (auth is DB-backed via authenticate_api_key, which already handles the master key). The middleware tests construct TokenBucketRateLimiter with a callable to match #70's new signature.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the Indexing page's per-collection Re-index / Index all button — which did nothing — with a Retry failed action that re-ingests the collection's failed documents, behind a confirmation.
Why
Tracing the old button,
index_collectionis a no-op: BM25 lives in the auto-maintainedchunks.text_tsvgenerated column, so there is nothing to rebuild (seeapi/services/indexing.py). A gap in coverage is never a missing index — it is an unfinished ingestion, split into in-flight (finishing on their own) and failed (stuck). Only the failed ones are actionable, and fixing them means re-ingesting (which re-embeds), so the button now does that.How
col.failed > 0; clicking opens aConfirmDialogthat states the cost (re-embeds from the stored file, spends provider quota; in-flight docs are left alone). Confirm calls the existing bulk retry.JobListQuery.collection_id+ aCollectionIdSpec(one new spec + one line inbuild_specs, the module's documented extension point). Integration test pins the two-overlapping-names case so a sibling is never swept in.useRetryFailedJobsnow also invalidates the index overview, so retried docs visibly move failed → ingesting.Cleanup
Removes the now-orphaned
useIndexCollectionhook andapi.indexCollectionclient method (no remaining callers; YAGNI). The backend/indexendpoint stays — it is a deliberately-retained, tested public API.Gate
ruff + mypy + unit/integration green (adds the scoping test); UI typecheck clean;
docs/openapi.yamlregenerated (addscollection_idto both jobs endpoints).Follow-up (not in this PR)
The Documents page still has a per-document "Index" button that fires the same no-op; flagged separately to decide whether that state is even reachable and, if so, point it at re-ingestion.