[Detail Bug] Ingestion: Batched local docs indexing can leave orphaned vectors when marker write fails mid-batch

[detail-app]

Detail Bug Report

https://app.detail.dev/org_befd6425-a158-4e24-9d4d-1e5c08769515/bugs/bug_1e099e25-62e0-491a-a6d5-762253f0323c

Introduced in #114 by @WilliamAGH on Jul 27, 2026

Summary

• Context: LocalDocsFileIngestionProcessor.flushNewFileBatch processes multiple files in a single batched upsert to Qdrant.
• Bug: When the batched upsert succeeds but marker write fails for any file in the batch, all subsequent files in the batch are left with orphaned vectors in Qdrant and no local ingestion markers.
• Actual vs. expected: Files after the failure point have vectors in Qdrant without ingestion markers. All files in the batch should have markers written, or the batch should be treated atomically (all markers or none).
• Impact: On subsequent runs, inspectUnmarkedVectors detects orphaned vectors and forces unnecessary reindexing. Wastes embedding API calls and compute.

Code with Bug

try {
    storage.hybridVector().upsert(collectionKind, combinedDocuments);  // SUCCEEDS FOR ALL FILES
} catch (EmbeddingServiceUnavailableException embeddingException) {
    // ... all files marked as failed, no vectors exist ...
    return false;
} catch (RuntimeException vectorStorageException) {
    // ... all files marked as failed, no vectors exist ...
    return false;
}

for (DocumentProcessingRequest processingRequest : preparedFiles) {
    LocalDocsFileOutcome outcome = completeDocumentsAfterStorage(processingRequest);
    outcomes.add(outcome);
    if (outcome.failure().isPresent()) {
        return false;  // <-- BUG 🔴 exits early, so later files never get marker writes even though vectors were already upserted
    }
}

Explanation

• upsert(collectionKind, combinedDocuments) submits all documents for the batch in a single call; once it succeeds, vectors for every file in preparedFiles already exist in Qdrant.
• Marker persistence happens later, per file, inside completeDocumentsAfterStorage.
• If any file’s marker write fails, flushNewFileBatch returns immediately, so marker writes for remaining files in the batch are never attempted.
• Resulting state after a mid-batch failure: vectors exist for files after the failure point, but their local ingestion markers do not.

Codebase Inconsistency

• The system has recovery logic (inspectUnmarkedVectors) specifically to detect “vectors exist without markers” and triggers a full reindex when it finds them. This bug creates that state as a normal failure mode.
• Reindexing is expensive here because embeddings are recomputed via live calls (embeddingClient.embed(...)), not loaded from an embedding cache.

Recommended Fix

Continue the marker-write loop even after a per-file marker failure; record failures but do not return early. After processing all files, return false if any outcome failed. This ensures that if vectors were already upserted for the batch, all files still get a best-effort marker write attempt, avoiding orphaned vectors for files after the first failure.

History

This bug was introduced in commit 7ad340c. The commit "fix(ingestion): coalesce new document embeddings" added batch processing to amortize embedding API costs by combining up to 32 files into single gateway calls. The bug slipped in because the new flushNewFileBatch method writes all markers after the combined upsert, creating a window where vectors exist in Qdrant but markers haven't been written for some files if a marker write fails mid-iteration.