Problem
`backend/scripts/seedComplianceKb.js` uses a "delete collection + recreate" strategy (smart-sync: deletes if point count mismatches). If the embed step fails mid-way (e.g. ollama is cold after a fresh deploy), the collection is wiped and never restored — `compliance_kb` ends up at 0 points.
Observed in prod: CD deployed → ollama cold → seed failed at first embed call → compliance_kb dropped from 116 to 0 → all DORA knowledge base queries returned empty.
Recovery required a full local re-embed (~30s locally) + manual upsert to prod Qdrant.
Fix
Option A — Upsert-in-place (no delete)
Use deterministic point IDs (SHA-256 of article content, already implemented) and upsert without deleting first. Existing points are overwritten; new points are added; stale points can be garbage-collected separately.
Option B — Pre-embed in CI, upsert in CD
Embed during CI/build (where ollama is available and not cold), commit the `points.json` artifact, and in CD only call Qdrant upsert (no embedding). Zero risk of cold-start failure.
Option C — Seed with a timeout + retry + rollback
If embed fails, abort without touching the existing collection. Retry up to N times. Only swap the collection if the full embed succeeded.
Option A is the minimal-risk fix (one-line change: remove `deleteCollection` call, trust deterministic IDs).
Problem
`backend/scripts/seedComplianceKb.js` uses a "delete collection + recreate" strategy (smart-sync: deletes if point count mismatches). If the embed step fails mid-way (e.g. ollama is cold after a fresh deploy), the collection is wiped and never restored — `compliance_kb` ends up at 0 points.
Observed in prod: CD deployed → ollama cold → seed failed at first embed call → compliance_kb dropped from 116 to 0 → all DORA knowledge base queries returned empty.
Recovery required a full local re-embed (~30s locally) + manual upsert to prod Qdrant.
Fix
Option A — Upsert-in-place (no delete)
Use deterministic point IDs (SHA-256 of article content, already implemented) and upsert without deleting first. Existing points are overwritten; new points are added; stale points can be garbage-collected separately.
Option B — Pre-embed in CI, upsert in CD
Embed during CI/build (where ollama is available and not cold), commit the `points.json` artifact, and in CD only call Qdrant upsert (no embedding). Zero risk of cold-start failure.
Option C — Seed with a timeout + retry + rollback
If embed fails, abort without touching the existing collection. Retry up to N times. Only swap the collection if the full embed succeeded.
Option A is the minimal-risk fix (one-line change: remove `deleteCollection` call, trust deterministic IDs).