fix(seed): compliance_kb seeding is destructive and can't embed on the prod box — move embedding off-box + make resilient

[AndreLiar]

Incident (2026-06-15)

After #432 (DORA KB bilingual, 58→116 entries) deployed to prod, the CD re-seed of compliance_kb failed and left the collection empty — regulation citations disappeared from prod chat for ~hours until manually restored.

Root cause (layered)

1. Cold sidecar at deploy. CD runs seedComplianceKb.js (docker exec -d) right after redeploy, while retrieva-ollama is still reloading models → first embed batch throws fetch failed → Seed failed.
2. Destructive delete-first. Smart-sync does deleteCollection() before re-embedding. When the embed fails, the previously-good collection is already gone → compliance_kb = 0 points.
3. Can't embed on the box. retrieva-ollama is single-slot, CPU-only, 4 GiB, and saturated by runtime RAG /v1/embeddings traffic. Measured embed latency on prod: ~80–170 s per document (even for short inputs), vs ~0.25 s locally. 116 docs ≈ 3–5 h, unreliable (Node undici/langchain fetch aborts slow embeds → retry-thrash, never completes).

How it was restored

Embedded the 116 articles locally with the same bge-m3 model (~29 s total), wrote a Qdrant points file, upserted directly into prod Qdrant (upsert is ~free CPU). Verified end-to-end: FR query → 5 FR snippets, official=false, EUR-Lex /FR/ link; EN query → EN snippets, official=true. Prod compliance_kb = 116 (58 en + 58 fr).

Restored data is persistent: next deploy's smart-sync sees 116 == 116 → no re-seed. The risk is any future re-seed trigger (article count change, --reset) re-enters the broken path.

Required fix

Embedding must not happen on the prod sidecar, and seeding must never leave the KB empty.

Preferred: embed vectors in CI (or locally) with bge-m3 and ship a precomputed points artifact; prod CD does an upsert-only step (no embedding). Same model → vectors stay compatible with prod query embeddings.

At minimum, make seedComplianceKb.js resilient:

• Upsert-in-place — never deleteCollection before the new data is ready (build into a temp collection then swap, or upsert+prune by id set).
• No-timeout transport — use node:http (or an undici dispatcher with headersTimeout/bodyTimeout disabled); never the default global fetch for long CPU embeds.
• Per-document upsert so the point count climbs incrementally and a crash loses nothing.
• Wait for the embedding endpoint to be ready (poll /api/tags + a warm-up embed) before seeding in CD.
• Idempotent + safe to retry.

Acceptance

• A failed/slow embed never empties compliance_kb in prod.
• A full (re)seed of 116 entries completes in CD without manual intervention.
• FR + EN retrieval verified post-deploy (lang filter, official flag, EUR-Lex URL).

Implementation deferred (tracked for a dedicated PR). Context: #424, #432.