Skip to content

perf(worker): replace cpu-only ollama with cloud api for document embedding #436

Description

@AndreLiar

Problem

Document indexing uses the self-hosted `retrieva-ollama` sidecar (CPU-only, single slot, `bge-m3:latest`). Two compounding problems:

  1. Speed: ~17s per chunk on CPU. A 297-chunk PDF takes ~85 minutes to index.
  2. Mutual starvation: single ollama slot is shared between indexing AND RAG query embedding. One user uploading a large PDF freezes "Ask AI" for every user on the platform.

Observed in prod: uploading `Azure-SOC-2-Type-II.pdf` blocked all chat for ~85 min.

Root cause

`config/embeddingProvider.js` defaults to `EMBEDDING_PROVIDER=ollama` → self-hosted sidecar. `fileIngestionService.js` uses the same embedding provider as the RAG runtime — no separation.

Proposed fix

Option A — Separate embedding providers by purpose (recommended)

  • Indexing path (`fileIngestionService`): `EMBEDDING_PROVIDER_INDEXING=openai` → `text-embedding-3-small` (~500ms/batch, already wired as fallback)
  • RAG query path: keep as-is or also move to OpenAI

Option B — Move all embedding to OpenAI globally

Set `EMBEDDING_PROVIDER=openai`. Already implemented and tested.

Expected outcome

  • Indexing time: 85 min → <1 min
  • Multiple users can index simultaneously without blocking each other or blocking chat
  • Removes single point of failure for all AI features

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions