docs(claude): lean CLAUDE.md index + path-scoped rules, and a standards-check skill

.claude/rules/api.md

@@ -0,0 +1,57 @@
+---
+paths:
+  - "api/routers/**"
+  - "api/schemas/**"
+  - "api/main.py"
+  - "api/middleware.py"
+  - "api/dependencies.py"
+---
+
+# HTTP API
+
+FastAPI app built in `api/main.py` (`root_path="/api"` — nginx strips the prefix). Layering is strict and
+one-way: **`router → service → (adapter | db)`** (see [`architecture.md`](architecture.md)).
+
+## Routers stay thin
+`api/routers/*.py` do path registration + `Depends(...)` + an authorization check, then a **single
+delegating call** to a service. No business logic, no raw SQL, no schema declarations. The routers:
+
+| Router | Base | Auth | Responsibility |
+|--------|------|------|----------------|
+| `health` | `/healthz`, `/metrics` | none | liveness snapshot |
+| `workspaces` | `/workspaces` | router `require_master` | workspace CRUD |
+| `collections` | `/workspaces/{ws}/collections` | router `require_master` | collection CRUD + API-key mgmt |
+| `documents` | nested + flat `/documents…` | per-route `require_auth` + `can_access` | upload / list / status / delete / download / reprocess |
+| `tags` | `/workspaces/{ws}` | router `require_master` | tag CRUD, merge, assignment |
+| `graph` | `/workspaces/{ws}` | router `require_master` | tag-correlation graph |
+| `search` | `POST /search` | per-route `require_master` | multi-collection hybrid search |
+| `indexing` | `/indexing/…` | mixed master/auth | BM25 (re)index status + enqueue |
+| `jobs` | `/ingestion/jobs…` | `require_master` | ingestion-queue list, stats, retry-failed |
+| `config` | `/config` | router `require_master` | live config GET/PUT, ollama-models, reload-status |
+| `ws` | `WS /ws` | `?key=` query param | Redis pub/sub → WebSocket (ingestion progress) |
+| `mcp` | mounted `/mcp` | in middleware | MCP ASGI sub-app, mounted **last** ([`mcp.md`](mcp.md)) |
+
+## Adding an endpoint
+1. **Route** → the matching `api/routers/<domain>.py`; handler resolves/authorizes then `return await <svc>.<fn>(...)`.
+2. **Model** → CRUD request bodies go in `api/schemas/<domain>.py`; richer domain/response/query models and
+   config go in `api/models/<domain>.py`. (`schemas/` = per-endpoint DTOs; `models/` = shared domain/config
+   contracts reused across layers. `api/tables/` is persistence, not Pydantic.)
+3. **Logic + DB** → new function in `api/services/<domain>.py` (`select(...)` on the injected `AsyncSession`;
+   raise `HTTPException` on domain errors — [`database.md`](database.md)).
+4. **Register** in `api/main.py` (`include_router`); keep the MCP mount last.
+5. **Inject deps** from `api/dependencies.py`: `db = Depends(get_db)`; auth `require_master` / `require_auth`
+   ([`permissions.md`](permissions.md)); adapters `Depends(require_embedding_adapter | require_vector_store)`
+   (503 if not ready) or `Depends(get_reranker)` (optional, may be `None`); config `Depends(get_search_config | get_app_config)`.
+
+## Errors, middleware, lifespan
+- Services raise `fastapi.HTTPException(status, detail)` → `{"detail": …}`. Codes in use: 401 (bad/missing key),
+  403 (wrong-collection / master-required), 404, 409 (conflict), 415 (unsupported file type), 422 (validation),
+  503 (backend/queue not ready). No custom global handlers — FastAPI defaults + the request-id middleware.
+- **Graceful degradation is a hard rule**: an optional stage that lacks a model / errors falls back and
+  **never 500s** a search or ingest.
+- Middleware (`api/middleware.py`): `RequestIDMiddleware` (outermost) mints a `uuid4` request id, times the
+  request, sets `X-Request-ID`, emits one structured log line; CORS inside it.
+- Lifespan (`api/main.py`): structlog → load `config.yaml` → `init_db()` (migrations) → **background**
+  `_warm_up_adapters` (embedding + vector-store + reranker load off the startup path, so uvicorn serves
+  immediately; getters return `None` and `/healthz` reports not-loaded until ready) → Redis → run the MCP
+  session manager for the app's lifetime.

.claude/rules/architecture.md

@@ -0,0 +1,42 @@
+---
+paths:
+  - "api/**"
+  - "worker/**"
+---
+
+# Architecture conventions — so "follow the pattern" is concrete
+
+The principles behind these are in [`code-standards.md`](code-standards.md). This file is the
+"how, in this repo."
+
+## Adapters — the extension seam
+`api/adapters/{embeddings,parsers,reranker,vector_store}` (also `tagging`, `llm_chat`):
+
+- A `runtime_checkable` **`Protocol` in `base.py`**, **one provider per file**, resolved by a
+  **`get_*(config)` factory** that dispatches on `config.provider` (or file type) with **lazy imports
+  inside each branch** and **raises `ValueError` on an unknown provider**.
+- A disabled optional stage's factory returns **`None`** so callers skip it with `is not None`
+  (see `get_reranker`).
+- **Adding a backend = a new file under `api/adapters/<kind>/` + one branch in that kind's factory.**
+  Callers depend on the Protocol, never on a concrete class. Do not edit callers to add a backend.
+- Keep the Protocols minimal (ISP). Optional capabilities stay *off* the Protocol — e.g. a parser's
+  `on_progress(current, total)` is detected by signature inspection in the worker, not declared on
+  `ParserAdapter`.
+
+## Graceful degradation — a hard rule
+Any stage that lacks a model, times out, or errors **falls back to prior behaviour and never turns a
+search / ingest into a 500.** Concretely: a missing reranker → RRF-only ranking; adjacency expansion is
+best-effort; `/healthz` surfaces the degrade. **A new stage that can 500 the request is a bug.**
+
+## Layering — one direction only
+`router → service → (adapter | db)`. Never the reverse; adapters/db never import services/routers.
+- **Routers** (`api/routers/`) are *routing-only*: path + `Depends(...)`, an authorization check, then a
+  single delegating call to a service. No business logic, no raw SQL, no schema declarations there.
+- **Services** (`api/services/`) own business logic + persistence; they raise `HTTPException` for domain errors.
+- Details of the HTTP layer: [`api.md`](api.md).
+
+## Where the specifics live
+- **Config** (pydantic models, secrets, hot-reload): [`configuration.md`](configuration.md).
+- **Metadata DB** (ORM-only, tables, Alembic): [`database.md`](database.md).
+- **Vector store** (the one raw-asyncpg exception): [`vector-db.md`](vector-db.md).
+- **Tests / fakes / DI-for-testability**: [`testing.md`](testing.md).

.claude/rules/code-standards.md

@@ -0,0 +1,53 @@
+---
+paths:
+  - "api/**"
+  - "worker/**"
+  - "tests/**"
+---
+
+# Code standards — DRY · KISS · YAGNI · SOLID
+
+The bar is **make it work, simply, without repeating yourself — and prove it.** These are the
+principles; the concrete repo conventions that implement them live in
+[`architecture.md`](architecture.md).
+
+## DRY — one fact, one place
+- **Search for an existing implementation before writing a new one**, and reuse it. Duplication
+  discovered later is a defect, not a style nit.
+- **Shared test doubles live in `tests/unit/fakes.py` and `tests/unit/fake_redis.py`** — import them
+  (`from tests.unit.fakes import FakeStore, FakeEmbedder`). **Never** re-declare `FakeStore` /
+  `FakeEmbedder` / `FakeRedis` inside a test module. A new reusable double goes in those modules,
+  not inline. (More in [`testing.md`](testing.md).)
+- Shared domain logic belongs in a service/adapter/module — never copy-pasted across routers, tasks, or tests.
+- A literal used in more than one place becomes a named constant (see `api/constants.py`).
+- **Fixing a DRY violation means finding *every* copy**, not only the one that was pointed out.
+
+## KISS — the simplest thing that works
+- Prefer a function to a class, a plain call to a framework, a straight line to a clever one.
+  Optimise for the next reader.
+- No premature abstraction: the **second** concrete case justifies an interface, the first does not.
+- If a change needs a paragraph to explain why it's clever, it's probably wrong.
+
+## YAGNI — build what's needed now
+- No config knobs, parameters, hooks, or "for later" branches without a **current** caller.
+  The roadmaps in `plans/` and `docs/` are staged on purpose — don't pull a future phase forward.
+
+## SOLID — where it earns its keep
+- **SRP** — one reason to change per unit. Parsing ≠ chunking ≠ embedding ≠ storing; keep them
+  separate, mirroring `api/services/` and `api/adapters/`.
+- **OCP** — extend by adding, not by editing callers. A new embedding / reranker / parser backend is
+  **a new file under `api/adapters/<kind>/` plus one branch in that kind's `get_*()` factory** —
+  callers depend on the factory, never on a concrete class. (Pattern in [`architecture.md`](architecture.md).)
+- **LSP** — a new adapter honours its Protocol's full contract, including graceful degradation.
+- **ISP** — keep the Protocols in `api/adapters/base.py` minimal. Optional capabilities stay *off* the
+  Protocol (see the `on_progress` note on `ParserAdapter`) so implementations that lack them still conform.
+- **DIP** — depend on the `Protocol`, inject the dependency. `_run_ingestion` takes
+  `embedder` / `vector_store` / `storage` as parameters — that's *why* it is testable with the fakes.
+  Follow that shape.
+
+## Libraries — code against *current* docs
+Before writing or debugging code against a library / framework / SDK / API / CLI, pull its **current**
+docs via the **Context7 MCP** — don't code from memory, training data lags real releases. This repo pins
+and verifies real versions (e.g. the `# verified against … via Context7` note atop
+`api/adapters/vector_store/pgvector.py`); keep that habit for new adapters and upgrades.
+Canonical rule: `CLAUDE.md` § Libraries & docs.

.claude/rules/configuration.md

@@ -0,0 +1,49 @@
+---
+paths:
+  - "api/models/config.py"
+  - "api/services/config_service.py"
+  - "api/services/config_env.py"
+  - "api/services/config_reload.py"
+  - "worker/config.py"
+  - "worker/config_reload.py"
+  - "config.yaml"
+  - "config.example.yaml"
+  - ".env"
+  - ".env.example"
+---
+
+# Configuration
+
+Config is **pydantic models** — `AppConfig` and its sub-configs in `api/models/config.py`
+(`embedding`, `vector_store`, `reranker`, `parsers`, `chunking`, `storage`, `mcp`, `search`, …).
+`config.yaml` and `.env` are **gitignored — never commit secrets.** The read/apply service is
+`api/services/config_service.py`; it backs the editable settings page.
+
+## Secrets — mask on GET, preserve on PUT
+- Write-only secret fields are listed in **`SECRET_PATHS`** in `config_service.py` and masked with
+  **`SECRET_MASK` (`"__SECRET_SET__"`)** by `get_masked_config()`. Currently: `embedding.api_key`,
+  `vector_store.password`, `reranker.api_key`.
+- **Adding a new secret field?** Add its path to `SECRET_PATHS` (or, for user-named backends, extend
+  `_storage_secret_paths()` — S3 `access_key_id` / `secret_access_key` are computed per-config because
+  backend names are dynamic). Otherwise a whole-config save from the UI round-trips it in the clear.
+- On PUT, `_merge_secrets()` restores any field the client echoed back as `SECRET_MASK` from the live config.
+
+## The save flow — build-then-commit, then fan out
+`apply_config()`:
+1. **Merge secrets** → **validate + dry-run build the adapters** (`_validate_and_build`). Building is the
+   dry run: a bad embedding model / unreachable backend fails **here (422)** before anything is persisted.
+   A rate-limited dimension probe → **503** (retry), not a rejection; the reranker degrades to `None` if
+   its model can't load (an unknown *provider* still raises).
+2. **Persist** `config.yaml` **in place** (not rename-swap — a rename splits Docker/WSL bind-mount views),
+   fsynced, with a `.bak` taken only when the current file is non-empty **and** parses (so a crash-truncated
+   file can't clobber the last-good backup — the boot recovery source in `config_env.load_config_data`).
+3. **Hot-swap** the API's live singletons (`_swap_live`), then **propagate to workers** over Redis and wait
+   for acks; a worker rejection triggers **rollback** (restore `.bak`, revert adapters, republish) and **409**.
+   No Redis (single-process/dev) → an API-local status record.
+
+## Gotchas
+- The config load path is `api/main.py::_load_app_config` with `.bak` crash recovery + env overlays
+  (`api/services/config_env.py`, e.g. `S3__<NAME>__*`). Startup sets `app.state.config` + `set_app_config`.
+- Worker-side reload lives in `worker/config_reload.py` / `worker/tasks.py::reload_adapters` (rebuilds the
+  worker's embedder/vector-store singletons on a new version).
+- Adapters are built from config via the `get_*()` factories — see [`architecture.md`](architecture.md).

.claude/rules/database.md

@@ -0,0 +1,30 @@
+---
+paths:
+  - "api/db.py"
+  - "api/tables/**"
+  - "api/alembic/**"
+  - "worker/db.py"
+---
+
+# Metadata DB — ORM only
+
+The metadata DB (**workspaces, collections, documents, tags, api_keys, job_records**) goes through the
+**SQLAlchemy 2.0 async ORM**. The `chunks` table is **not** here — it's the vector store's, raw asyncpg,
+see [`vector-db.md`](vector-db.md).
+
+## The rules
+- **Engine + session factory live in `api/db.py`** (`engine`, `AsyncSessionLocal`, `init_db`). Table objects
+  live in `api/tables/` and are re-exported from `api.db`, so `from api.db import documents` stays valid.
+- **Build queries with `select(...)` / `insert` / `update` / `delete` and the table objects on the injected
+  `AsyncSession`.** **Do not hand-write raw SQL against the metadata DB.**
+- **Every schema change is an Alembic migration** in `api/alembic/versions/`. `init_db()` runs
+  `alembic upgrade head` at startup (safe on every boot) from a thread executor.
+- Services own the queries and commit explicitly; routers stay thin (see [`api.md`](api.md)). The request
+  gets its `AsyncSession` via `Depends(get_db)` — unhandled exceptions auto-rollback.
+
+## Good to know
+- URL comes from `DATABASE_URL` (e.g. `postgresql+psycopg://…`); the dev fallback is
+  `sqlite+aiosqlite:///…`. SQLite-only pragmas (WAL, `foreign_keys=ON`) are set on the sync connection at
+  connect time — don't apply them to Postgres.
+- Hierarchy is `workspaces → collections → (documents, api_keys, job_records)` with FK `CASCADE`.
+- `expire_on_commit=False`, `autoflush=False` — flush/commit intentionally explicit.

.claude/rules/ingestion.md

@@ -0,0 +1,83 @@
+---
+paths:
+  - "worker/tasks.py"
+  - "worker/celery_app.py"
+  - "api/services/ingestion.py"
+  - "api/services/indexing.py"
+  - "api/services/upload.py"
+  - "api/services/storage.py"
+  - "api/services/documents.py"
+  - "api/services/jobs.py"
+  - "api/services/tasks.py"
+  - "api/services/realtime.py"
+  - "api/adapters/parsers/**"
+  - "api/routers/documents.py"
+  - "api/routers/indexing.py"
+  - "api/routers/jobs.py"
+---
+
+# Ingestion pipeline
+
+Flow: **upload → parse → chunk → embed → store**. The DI seam is `worker/tasks.py::_run_ingestion`.
+
+## Sync (API) vs async (worker)
+- **API** (`api/routers/documents.py::upload_document`): reject unsupported ext (415), stream to storage with
+  a size guard (`api/services/upload.py`, `.tmp` + atomic `os.replace`), insert `documents` + `job_records`,
+  enqueue. The **API never imports the worker package** — `api/services/tasks.py::enqueue_ingest` dispatches by
+  task-name string over Redis, keeping heavy parser/embedding deps out of the API image.
+- **Worker** (`worker/tasks.py::ingest_document → _run_ingestion`): fetch bytes to a local temp
+  (`storage.fetch_to_temp`), **parse** (`get_parser(...)`), **chunk** (`api/services/ingestion.py::sliding_window`,
+  tiktoken `cl100k_base`, called inside the txt/markdown parsers), **embed + store** in resumable batches
+  (`_embed_and_store` → `embedder.embed_batch` → `vector_store.upsert`), mark done.
+- **BM25 is not a stage** — it's the stored `chunks.text_tsv` maintained by the upsert ([`vector-db.md`](vector-db.md)).
+
+## Jobs, status, progress
+- **`job_records` table.** Status: `pending → processing → done | failed | rate_limited`. `_claim_job` atomically
+  flips pending→processing; a **Redis heartbeat** `ingest:hb:{job_id}` (TTL 600s) is refreshed on each progress event.
+- **Progress**: the `emit()` closure publishes via `api/services/realtime.py` to two pub/sub topics
+  (`ingestion:{col}` for the Documents view, `ingestion-queue` for the Queue tab) with `snapshot_key=document_id`
+  so late joiners get replayed the latest per-document state. Phases: `parsing/embedding/storing/done/failed/rate_limited`.
+  PDF page callbacks are coalesced to ~1 frame/percent.
+- **Queue read side** (master key): `api/routers/jobs.py` — `GET /ingestion/jobs` (paginated/filtered),
+  `GET /ingestion/jobs/stats`.
+
+## DI shape (why it's testable)
+`_run_ingestion(job_id, file_path, collection_id, document_id, file_type, *, session_factory=None,
+embedder=None, vector_store=None, redis_client=None, config=None, storage=None)`. Each `None` resolves to a
+lazy prod singleton; unit tests pass fakes ([`testing.md`](testing.md)). Keep new stages inject-everything.
+
+## Retries / failure / resume
+- `ingest_document` is `bind=True, max_retries=3, retry_backoff=True`. Except ladder: `SoftTimeLimitExceeded`
+  first (fail, no retry) → **rate limit** (`_is_rate_limit`: typed `RateLimitError` / httpx 429 / narrow phrase —
+  deliberately not bare "429"/"quota") → other (fail + `self.retry`).
+- **Rate limit is not a failure**: `_pause_embedding(delay)` sets a **global** TTL key `EMBEDDING_PAUSE_KEY`,
+  job → `rate_limited`, returns None (acks). One 429 pauses **all** ingestion so N queued docs don't each
+  re-hit the exhausted quota (circuit breaker); `/ingestion/jobs/stats` surfaces remaining backoff.
+- **Resume-aware, never restart from chunk 0**: `_embed_and_store` skips chunks already in
+  `vector_store.document_chunk_ids_at_model(col, doc, model)`; per-batch upsert persists immediately.
+  Deterministic `make_chunk_id` makes re-parse yield the same ids; a model change re-embeds only stale chunks.
+- **Beat sweeps** (`worker/celery_app.py`, embedded `-B`, every 300s): `retry_rate_limited_ingests` requeues
+  `rate_limited` + stale-heartbeat `processing` + orphaned `pending` via `_requeue_by_status` — re-enqueued as
+  **fresh** Celery tasks (uncapped lineage), **same `job_id`** (so `_claim_job` dedups), bounded batch. Gated
+  off while paused. `purge_expired_documents` cleans temp docs.
+- **Retry all failed**: `POST /ingestion/jobs/retry-failed` → `api/services/jobs.py::reprocess_failed_documents`
+  re-enqueues every **currently**-failed doc (latest attempt failed) matching the queue's active filters, each
+  via `documents.py::reprocess_document` (idempotent). **Single retry**: `POST /documents/{id}/reprocess`.
+- The ingestion-queue **refetch-storm fix is frontend-only** (`ui/src/realtime/useIngestionQueue.ts` debounces
+  invalidation) — no backend change; don't look for it in `api/`/`worker/`.
+
+## Changing a stage
+Stage → file: parse = `api/adapters/parsers/<ext>.py` (+ registry in `parsers/__init__.py`); chunk =
+`api/services/ingestion.py::sliding_window`; embed = `api/adapters/embeddings/<provider>.py`; store + BM25 =
+`api/adapters/vector_store/pgvector.py`; storage backend = `api/services/storage.py`. All resolve through
+`get_*()` factories — add a backend = new file + one factory branch ([`architecture.md`](architecture.md)).
+Knobs live in `api/models/config.py` ([`configuration.md`](configuration.md)).
+
+## Hard rules / gotchas
+- **Graceful degradation**: progress/status reporting is best-effort and wrapped — it must never mask the
+  original exception or 500 an ingest. Heartbeat/Redis reads **fail open** (`_job_alive` assumes alive on error,
+  so a blip can't cause double-processing). Misclassifying a 429 as a hard failure loses the infinite-resume path.
+- **Idempotency**: `_claim_job` skips `done` and live `processing`; requeue/reprocess keep the same `job_id`;
+  delete is idempotent. **Storage key is single-source-of-truth** — import `document_key`, never re-encode it.
+- RPM throttle: module-level `_RpmLimiter` (single-process — valid because worker `--concurrency=1`).
+- Effective tags (workspace→collection→document union) are folded into each chunk's metadata at embed time.