feat: codebase embedding index for semantic search

[alpibrupa]

Problem

All navigation is grep + glob — pattern matching on names. On a large codebase, explore/plan quality is capped because the agent cannot find functions by semantic intent (e.g. "functions that validate A2A envelopes").

Design

New tool: semantic_search

semantic_search(query: Str, top_k: Int) -> List[SearchResult]

Each result: file path, function name, SigId, relevance score, one-line summary.

Implementation requires:

1. Index builder: walk all .lex files, extract function signatures + docstrings + examples {} blocks, embed via a local embedding model (or the configured LLM provider's embedding endpoint)
2. Store embeddings in .lex/index.db (sqlite-vec or similar)
3. On query: embed query, ANN search, return top_k

The index should be lazily rebuilt when source files change (compare mtime or SigId against stored hash).

Dependencies

• Check if lex-llm exposes an embedding API (prov.embed(text) -> Vec[Float])
• If not, file a companion ticket in lex-llm

Acceptance

• semantic_search "validate A2A envelope" returns relevant functions
• Available to explore, plan, review agents
• Index rebuild is incremental (unchanged files not re-embedded)

[alpibrupa]

Checked the two dependencies before starting. The premise holds — navigation really is grep and glob only — but both of the issue's implementation choices are blocked, and one of them the issue already suspected.

1. lex-llm exposes no embedding API

The issue asks to "check if lex-llm exposes an embedding API (prov.embed(text) -> Vec[Float])". It does not, and there is nothing close to one:

type Provider = { name :: Str,
                  chat   :: (ModelRef, List[msg.Message], List[t.Tool]) -> [net, llm] Iter[d.Delta],
                  stream :: Option[StreamChat] }

grep -rn 'embed' lex-llm/src/ returns only prose in comments. No embedding function, no vector type, and no provider carries an embeddings endpoint.

2. .lex/index.db is not reachable from a tool

type Tool = { …, execute :: (jv.Json) -> [net, io, proc] Result[jv.Json, e.Errors], … }

That row is fixed, and §1.6 makes record-field effect rows unify by equality — no tool can widen it. There is no sql, so semantic_search cannot open a SQLite index, sqlite-vec or otherwise. This is the same wall that stopped attestation_query from reading a trail database in #32 and forced .lex/verified.jsonl in #74.

What is actually buildable

A tool does have net, io and proc, and that is enough:

• Embeddings over net. Call an embeddings endpoint directly from the tool rather than through a Provider. Ollama's /api/embeddings is the obvious default — local, free, no key — with the endpoint configurable, following the same "off unless configured" discipline as the OTel exporter in feat: wire lex-log into agent/tool execution for distributed tracing #16. That also avoids adding an embedding half to Provider, which would be a lex-llm change with its own release coordination.
• The index as a file, not a table. .lex/index.jsonl, written and read with [io], exactly like verified.jsonl. Cosine similarity is a fold over List[Float] — no extension needed, and it keeps the index inspectable.
• Incremental rebuild. Both signals the issue suggests are available: fs.stat().mtime (in seconds — time.now_ms() is milliseconds, a mismatch that has already caused one bug here) and lex hash for SigIds. SigId is the better key, since it is exactly "did this function's content change".

So the issue is implementable, but not as prov.embed + sqlite-vec. Recording this before writing code so the design decision is visible rather than buried in a diff.

---

Generated by Claude Code