Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
- `packages/ragmir-tts` is the optional local/offline audio add-on used by `rgr audio`.
- `packages/ragmir-landing` is a self-contained, telemetry-free Astro site. Keep it static, open-source focused, and free of vendor deployment configuration.
- Ragmir Core stays retrieval-first: `local-hash` supports offline retrieval, `transformers` is the explicit semantic option, and local chat remains a separate add-on.
- Long-running Node.js processes use one `RagmirClient` per project root and close it during shutdown. Keep the top-level API for one-shot scripts.
- Ragmir does not provide an HTTP server or fixed port. A network-facing host owns transport security, authentication, authorization, and rate limits.
- Ingestion is serialized per local index inside one Node.js process; do not claim a distributed writer lock.
- Public copy must lead with model-agnostic Core and the choice between the user's preferred AI or automation and a fully local consumer. Qwen and Gemma are optional Chat profiles, never Core or MCP requirements.

## Privacy and ingestion
Expand Down
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ native desktop shell, or cloud-vendor deployment configuration. Keep OCR optiona
local, remote model downloads explicit, and normal confidential retrieval offline.
Describe Core as model-agnostic: users can connect their preferred AI or automation, or keep the
consumer local. Qwen and Gemma are optional Chat profiles, never Core or MCP requirements.
For repeated retrieval in a stateful Node.js process, use one `RagmirClient` per project root and
close it during shutdown. Ragmir does not provide an HTTP server or fixed port; network-facing hosts
own transport security, authentication, authorization, and rate limits.

<!-- gitnexus:start -->
# GitNexus — Code Intelligence
Expand Down
47 changes: 31 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,14 @@ owns the source files.
| Interface | Use it for | Result |
| --- | --- | --- |
| `rgr` CLI | Setup, ingest, search, audit, and maintenance | Human-readable or JSON output |
| TypeScript API | Embed retrieval in a Node.js application | Typed results with citations |
| TypeScript API | Embed retrieval in a script or stateful Node.js worker | Typed results with citations and explicit lifecycle |
| Local MCP server | Give your preferred agent bounded project context | Read-focused retrieval tools |
| Ragmir Chat | Keep answer generation on the workstation | Cited offline synthesis |
| Ragmir TTS | Turn a text brief into audio | Local WAV or explicit online MP3 |

Use the CLI or MCP for interactive agent work. Use the TypeScript API when a repeatable Node.js
process owns the control flow.
process owns the control flow. Ragmir does not open an HTTP port: applications own their network,
authentication, and authorization boundary.

Ragmir Core stays retrieval-first. `ask()` returns cited context without calling an LLM. Local chat
and audio are separate capabilities, so retrieval remains useful on machines that should not run a
Expand Down Expand Up @@ -180,23 +181,37 @@ binary support.
## TypeScript API

```ts
import { ingest, search } from "@jcode.labs/ragmir"

await ingest({ cwd: process.cwd() })

const results = await search("Which decision changed the rollout?", {
topK: 5,
explain: true,
})

for (const result of results) {
console.log(result.citation, result.text)
import { createRagmirClient, isRagmirError } from "@jcode.labs/ragmir"

const ragmir = await createRagmirClient({ cwd: process.cwd() })
try {
await ragmir.ingest({ timeoutMs: 120_000 })

const results = await ragmir.search("Which decision changed the rollout?", {
topK: 5,
explain: true,
timeoutMs: 10_000,
})

for (const result of results) {
console.log(result.citation, result.text)
}
} catch (error) {
if (isRagmirError(error)) console.error(error.code, error.retryable)
else throw error
} finally {
await ragmir.close()
}
```

Core also exports `previewChunks`, `ask`, `research`, `audit`, `doctor`, `securityAudit`,
`discoverKnowledgeBases`, bounded context helpers, `serveMcp`, and setup helpers. See the
[API reference](./docs/api-reference.md) for the public surface.
Reuse one client per project root in a long-running process. It keeps one local LanceDB connection,
serializes ingestion for the same index inside that process, accepts `AbortSignal` and `timeoutMs`,
and waits for active operations during `close()`. One-shot `ingest`, `search`, `ask`, and `research`
functions remain available for short scripts.

Core also exports `previewChunks`, `audit`, `doctor`, `securityAudit`, bounded context helpers,
closeable MCP construction helpers, and setup helpers. See the [API reference](./docs/api-reference.md)
for the complete public surface.

## Privacy boundaries

Expand Down
2 changes: 2 additions & 0 deletions context7.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
"rules": [
"Ragmir Core returns cited retrieval context only; it does not synthesize answers itself.",
"Any compatible coding agent, script, CLI, TypeScript application, or MCP client can consume Core results; use a local consumer when no passage may leave the machine.",
"Use one `createRagmirClient()` per project root for repeated work in a stateful Node.js process; close it during shutdown and use the top-level functions for one-shot scripts.",
"Ragmir does not open an HTTP port; a network-facing host owns its transport, authentication, authorization, and rate limits.",
"`rgr chat` is optional add-on generation via @jcode.labs/ragmir-chat; the core remains retrieval-only.",
"Qwen and Gemma are optional Ragmir Chat profiles, never requirements of Core, the CLI, the TypeScript API, or MCP.",
"The `local-hash` embedding provider (default) is a lexical sha256 embedding, not semantic; use `transformers` for semantic retrieval.",
Expand Down
6 changes: 6 additions & 0 deletions docs/agent-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,12 @@ The generated helpers cover Claude Code, Codex, Kimi, OpenCode, and Cline. Other
the same evidence through the CLI, TypeScript API, or any compatible MCP client. Hermes, n8n
workers, CI jobs, and internal applications do not require a dedicated Ragmir model integration.

Embedding applications can call `createMcpServer(cwd)` to register a caller-owned transport, or
`connectMcpServer(transport, cwd)` to connect it and receive a closeable server handle. The standard
`serveMcp(cwd)` helper remains the simplest local stdio entry point. MCP cancellation propagates to
search, ask, research, and citation expansion. Ragmir does not open an HTTP port; applications that
expose a network transport own its authentication and authorization boundary.

## Verify

```bash
Expand Down
59 changes: 54 additions & 5 deletions docs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,51 @@ for (const result of results) {
Search results include `relativePath`, `citation`, `chunkIndex`, exact text, line ranges, page ranges
when available, structural context, and optional score explanations.

### Persistent client for Node.js workers

Use one client per project root when a stateful Node.js process performs repeated retrieval. The
client reuses one strongly consistent LanceDB connection and resolves its project root at creation.

```ts
import { createRagmirClient, isRagmirError } from "@jcode.labs/ragmir"

const controller = new AbortController()
const ragmir = await createRagmirClient({ cwd: process.cwd() })

try {
await ragmir.ingest({ signal: controller.signal, timeoutMs: 120_000 })
const results = await ragmir.search("release approval", {
topK: 5,
signal: controller.signal,
timeoutMs: 10_000,
})
console.log(results.map(({ citation }) => citation))
} catch (error) {
if (isRagmirError(error)) {
console.error(error.code, error.retryable)
} else {
throw error
}
} finally {
await ragmir.close()
}
```

`RagmirClient` exposes `ingest`, `search`, `ask`, `research`, `expandCitation`, `status`, `sources`,
and an idempotent `close`. `close()` rejects new work, waits for active operations, then closes the
shared connection. Ingestion targeting the same storage directory is serialized inside one Node.js
process. Cancellation is cooperative between parsing, embedding, storage, and retrieval phases.

`RagmirError.code` is one of `ABORTED`, `CLIENT_CLOSED`, `INTERNAL`, `INVALID_ARGUMENT`, or
`TIMEOUT`. `retryable` is true for cancellation and timeout errors.

The writer queue coordinates clients inside one Node.js process. If several OS processes can ingest
the same storage directory, the host must elect one writer or serialize those ingestion jobs.

This API targets stateful Node.js processes with a local filesystem. It is not an edge or stateless
serverless API, and Ragmir does not provide an HTTP listener. A network-facing application owns
authentication, authorization, rate limits, and transport security.

### Project and source setup

| Export | Purpose |
Expand Down Expand Up @@ -65,9 +110,11 @@ when available, structural context, and optional score explanations.
| `evaluateGoldenQueries(options)` | Score retrieval against a local golden-query file. |

`SearchOptions` accepts `cwd`, `topK`, `contextRadius`, `includePaths`, `excludePaths`,
`contextPaths`, and `explain`. When explanation is enabled, each result includes reciprocal-rank
fusion contributions, one-based vector and lexical ranks, vector distance, lexical backend score,
and matched query terms. `ExpandCitationOptions.contextRadius` is clamped to three chunks.
`contextPaths`, `explain`, `signal`, and `timeoutMs`. `IngestOptions`, `ResearchOptions`, and
`ExpandCitationOptions` also accept `signal` and `timeoutMs`. When explanation is enabled, each
result includes reciprocal-rank fusion contributions, one-based vector and lexical ranks, vector
distance, lexical backend score, and matched query terms. `ExpandCitationOptions.contextRadius` is
clamped to three chunks.

Structural context comes from Markdown headings or structured-data paths. It can improve candidate
selection without changing the exact text, offsets, or citations returned to the caller.
Expand Down Expand Up @@ -106,6 +153,8 @@ model download must be explicitly enabled before local inference can use it.

| Export | Purpose |
| --- | --- |
| `createMcpServer(cwd?)` | Construct the read-focused MCP server without selecting a transport. |
| `connectMcpServer(transport, cwd?)` | Connect a caller-owned MCP transport and return a closeable server handle. |
| `serveMcp(cwd?)` | Start the local stdio MCP server. |
| `installAgentSkills(options?)` | Install the canonical skill kit for selected native agents. |
| `installSkill(options?)` | Install one bundled skill with ownership checks. |
Expand All @@ -121,7 +170,7 @@ model download must be explicitly enabled before local inference can use it.
New integrations should use `rgrCommand` and the `rgr` CLI name. MCP retrieval tools accept a
`maxBytes` value below the configured `mcpMaxOutputBytes` ceiling. Search, ask, and research also
accept compact output. Metrics are returned under `_meta["ragmir/output"]` and summarized by the
metadata-only usage report.
metadata-only usage report. MCP cancellation signals propagate into Core retrieval operations.

### Core type exports

Expand All @@ -136,7 +185,7 @@ types that callers commonly compose explicitly.
| Retrieval | `SearchOptions`, `SearchResult`, `SearchContextChunk`, `SearchScoreExplanation`, `AskResult`, `CompactSearchResult`, `ExpandCitationOptions`, `ExpandedCitation` |
| Research and evaluation | `ResearchOptions`, `ResearchReport`, `ResearchEvidence`, `CodeEvidence`, `SourceDiagnostics`, `SourceDuplicateCandidate`, `SourcePathCandidate`, `EvaluationOptions`, `EvaluationResult`, `EvaluationCaseResult`, `GoldenQuery` |
| Bases and sources | `KnowledgeBaseIdentity`, `KnowledgeBaseInfo`, `KnowledgeBaseInventory`, `KnowledgeBaseContextReport`, `KnowledgeBaseSourceCatalog`, `AddSourceEntriesOptions`, `AddSourceEntriesResult`, `SourceEntriesResult` |
| Operations | `DoctorReport`, `SecurityAuditReport`, `DestroyIndexResult`, `AccessLogAction`, `AccessLogUsageOptions`, `AccessLogUsageReport`, `McpOutputTool`, `McpOutputUsageReport`, `RedactionCount` |
| Operations | `RagmirClientOptions`, `OperationOptions`, `RagmirErrorCode`, `DoctorReport`, `SecurityAuditReport`, `DestroyIndexResult`, `AccessLogAction`, `AccessLogUsageOptions`, `AccessLogUsageReport`, `McpOutputTool`, `McpOutputUsageReport`, `RedactionCount` |
| Embeddings and OCR | `EnableSemanticEmbeddingsResult`, `PullEmbeddingModelResult`, `ConfigurePdfOcrOptions`, `ConfigurePdfOcrResult`, `ExtractPdfPageOptions`, `OcrExecutableStatus`, `PdfOcrEngine`, `PdfOcrEngineSelection`, `PdfOcrStatus` |
| Agent integration | `AgentHelperFile`, `AgentInstallMode`, `AgentInstallScope`, `AgentIntegrationReport`, `AgentSkillInstallation`, `AgentTarget`, `InstallAgentSkillsOptions`, `InstallAgentSkillsResult`, `InstallSkillOptions`, `InstallSkillResult`, `RagmirRunnerMode` |
| Setup and commands | `SetupOptions`, `SetupResult`, `SetupSemanticResult`, `PackageManager`, `RagmirCommand`, `PromptRouteDecision`, `PromptRouteTool` |
Expand Down
5 changes: 5 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,8 @@ RAGMIR_MCP_MAX_OUTPUT_BYTES=16384 rgr mcp

Environment overrides cover selected runtime settings such as models, retrieval limits, access logs,
and extractor commands. Run `rgr status --json` to inspect the effective result.

For a long-running process that hosts more than one isolated project workflow, create one
`RagmirClient` per project root and keep process-wide environment overrides stable after startup.
Close every client during shutdown. If several OS processes can ingest the same storage directory,
the host must coordinate a single writer.
4 changes: 4 additions & 0 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ store.
agent or optional local chat add-on.
- Consumer choice: use the AI or automation you already have through CLI, TypeScript, or MCP. Use a
local consumer or optional Chat when no retrieved passage may leave the machine.
- Persistent API: `createRagmirClient()` reuses one local connection in a stateful Node.js worker,
supports cancellation and timeouts, and closes gracefully after active operations.
- Service boundary: Ragmir does not open an HTTP port. Network transports, authentication, and
authorization belong to the embedding application.
- Default retrieval: local-hash, with no model download. Semantic embeddings and local GGUF chat are
explicit opt-ins.
- Optional packages: `@jcode.labs/ragmir-chat` for local cited GGUF synthesis and
Expand Down
3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
"sovereign-rag",
"mcp",
"ai-agents",
"coding-agents",
"nodejs",
"typescript",
"private-ai",
"local-first",
"knowledge-base"
Expand Down
40 changes: 25 additions & 15 deletions packages/ragmir-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,9 @@ Use that command from a local shell script, a Node.js worker, or a CI step after
and its local `.ragmir/` state. The process returns machine-readable cited passages; the workflow
decides whether to continue, request human approval, or stop.

For long-running integrations, start the local stdio server with `npx rgr serve-mcp`. The MCP surface
is bounded and read-focused: status, source coverage, search, retrieval-only ask, research, exact
citation expansion, audit, evaluation, usage, and security checks. It never exposes index deletion.
For long-running agent integrations, start the local stdio server with `npx rgr serve-mcp`. For a
Node.js worker that owns the control flow, use `createRagmirClient()` and reuse one client per project
root. Ragmir does not open an HTTP port or define an authentication layer.

## Search directly from the CLI

Expand Down Expand Up @@ -113,32 +113,42 @@ multi-query retrieval pass and reports missing or weak evidence.
## TypeScript API

```ts
import { ingest, search } from "@jcode.labs/ragmir"

await ingest({ cwd: process.cwd() })

const results = await search("Which decision changed the rollout?", {
cwd: process.cwd(),
topK: 5,
})

for (const result of results) {
console.log(result.citation, result.text)
import { createRagmirClient } from "@jcode.labs/ragmir"

const ragmir = await createRagmirClient({ cwd: process.cwd() })
try {
await ragmir.ingest({ timeoutMs: 120_000 })

const results = await ragmir.search("Which decision changed the rollout?", {
topK: 5,
timeoutMs: 10_000,
})

for (const result of results) {
console.log(result.citation, result.text)
}
} finally {
await ragmir.close()
}
```

The persistent client reuses one local database connection, supports cooperative cancellation, and
waits for active work before shutdown. The top-level functions remain the smallest API for one-shot
scripts.

Frequently used exports:

| Export | Purpose |
| --- | --- |
| `setupProject`, `addSourceEntries` | Initialize project state and select files |
| `discoverKnowledgeBases`, `knowledgeBaseIdentity` | Route root and nested monorepo bases |
| `getKnowledgeBaseContext`, `getKnowledgeBaseSourceCatalog` | Give agents bounded readiness and source context |
| `createRagmirClient`, `RagmirClient` | Reuse local retrieval safely in a stateful Node.js process |
| `ingest`, `audit` | Build the index and compare it with files on disk |
| `previewChunks` | Inspect redacted chunks and distributions without writing storage |
| `search`, `ask`, `research`, `expandCitation` | Retrieve or expand cited passages |
| `doctor`, `securityAudit` | Inspect readiness and local privacy posture |
| `serveMcp` | Start the read-focused local MCP server |
| `createMcpServer`, `connectMcpServer`, `serveMcp` | Construct, connect, or start the read-focused local MCP server |
| `configurePdfOcr`, `inspectPdfOcr` | Configure and inspect local PDF OCR |

See the [complete API reference](https://github.com/jcode-works/jcode-ragmir/blob/main/docs/api-reference.md)
Expand Down
Loading