Apex Accelerator

Observable RAG Platform with Agent Orchestration — API-first document ingestion, hybrid retrieval, and YAML-driven agent graphs with full observability, multi-tenant isolation, offline licensing, and evidence-based answers.

Elevator Pitch

The 80% you shouldn't build yourself

In every AI-augmented solution, roughly 80% of the engineering effort is the same: document ingestion, hybrid search, guardrails, audit logging, tenant isolation, observability, licensing. The remaining 20% — your domain logic, your agent workflows, your unique retrieval strategy — is where your competitive edge lives.

Apex Accelerator is that 80%. A production-ready Rust platform that handles ingestion across 57 file types, hybrid retrieval, multi-step agent orchestration, and end-to-end provenance. Your team owns the 20% that matters.

"We built the platform. You build the solution."

Key Features

Retrieval & Search

Hybrid Retrieval: Dense (vector) + Sparse (BM25) + FTS with RRF fusion
4 Search Modes: Dense, Sparse, Hybrid (RRF fusion), Auto (system-selected)
Advanced Filtering: By document ID, tags (OR semantics), language, tenant
Context Assembly: Deduplication, token budget management, citation generation

Observability & Tracing

Full Observability Stack: OpenTelemetry traces, Prometheus metrics, Grafana dashboards, Loki logs
Provenance Tracing: End-to-end lineage from source documents to answers with structured events
27 SLO Alert Rules: Availability, latency, saturation, data quality monitoring
Langfuse Integration: Optional LLM observability with trace correlation

Authentication & Security

Multi-mode Auth: None, API Key, or OIDC (Keycloak)
RBAC: 4-role hierarchy (Viewer/Editor/AgentOperator/Admin) with explicit policy matrix
License Enforcement: PASETO v4.public middleware — request blocking, degradation, or warn mode
Multi-tenant Isolation: Tenant-scoped caching, filtering, data access, and conversation tracking
Security Guardrails: Prompt injection detection, PII redaction, cross-tenant access prevention, SSRF protection
SQL Safety: Allowlist-based query validation for agent tools

Intelligence & Reasoning

Agent Framework: YAML-driven agent specifications with graph-flow execution
Checkpoint Governance: Human-in-the-loop approve/reject, wall-clock timeout governance (Fallthrough/Fail/Escalate), high-impact action gating via LLM classification
Agent Routing: Off/Manual/Auto selection modes with confidence scoring, collection eligibility, fallback to classic chat
8 Specialized Tools: Semantic, Sparse, Hybrid, FTS search + SQL allowlist + Document retrieval + Evidence map + Policy check
Query Classification: LLM-based intent classification (6 categories) with entity extraction and conversation-aware history modes
ReAct Pattern Support: Iterative reasoning-action loops for complex queries; UPEO fallback when ReAct fails
Graph Visualization: Mermaid flowchart generation from agent specs
Conversation Tracking: 4-level ID hierarchy (request → run → conversation → session)
Evidence-based Answers: Citations with source URIs, licenses, ACL tags
Run Management: List, delete (single/bulk), and export agent runs with full audit trail
Prompt Catalog: DB-backed prompt management with immutable revisions (SHA-256), specificity scoring, publish/unpublish/rollback lifecycle, runtime resolution with provenance tracking

Evidence Packs & Compliance

Evidence Pack Export: ZIP bundles with markdown audit report, run metadata, and source citations
Digital Signatures: Ed25519 signing of evidence packs with configurable key management (future: KMS)
ISO 24970 Logging: Structured governance fields (system version, LLM model, embedding model, event type)
Locale-aware Formatting: ~20 locales for human-readable dates/numbers in audit reports (en, de, fr, es, it, ja, zh, ru, etc.)

Cost Tracking

Per-1M Token Pricing: Industry-standard pricing with 3-tier cost model (input, cached input, output)
Cached Input Discount: Automatic 50% discount tracking for cached prompt tokens
Reasoning Token Accounting: Reasoning-era model tokens (o1, o3) billed as output
Per-request Cost Metrics: Histogram and cumulative counters for chat and embedding costs

Licensing

PASETO v4.public Tokens: Offline-capable license system using Ed25519-signed tokens — no license server required
Self-Audit Loop: Background engine re-verifies license every N seconds, emitting structured events on state transitions
7 License States: Valid, GracePeriod, Expired, NotYetValid, Invalid, Missing, Demo — with automatic temporal recovery
Degradation Policies: Configurable on_expiry (DEGRADE/STOP/WARN_ONLY) and on_limit_exceeded (THROTTLE/DENY/WARN_ONLY)
Feature Gating: Per-feature licensing (ingest, search, chat, agents, evidence_pack, oidc_auth, multi_tenant)
Prometheus Metrics: license_valid, license_expiry_seconds_remaining, license_check_total, license_blocked_total
Vendor CLI (license-gen): Generate keypairs, sign/verify/inspect license files — never shipped in production
3 Operating Modes: enforce (production), demo (development), warn (migration)

Document Processing

Multi-Format Ingestion: 15 built-in formats covering 57 file extensions, extensible via FormatExtractor trait + ExtractorRegistry:
- PDF (.pdf) — PDFium text extraction + Tesseract OCR fallback, heading capture (font-size heuristic), native metadata (info dict: Title, Author, Subject, Keywords, etc.), page count
- Markdown (.md, .markdown) — ATX heading parser with level/title/char_offset, pass-through text
- Plain text (.txt, .text) — Direct text ingestion, no structure extraction
- DOCX (.docx) — OOXML text + heading extraction via docx-rust, Dublin Core metadata from docProps/core.xml, page count from docProps/app.xml
- PPTX (.pptx) — Slide text + speaker notes extraction, heading detection from slide titles
- XLSX (.xlsx, .xls, .ods) — Multi-sheet extraction via calamine with automatic header detection
- HTML (.html, .htm) — HTML content extraction with tag stripping, DOM depth limit
- CSV (.csv, .tsv) — Tabular data ingestion (comma and tab delimited)
- JSON (.json, .jsonl, .ndjson) — Structured and newline-delimited JSON
- YAML (.yaml, .yml) — YAML document ingestion
- XML (.xml) — XML content extraction
- Code (.py, .rs, .js, .ts, .tsx, .jsx, .java, .go, .c, .cpp, .h, .hpp, .rb, .sh, .bash, .toml, .ini, .cfg, .sql, .r, .swift, .kt, .scala, .lua, .pl, .php, .cs) — Source code ingestion (27 extensions)
- EML (.eml) — Email body text extraction via mail-parser with recursive attachment processing
- Images (.png, .jpg, .jpeg, .tiff, .tif, .bmp, .gif, .webp) — OCR via Tesseract for text extraction from images
- ZIP (.zip) — Recursive archive extraction with configurable depth/size/entry limits, preflight validation (path traversal, symlinks, zip bombs), global budget tracking, OOXML signature acceptance
Magic-Byte Validation: PDF (%PDF), DOCX/PPTX/XLSX (PK\x03\x04 ZIP signature), ZIP (ZIP signature with non-OOXML content); rejected files tracked via ingest_file_rejected_total metric
50 MiB File-Size Guard: Configurable per-file size limit for text-based formats to prevent memory exhaustion
9 Chunking Strategies: Tokens, sentences, paragraphs, markdown, pages, recursive, semantic (LLM-guided)
Metadata Sidecar: JSON-based ingestion configuration per document
Configurable File Types: ingest_allowed_file_types controls accepted formats with group aliases: office (pdf, docx, pptx, xlsx), text-formats (md, txt, html, csv, json, yaml, xml), code (all code extensions), email (eml), images (png, jpg, jpeg, tiff, tif, bmp, gif, webp), archive (zip), all (everything). Default: pdf

Document Lifecycle Management

Refresh Cadence Scheduler: Automated document staleness detection and notifications
- 4 scheduled jobs: Age tracking, stale detection, grace cleanup, monthly health reports
- Email notifications (SMTP) to content owners with escalation to admins
- Auto-deprecation with configurable grace periods (default: 90 days)
- Archive-before-delete with circuit breaker protection against data loss
Archive Backend: Configurable archival before document deletion
- Filesystem backend with retention policies (default: 365 days)
- Circuit breaker prevents deletions when archival fails
- Batch archival with metadata preservation

Production Ready

Connection Pooling: Postgres (10-20), Qdrant (5-10) with configurable sizing
Caching: Embedding cache (50K/2hr), Retrieval cache (1K/10min) with hit rate metrics
Async Jobs: Background ingestion workers for large batches
Rate Limiting: Token bucket (50 RPS / 100 burst)
Cost Tracking: Per-request cost metrics with cached input discount and reasoning token accounting
License Audit: Background self-verification with SHA-256 event chain and structured audit trail
Re-embedding: Admin endpoint for mass re-embedding after model changes with drift tracking

Quality Assurance

Evaluation Framework: Recall@k, MRR, NDCG, MAP metrics with drift detection
Red-team Testing: 29 adversarial security tests across 6 categories (95%+ pass rate)
Hallucination Prevention: Provenance validator with claim-level grounding, retry logic, and abstain behavior
~2,400 Unit/Integration Tests: Comprehensive workspace-wide test coverage
Smoke Tests: 18 validation suites (milestones, auth, observability, chat, retrieval, guardrails, refresh, validator, and more)

Architecture

Apex Accelerator is built as a layered, API-first system with clear separation of concerns:

┌───────────────────────────────────────────────────────────────────────┐
│                         Client Layer                                  │
│  Next.js Web Client • rag-cli • HTTP/REST clients                     │
└───────────────────────────────────────────────────────────────────────┘
                                   ↓
┌───────────────────────────────────────────────────────────────────────┐
│                    HTTP API Layer (Axum)                              │
│  REST Endpoints • OpenAPI/Swagger • CORS • Rate Limiting              │
│  Middleware: Auth, License Guard, Tenant Extraction, Tracing          │
└───────────────────────────────────────────────────────────────────────┘
                                   ↓
┌───────────────────────────────────────────────────────────────────────┐
│                 Authentication, Authorization & Licensing             │
│  Auth: None, API Key, OIDC (Keycloak)  •  JWT & Tenant Claims         │
│  License: PASETO v4.public (Ed25519)  •  Self-Audit Loop  •  Metrics  │
└───────────────────────────────────────────────────────────────────────┘
                                   ↓
        ┌──────────────────────────┴───────────────────────────┐
        ↓                                                      ↓
┌───────────────────────┐                         ┌────────────────────────┐
│  Ingestion Pipeline   │                         │  Query Pipeline        │
│  (rag-core)           │                         │  (agent-core)          │
├───────────────────────┤                         ├────────────────────────┤
│ • Multi-Format Extract│                         │ • Query Classification │
│ •  (15 formats/57 ext)│                         │ • Tool Routing         │
│ • 9 Chunking Strats   │                         │ • Graph Orchestration  │
│ • Embedding (cache)   │                         │ • Fallback Handling    │
│ • Batch Storage       │                         │   ↓ uses               │
│ • Async Jobs          │                         │  (rag-core)            │
└───────────────────────┘                         ├────────────────────────┤
        ↓                                         │ • Hybrid Retrieval     │
        │                                         │ • Context Assembly     │
        │                                         │ • Guardrails           │
        │                                         └────────────────────────┘
        ↓                                                      ↓
┌───────────────────────────────────────────────────────────────────────┐
│                         Storage Layer                                 │
│  Postgres: Metadata, Users, Sessions, Conversations, Jobs             │
│  Qdrant: Dense Vectors, BM25 Sparse Vectors, Payloads                 │
└───────────────────────────────────────────────────────────────────────┘
                                   ↓
┌───────────────────────────────────────────────────────────────────────┐
│                    Observability & Monitoring                         │
│  Prometheus (metrics) • Grafana (dashboards) • Jaeger (traces)        │
│  Loki (logs) • Fluent Bit (log shipping) • Langfuse (LLM obs)         │
│  Alertmanager (27 SLO rules) • Custom provenance events               │
└───────────────────────────────────────────────────────────────────────┘

Key Architectural Principles

API-First: All functionality exposed via REST API; no direct CLI database access
Separation of Concerns: Agent orchestration (agent-core) separated from RAG primitives (rag-core)
Async Runtime: Tokio-based async throughout (database, HTTP, embeddings)
Error Context: Anyhow-based error chains with operation-specific context
Tenant Isolation: All operations scoped by tenant (header or JWT claim)
Observability by Default: Every request traced with OpenTelemetry spans
Type Safety: Rust's type system prevents common bugs (no nulls, exhaustive matching)
No Panics: All errors returned as Result types; .expect() forbidden in production
Offline Licensing: PASETO v4.public tokens verified locally — no license server call-home

Quick Start

For detailed setup instructions, see Quick Start Guide

Prerequisites

Rust 1.93+ (edition 2024)
Docker & Docker Compose
PDFium library (included in tools/vendor/pdfium/)
Optional: Tesseract for OCR — PDF fallback and image extraction (brew install tesseract on macOS)

Installation

# Clone repository
git clone https://github.com/HendrikReh/projectAlpha.git
cd projectAlpha

# Start dependencies (Postgres, Qdrant, observability stack)
just up

# Run server with mock embedder (no OpenAI API key needed)
just run-server-mock

# Or with OpenAI embeddings
export OPENAI_API_KEY=sk-...
just run-server

# Or with full observability (OpenTelemetry tracing)
just run-server-otel

# Start the web client (in another terminal)
just web-dev

Ingest Your First Document

# Ingest documents from directory (15 formats supported — controlled by ingest_allowed_file_types)
cargo run -p rag-cli --bin rag-cli -- ingest \
  --dir data/rag_import/security \
  --collection hybrid_docs \
  --server http://127.0.0.1:8080

# Dry run (extract/chunk only, no persistence)
cargo run -p rag-cli --bin rag-cli -- ingest \
  --dir data/rag_import/security \
  --dry-run

# Preview chunking strategies locally (no server required)
cargo run -p rag-preview -- \
  --file data/rag_import/security/2511.03675v1.pdf \
  --chunking-strategy tokens,sentences,markdown

Query via API

# Health check
curl http://localhost:8080/health

# Hybrid search
curl -X POST http://localhost:8080/search \
  -H "Content-Type: application/json" \
  -H "x-tenant: default" \
  -d '{
    "query": "transformer architecture",
    "top_k": 10,
    "mode": "hybrid"
  }'

# View metrics
curl http://localhost:8080/metrics | grep ingest_

Components

Core Crates

Crate	Purpose	Description
agent-core	Agent orchestration	YAML-driven agent specs, graph-flow execution, ReAct loops, 8 specialized tools
rag-core	Core library	Multi-format extraction (15 formats / 57 extensions), chunking, embeddings, storage (Postgres + Qdrant), evidence packs, prompt catalog
rag-server	REST API	Axum server with rate limiting, multi-tenant auth, observability
rag-cli	CLI client	Remote ingestion via HTTP API
rag-chunking	Chunking library	9 strategies: tokens, sentences, paragraphs, markdown, semantic, etc.
rag-preview	Local preview	Standalone CLI for chunking experimentation (no server required)
rag-eval	Evaluation	Retrieval metrics (Recall@k, MRR, NDCG, MAP) + red-team tests (29 cases)
rag-evidence	Evidence packs	Evidence pack export, Ed25519 signing/verification, audit reports
rag-license	Licensing	PASETO v4.public license parsing, audit engine, feature gating
rag-notifications	Email alerts	SMTP notifications for document refresh cadence
rag-sbom	SBOM generation	Compile-time CycloneDX SBOM, runtime enrichment, Ed25519 signing, SPDX conversion
rag-client	HTTP client	Thin tenant-aware HTTP client for server API
obfuscate-macros	Anti-RE	Compile-time XOR string obfuscation proc-macro (`#[obfuscate_strings]`)
test-support	Test utilities	`spawn_app(Router)` for ephemeral Axum test servers
loc-checker	Code quality	Lines-of-code checker for enforcing file size limits
license-gen	Vendor tooling	Generate keypairs, sign/verify/inspect PASETO v4.public license files (not shipped in production)
agent-spec-validator	Validation	YAML schema validator for agent specification files
check-openai-key	Diagnostics	Quick OpenAI API key validity checker

Desktop Tools

Tool	Purpose	Description
prompt-editor-tauri	Prompt lifecycle	Tauri + Leptos app for prompt catalog management (list/create/revise/publish/unpublish/rollback)
evidence-pack-viewer-tauri	Evidence inspection	Dual-mode (desktop + web) viewer with signature verification, 6 views, client-side ZIP parsing
license-manager-tauri	License management	Dashboard, create/verify/browse licenses, keypair management, TOML-backed settings
agent-viewer-tauri	Agent visualization	Tauri + Leptos desktop app for browsing agent YAML configs, graph visualization
metadata-editor-tauri	Metadata editing	Tauri + Leptos app for editing document metadata sidecars

API Endpoints

Public Endpoints (No Authentication)

Endpoint	Method	Description
`/health`	GET	Liveness probe
`/license/health`	GET	License validity probe (200 valid/demo, 503 otherwise)
`/metrics`	GET	Prometheus metrics
`/openapi.yaml`	GET	OpenAPI 3.0 specification
`/docs`	GET	Interactive Swagger UI

Protected Endpoints (Require `x-tenant` Header)

Endpoint	Method	Description
`/ingest`	POST	Trigger document ingestion (batch)
`/ingest/async`	POST	Async ingestion job (returns job_id)
`/ingest/jobs/:job_id`	GET	Get async job status
`/search`	POST	Hybrid search with filters
`/chat`	POST	RAG chat with conversation tracking and optional provenance validation
`/chat/suggestion-click`	POST	Track suggestion click analytics
`/chat/suggestion-feedback`	POST	Track suggestion feedback
`/collections`	GET	List available Qdrant collections
`/collections/:collection/stats`	GET	Collection statistics (document/chunk counts)
`/collections/:collection/languages`	GET	Language distribution for collection
`/collections/:collection/recompute-avgdl`	POST	Recompute BM25 average document length
`/license/status`	GET	Full license details (edition, features, limits, expiry)
`/status`	GET	Document/chunk counts, endpoint info
`/chunks/preview`	POST	Preview chunking before ingestion
`/retrieval/plan/execute`	POST	Execute dynamic retrieval plan
`/evidence-packs`	GET	List exported evidence packs
`/evidence-packs/:filename`	GET	Download an evidence pack (signed ZIP)

Prompt Catalog Endpoints

Endpoint	Method	Description
`/prompts`	GET	List prompt definitions
`/prompts`	POST	Create a new prompt definition
`/prompts/resolve`	GET	Resolve prompt by name (with provenance headers)
`/prompts/:name`	GET	Get prompt definition details
`/prompts/:name/content`	GET	Get rendered prompt content
`/prompts/:name/revisions`	POST	Add a new revision
`/prompts/:name/publish`	POST	Publish current revision
`/prompts/:name/unpublish`	POST	Unpublish prompt
`/prompts/:name/rollback`	POST	Rollback to previous revision

Agent Endpoints

Endpoint	Method	Description
`/agents`	GET	List all available agent specifications
`/agents/:agent_id/execute`	POST	Execute an agent graph with query and filters
`/agents/:agent_id/graph`	GET	Get agent graph structure + Mermaid diagram
`/agents/:agent_id/spec`	GET	Get full agent specification including raw YAML

Run Management & Evidence Packs

Endpoint	Method	Description
`/runs`	GET	List agent runs (pagination, status filter)
`/runs`	DELETE	Bulk delete runs by date range or pattern
`/runs/:run_id`	DELETE	Delete a single run
`/runs/:run_id/approve`	POST	Approve a pending checkpoint
`/runs/:run_id/reject`	POST	Reject a pending checkpoint
`/runs/:run_id/export`	GET	Export evidence pack (signed ZIP with audit report)

Metadata Generation

Endpoint	Method	Description
`/metadata/generate`	POST	Generate metadata for documents (LLM-driven)
`/metadata/review`	POST	Review/validate metadata
`/metadata/explain`	POST	Explain metadata field

Admin Endpoints

Endpoint	Method	Description
`/admin/refresh/trigger`	POST	Trigger refresh cadence check
`/admin/refresh/stale`	GET	List stale documents requiring refresh
`/admin/refresh/status`	GET	Refresh scheduler status
`/admin/refresh/deprecate`	POST	Mark document as deprecated
`/admin/refresh/un-deprecate`	POST	Restore deprecated document
`/admin/reindex-bm25`	POST	Reindex BM25 sparse vectors
`/admin/reembed`	POST	Re-embed all vectors (after model change)
`/admin/feedback/export`	GET	Export suggestion feedback data

Note: Admin endpoints may require additional authorization. See Refresh Cadence API for details.

Chat Response with Provenance Validation

When HALLUCINATION_VALIDATOR_ENABLED=true, the /chat endpoint returns claim-level grounding info:

{
  "answer": "The transformer architecture uses...",
  "conversation_id": "uuid",
  "claims": [
    {
      "text": "The transformer architecture uses self-attention",
      "citations": ["chunk-123", "chunk-456"],
      "grounded": true,
      "confidence": 0.95
    }
  ],
  "validation": {
    "overall_grounded": true,
    "ungrounded_count": 0,
    "total_claims": 3,
    "retry_count": 0
  }
}

Example: Execute Agent

# List available agents
curl http://localhost:8080/agents \
  -H "x-tenant: default"

# Execute an agent with a query
curl -X POST http://localhost:8080/agents/deep_research/execute \
  -H "Content-Type: application/json" \
  -H "x-tenant: default" \
  -d '{
    "query": "What are the security implications of transformer architectures?",
    "collection": "hybrid_docs",
    "top_k": 10
  }'

# Get agent graph visualization (Mermaid)
curl http://localhost:8080/agents/deep_research/graph \
  -H "x-tenant: default"

Example: Ingestion with Headers

curl -X POST http://127.0.0.1:8080/ingest \
  -H "Content-Type: application/json" \
  -H "x-tenant: demo-tenant" \
  -H "Authorization: Bearer $INGEST_API_KEY" \
  -H "x-request-id: $(uuidgen)" \
  -d '{
    "paths": ["data/rag_import/security"],
    "collection": "hybrid_docs"
  }'

Configuration

Environment Variables

Configuration uses a two-tier system:

config/app.toml: Non-secret defaults (override via APP_CONFIG_PATH) — sections include [app], [security], [cost], [i18n], [guardrails], [retrieval], [chunking], [hallucination], [refresh], [evidence_pack.signing], [prompt], [metrics], and more
.env: Secrets and explicit overrides

Key Variables

Category	Variable	Default	Description
Auth	`AUTH_MODE`	`none`	`none`, `api-key`, or `oidc`
	`AUTH_API_KEYS`	-	Comma-separated API keys (api-key mode)
	`OIDC_ISSUER_URL`	-	OIDC provider URL (oidc mode)
	`OIDC_AUDIENCE`	-	Expected JWT audience (oidc mode)
	`TENANT_HEADER`	`x-tenant`	Tenant header name
Ingest	`INGEST_ALLOWED_FILE_TYPES`	`pdf`	Comma-separated or group alias: `pdf`, `md`, `office`, `email`, `images`, `archive`, `all`
Limits	`INGEST_BODY_LIMIT_BYTES`	`10485760`	Max request body (10MB)
	`INGEST_MAX_PATHS`	`100`	Max paths per request
	`RATE_LIMIT_RPS`	`50`	Requests per second
	`RATE_LIMIT_BURST`	`100`	Burst capacity
ZIP	`ZIP_MAX_DEPTH`	`3`	Max ZIP nesting depth
	`ZIP_MAX_ENTRIES`	`1000`	Max entries per archive
	`ZIP_MAX_TOTAL_UNCOMPRESSED_MB`	`500`	Global budget across recursion (MiB)
	`ZIP_MAX_ENTRY_MB`	`100`	Per-entry size cap (MiB)
	`ZIP_MAX_COMPRESSION_RATIO`	`100`	Max compression ratio (zip bomb guard)
OTEL	`OTEL_ENABLED`	`false`	Enable OpenTelemetry
	`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4317`	OTLP gRPC endpoint
Postgres	`DATABASE_URL`	-	Postgres DSN
	`POSTGRES_MAX_CONNECTIONS`	`10`	Connection pool size
Qdrant	`QDRANT_URL`	`http://127.0.0.1:6334`	Qdrant gRPC endpoint
	`QDRANT_POOL_SIZE`	`5`	Connection pool size
Embeddings	`EMBEDDER`	`openai`	`openai` or `mock`
	`OPENAI_API_KEY`	-	OpenAI API key
	`EMBEDDING_MODEL`	`text-embedding-3-small`	Model name
Agents	`AGENTS_DIR`	`config/agents`	Directory containing agent YAML specs
Langfuse	`LANGFUSE_ENABLED`	`false`	Enable LLM observability
	`LANGFUSE_PUBLIC_KEY`	-	Langfuse public key
	`LANGFUSE_SECRET_KEY`	-	Langfuse secret key
Validator	`HALLUCINATION_VALIDATOR_ENABLED`	`false`	Enable provenance validation
	`HALLUCINATION_VALIDATOR_MAX_RETRIES`	`2`	Max retry attempts
	`HALLUCINATION_VALIDATOR_BACKOFF_MS`	`150`	Backoff between retries
Cost	`COST_MODEL_RATES`	-	JSON map of model cost rates (per-1M tokens)
i18n	`I18N_DEFAULT_LOCALE`	`en`	Default locale for evidence pack formatting
PII	`PII_HASH_SECRET`	-	HMAC-SHA256 secret for PII hashing (required — server panics without it)
Signing	`APEX_SIGNING_KEY`	-	Ed25519 private key for evidence pack signing
License	`LICENSE_MODE`	`demo`	`enforce`, `demo`, or `warn`
	`LICENSE_PATH`	-	Path to PASETO v4.public license file
	`LICENSE_PUBLIC_KEY_PATH`	-	Path to Ed25519 public key PEM
	`LICENSE_PUBLIC_KEY`	-	Base64-encoded public key (alternative to file)
	`LICENSE_CHECK_INTERVAL_SECS`	`300`	Re-verification interval (seconds)
	`INSTALLATION_ID`	-	Installation fingerprint for binding
	`CLUSTER_ID`	-	Cluster fingerprint for binding
Scheduler	`REFRESH_SCHEDULER_ENABLED`	`true`	Enable refresh cadence scheduler
	`REFRESH_NOTIFICATIONS_ENABLED`	`true`	Enable email notifications
	`REFRESH_SMTP_HOST`	`localhost`	SMTP server hostname
	`REFRESH_SMTP_PORT`	`587`	SMTP server port
	`REFRESH_SMTP_USER`	-	SMTP authentication username
	`REFRESH_SMTP_PASSWORD`	-	SMTP authentication password
	`REFRESH_SMTP_FROM_ADDRESS`	-	Email sender address
	`REFRESH_GRACE_PERIOD_DAYS`	`90`	Days before deprecated docs deleted
	`REFRESH_AUTO_HARD_DELETE`	`false`	Auto-delete after grace period
	`REFRESH_ARCHIVE_BEFORE_DELETE`	`false`	Archive documents before deletion
	`REFRESH_ARCHIVE_FILESYSTEM_PATH`	`data/archive`	Archive storage path

Observability

Stack Components

Component	Port	Purpose
Grafana	3000	Dashboard visualization (admin/admin)
Prometheus	9090	Metrics collection & querying
Jaeger	16686	Distributed tracing UI
Loki	3100	Log aggregation
Fluent Bit	-	Log shipping to Loki
Langfuse	3001	LLM observability (optional, self-hosted)
Keycloak	8180	OIDC authentication (admin/admin)

Quick Access

# Fastest local review: start services, run both observability smoke suites, open dashboards
./scripts/tests/review-observability-stack.sh

# Headless verification for CI or repeated local checks
./scripts/tests/review-observability-stack.sh --no-open --exit-after-tests

# Manual flow if you only need the individual commands
just up
just run-server-otel

# Access dashboards
open http://localhost:3000  # Grafana
open http://localhost:16686 # Jaeger
open http://localhost:9090  # Prometheus
open http://localhost:3001  # Langfuse

Pre-configured Grafana Dashboards

Auto-loaded in the "Apex Accelerator" folder:

Tracing Overview — Recent traces with Jaeger links, log correlation
API Health — Request rates, 4xx/5xx errors, rate-limit hits
Chat & Search — Retrieval timings, token usage, chat latency
Ingest Pipeline — P50/P95 latency per stage, error rates, throughput
Hallucination — Citation coverage, abstain/retry rates, guardrail actions
BM25 Retrieval — Sparse retrieval metrics and avgdl drift
SLO — Error budgets, latency objectives, infra health
Evaluation — Recall, MRR, NDCG, red-team and validator quality metrics

Key Metrics

# View all metrics
curl http://localhost:8080/metrics

# Stage latencies
curl http://localhost:8080/metrics | grep ingest_

# HTTP metrics
curl http://localhost:8080/metrics | grep http_

Key Metrics:

ingest_extract_seconds, ingest_chunk_seconds, ingest_embed_seconds — Ingestion stage latencies
search_embed_seconds, search_qdrant_seconds, context_build_seconds — Search latencies
chat_llm_call_seconds — Chat LLM call latency (by model)
chat_prompt_tokens_total, chat_completion_tokens_total — Token counters (by model)
chat_cached_input_tokens_total, chat_uncached_input_tokens_total, chat_reasoning_tokens_total — Token breakdown
chat_request_cost_usd, chat_cost_micro_usd_total, embed_cost_micro_usd_total — Cost tracking
guardrail_check_seconds, guardrail_safety_flagged_total, guardrail_policy_violation_total — Security guardrails
hallucination_rate, citation_coverage, abstain_responses_total — Quality metrics
reembed_documents_total, reembed_duration_seconds, reembed_drift_mean_similarity — Re-embedding
http_requests_total, http_requests_4xx_total, http_requests_5xx_total — Request counters
embedding_cache_hits_total, retrieval_cache_hits_total — Cache performance
ingest_jobs_* — Async job queue metrics
query_rewrite_total, query_intent_classification_total — Query processing
scheduler_job_* — Scheduled job execution (duration, success/failure counts)
scheduler_notifications_sent_total — Email notification delivery metrics
scheduler_archive_* — Archive operation metrics (success, failures, circuit breaker state)
corpus_stale_documents_total, corpus_document_age_days — Corpus freshness
license_valid, license_expiry_seconds_remaining — License state and countdown
license_check_total, license_blocked_total, license_audit_events_total — License enforcement

Provenance Tracing

Every chat request creates a complete trace from source documents to final answer:

rag.request (root span)
  ├─ rag.retrieve → chunk_source events (doc_id, uri, license, acl_tag)
  ├─ rag.context.assembly → citation_link events
  ├─ rag.reason → chain_step, tool_call events
  └─ rag.response → safety_finding events + SHA-256 hash

Use Cases:

Debug answers: See which chunks were retrieved and ranked
License compliance: Query all documents by license field
Security audit: Track ACL violations via acl_tag
Performance: Correlate slow traces with specific pipeline stages
Answer deduplication: Use SHA-256 hash to find duplicates

View traces:

# Get trace_id from response header
curl -i http://localhost:8080/chat ... | grep x-trace-id

# Open in Jaeger
open http://localhost:16686/trace/{trace_id}

See docs/developer/observability.md for complete documentation.

Authentication

Keycloak OIDC Provider

Apex Accelerator uses Keycloak as its identity provider for secure, multi-tenant authentication.

Quick Start

# Start all services including Keycloak
just up

# Access Keycloak admin console
open http://localhost:8180

# Login: admin / admin

Pre-configured Realm

The projectalpha realm is auto-imported with:

Resource	Description
Clients	`next-web` (confidential), `rag-server` (bearer-only)
Roles	`user`, `admin` (composite)
Scopes	`tenant-scope`, `roles-scope` (custom claims)
Test Users	See table below

Test Users

Username	Password	Tenant	Role
`testuser`	`testuser123`	default	user
`adminuser`	`adminuser123`	default	admin
`tenant2user`	`tenant2user123`	tenant2	user

JWT Token Claims

Tokens include custom claims for multi-tenant authorization:

{
  "tenant": "default",
  "roles": ["user"],
  "sub": "user-uuid",
  "preferred_username": "testuser"
}

Web Client Authentication

The Next.js web client uses NextAuth v5 with Keycloak provider:

# Required environment variables (web-client/.env.local)
KEYCLOAK_ISSUER=http://localhost:8180/realms/projectalpha
KEYCLOAK_CLIENT_ID=next-web
KEYCLOAK_CLIENT_SECRET=next-web-secret-change-in-production
AUTH_SECRET=your-32-char-secret-here
NEXTAUTH_URL=http://localhost:3000

See web-client/README.md for detailed authentication configuration.

Testing

Run All Tests

# Full test suite (fmt, clippy, cargo test, doctests)
just test
just test-metrics-validator # Focused rerun for rag-eval metrics-validator tests (already in `just test`)

# Scoped test suites
just test-backend            # Smoke-test backend crates (libraries + server)
just test-frontend           # Smoke-test Tauri desktop app backends
just test-tools              # Smoke-test CLI tools and utilities

# Individual crate tests
cargo test -p rag-core
cargo test -p rag-server
cargo test -p rag-chunking

Smoke Tests

# All smoke tests (requires running server + services)
just smoke-tests

# Unit smoke tests only (no server required - just M5)
just smoke-tests-unit

# AI Compliance Investigator agent example (requires run-server-mock + ai_governance corpus)
just smoke-ai-compliance-investigator

# One-command observability review: starts OTEL server, runs both observability smoke suites, opens dashboards
./scripts/tests/review-observability-stack.sh

# Individual milestone smoke tests
just smoke-m2              # Milestone 2: Extraction & Chunking
just smoke-m3              # Milestone 3: Storage & Retrieval
just smoke-m4              # Milestone 4: Context & Guardrails
just smoke-m5              # Milestone 5: Agent Graphs
just smoke-observability   # Observability & Tracing
just smoke-auth            # Authentication & Authorization
just smoke-provenance      # Provenance Tracing (OTEL spans)
just smoke-validator       # Provenance Validator (grounding, retries, abstain)
just smoke-corpus-stats    # BM25 Corpus Statistics
just smoke-chat            # Chat Endpoint Validation
just smoke-retrieval       # Retrieval Quality + Latency
just smoke-multi-collection # Multi-Collection Search
just smoke-async-jobs      # Async Job Lifecycle
just smoke-refresh-workflow # Document Refresh Cadence
just smoke-guardrails-classifiers # Guardrail Classifiers
just smoke-suggestions     # Suggestion Analytics
just smoke-abstain         # Abstain Behavior
just smoke-ai-compliance-investigator # AI Compliance Agent (mock + corpus)

Integration Tests

# Scheduler integration tests (requires Postgres + SMTP mock server)
RUN_QDRANT_INTEGRATION_TESTS=1 cargo test --test integration_scheduler_smtp -p rag-server

# All integration tests
just integration-tests

SQLx Compile-Time DB Check (Troubleshooting)

rag-core uses sqlx::query! macros, which validate SQL at compile time. If Postgres is not reachable when cargo check/cargo test runs, you may see:

error communicating with database: Connection refused (os error 61)

Recommended flow before workspace checks/tests:

just up
until docker compose -f docker-compose.yml exec -T postgres \
  pg_isready -h 127.0.0.1 -p 5432 -d postgres -U postgres >/dev/null 2>&1; \
  do sleep 1; \
done

Then run your normal command (just test, just check-local, etc.).

For frontend-only local loops where live SQL validation is not needed:

SQLX_OFFLINE=true cargo check -p prompt-editor-frontend -p prompt-editor-tauri

Web Client E2E (Playwright)

# Full suite (requires backend on :8080)
just web-e2e

# Smoke-only UI checks (no backend required)
just web-e2e-smoke

# Debugging modes
just web-e2e-ui
just web-e2e-headed

Notes:

Local Playwright runs start Next.js with E2E_BYPASS_AUTH=true.
The middleware bypass is active only in non-production (NODE_ENV != production).
If Playwright reuses an existing next dev server started without the bypass, tests can redirect to /login. Stop the reused dev server and rerun just web-e2e.
For full E2E (chat.spec.ts), run just run-server-mock in parallel.

Evaluation & Red-Team Testing

# Retrieval evaluation with metrics (Recall@k, MRR, NDCG, MAP)
cargo run -p rag-eval --bin eval-runner -- \
  --dataset data/eval/golden_dataset.json \
  --server http://127.0.0.1:8080 \
  --collection eval_collection \
  --output-json data/eval/eval_results.json

# Drift detection (compare current vs baseline)
cargo run -p rag-eval --bin drift-detector -- \
  --baseline data/eval/baseline.json \
  --current data/eval/eval_results.json \
  --output-json data/eval/drift_report.json

# Red-team security testing (29 adversarial cases)
cargo run -p rag-eval --bin redteam-runner -- \
  --dataset data/eval/redteam_cases.json \
  --server http://127.0.0.1:8080 \
  --collection redteam_collection \
  --output-json data/eval/redteam_report.json

# Automated nightly evaluation (requires running server)
bash scripts/tests/nightly-eval.sh

Documentation

Core Guides

Quick Start Guide - Get up and running in minutes
Testing Guide - Comprehensive testing strategies
DevOps Guide - Deployment, monitoring, operations
Developer Guide - Auth, tenant, request context patterns
Ingestion Guide - Multi-format extraction and chunking
API Testing - Manual curl examples

Operational Guides

Production Hardening Guide - Dev-to-production configuration checklist (auth, secrets, CORS, rate limiting, signing)
Pool Tuning - Connection pool optimization
Caching Guide - Cache configuration and monitoring
Async Ingestion - Background job processing
SLO Runbook - Alert definitions and troubleshooting
Observability Guide - Metrics, tracing, provenance
Refresh Cadence Runbook - Document refresh system operations
Refresh Cadence Migration - Upgrading to refresh cadence system

Specialized Guides

Authentication - Multi-mode auth (None, API Key, OIDC)
Licensing - PASETO v4.public token system, self-audit loop, feature gating
License Deployment - Key management, configuration, renewal procedures
License Operations - Monitoring, alerting, incident response for license events
Configuration Reference - Full configuration reference (cost, i18n, signing, license, etc.)
Evidence Pack - Evidence pack export, digital signatures, locale formatting
ISO 24970 Logging - Structured governance logging
ADR-006: License V2 - Architecture decision record for PASETO licensing
Evaluation - Retrieval quality metrics and drift detection
Red-Team Testing - Adversarial validation framework
Content Owner Guide - Document refresh workflows for content owners
Refresh Cadence API - Admin API for document refresh management

Crate READMEs

agent-core - Agent orchestration, graph-flow execution, ReAct support
rag-core - Core library architecture
rag-server - REST API documentation
rag-cli - CLI client usage
rag-chunking - Chunking strategies
rag-preview - Local preview tool
rag-eval - Evaluation framework
rag-evidence - Evidence pack signing and verification
rag-license - PASETO license system and audit engine
rag-notifications - SMTP email notifications
rag-sbom - CycloneDX SBOM generation, enrichment, and signing
rag-client - Tenant-aware HTTP client
test-support - Test utilities

Vendor Tools

license-gen - Generate keypairs, sign/verify/inspect PASETO license files
agent-spec-validator - Agent YAML schema validator
check-openai-key - OpenAI API key checker

Desktop Tools

prompt-editor-tauri - Prompt catalog lifecycle management
evidence-pack-viewer-tauri - Evidence pack inspection with signature verification
license-manager-tauri - License dashboard and management
agent-viewer-tauri - Agent catalog viewer desktop app
metadata-editor-tauri - Document metadata editor

Development

Common Commands

# Start dependencies
just up

# Stop dependencies
just down

# View logs
just logs

# Run server modes
just run-server              # Standard (requires OPENAI_API_KEY)
just run-server-mock         # Mock embedder (no API key)
just run-server-local        # Lock-friendly standard server run (isolated target dir)
just run-server-mock-local   # Lock-friendly mock server run (isolated target dir)
just run-server-demo         # Demo license mode (no license file needed)
just run-server-otel         # With OpenTelemetry tracing
just restart-server          # Restart (Docker Compose or Kubernetes only)
just web-dev                 # Next.js web client dev server (port 3002)

# Desktop tools
just metadata-editor          # Metadata Sidecar Studio (frontend dev, port 8081)
just metadata-editor-tauri    # Metadata Sidecar Studio (desktop app)
just agent-viewer             # Agent Catalog Viewer (frontend dev, port 8082)
just agent-viewer-tauri       # Agent Catalog Viewer (desktop app)
just prompt-editor            # Prompt Editor (frontend dev, port 8083)
just prompt-editor-tauri      # Prompt Editor (desktop app)
just evidence-viewer          # Evidence Pack Viewer (frontend dev, port 8084)
just evidence-viewer-tauri    # Evidence Pack Viewer (desktop app)
just license-manager          # License Manager (frontend dev, port 8085)
just license-manager-tauri    # License Manager (desktop app)

# Testing
just test                    # Full test suite
just test-backend            # Smoke-test backend crates (libraries + server)
just test-frontend           # Smoke-test Tauri desktop app backends
just test-tools              # Smoke-test CLI tools and utilities
just test-metrics-validator  # Focused rerun for rag-eval metrics-validator tests
just check-local             # Lock-friendly check (isolated target dir)
just clippy-local            # Lock-friendly clippy (isolated target dir)
just test-local              # Lock-friendly full test suite (isolated target dir)
just smoke-tests             # All smoke tests (requires server)
just smoke-tests-unit        # Unit smoke tests (no server)
just integration-tests       # All integration tests

# Code quality
just fmt                     # Format code
just clippy                  # Lint code
just audit                   # Security audit

# Coverage
just coverage                # Unit-test coverage report (no services needed)
just coverage-full           # Full coverage incl. integration tests (requires services)

Cargo Build Lock Contention (Local Dev)

If you see Blocking waiting for file lock on build directory, multiple local Cargo processes are contending for the same target directory (commonly rust-analyzer + terminal commands).

Use these defaults for interactive work:

just check-local
just clippy-local
just test-local
just run-server-local
just run-server-mock-local
just run-server-demo-local    # Lock-friendly demo mode server
just run-server-otel-local    # Lock-friendly otel server

Keep just test as the canonical CI parity command.
Team-shared VS Code settings in .vscode/settings.json set rust-analyzer.cargo.targetDir = "target/rust-analyzer" so IDE builds do not contend with terminal recipes.

Justfile Recipes

See Justfile for the complete list of available commands.

Security

Guardrails

SSRF Protection: Two-layer defense (registration-time + execution-time) blocks private IPs, cloud metadata endpoints, and loopback — with DNS resolution timeout and redirect-following disabled
Prompt Injection Detection: Blocks jailbreak attempts and system prompt leakage
PII Redaction: Optional redaction of SSNs, credit cards, emails
Cross-Tenant Access Prevention: Enforces tenant boundaries at all layers
SQL Safety: Allowlist-based SQL query validation for agent tools
Response Size Limits: Stream-limited response reading for external tool calls prevents memory exhaustion

Red-Team Testing

29 security test cases across 6 categories:

Prompt Injection (8 cases) - Target: 100% blocked
Cross-Tenant Access (4 cases) - Target: 100% blocked
PII Extraction (4 cases) - Target: 100% flagged
Data Exfiltration (4 cases) - Target: 100% blocked
Guardrail Evasion (4 cases) - Target: 100% blocked
Safe Queries (5 cases) - Target: 100% allowed

# Run red-team tests
cargo run -p rag-eval --bin redteam-runner -- \
  --dataset data/eval/redteam_cases.json \
  --server http://127.0.0.1:8080 \
  --collection redteam_collection \
  --tenant redteam-test \
  --output-json data/eval/redteam_report.json

# Expected: 95%+ pass rate, 0 critical failures

Deployment

Docker Compose

# Start all services
docker compose up -d

# Scale server instances
docker compose up -d --scale rag-server=3

# View logs
docker compose logs -f rag-server

Environment Setup

# Copy example environment
cp .env.sample .env

# Edit configuration
vim .env

# Start services
just up
just run-server

Performance

Typical Latencies (MacBook Pro M1)

Operation	Time	Notes
PDF extraction	100-500ms	Depends on PDF size/complexity
Token chunking	10-50ms	Fast, no API calls
Semantic chunking	1-3s	OpenAI API latency
Embedding generation	200-500ms	OpenAI API, batched
Hybrid search	50-200ms	Qdrant query time
Email notification (SMTP)	100-500ms	Per email sent
Health report generation	500-2000ms	Monthly aggregation across all documents

Optimization

Connection Pooling: Postgres (10-20 connections), Qdrant (5-10 clients)
Caching: Embedding cache (50K entries/2hr), retrieval cache (1K entries/10min)
Rate Limiting: Token bucket (50 RPS / 100 burst)
Async Jobs: Background ingestion worker for large batches

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Development Workflow

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Make your changes
Run tests (just test)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is UNLICENSED - Proprietary software for internal use only.

Acknowledgments

OpenAI - Embedding and semantic chunking APIs
Qdrant - Vector database with hybrid search
Keycloak - Open source identity and access management
OpenTelemetry - Distributed tracing infrastructure
Grafana Labs - Observability stack (Grafana, Loki)
CNCF / Jaeger - Distributed tracing
Langfuse - LLM observability platform (self-hosted)

Support

Documentation: docs/
Issues: GitHub Issues
Issue Templates: Feature Request, Bug Report, Performance, Security, Migration, etc.
Discussions: GitHub Discussions

Version: 0.30.1 Last updated: 2026-03-15 Maintained By: hendrik.reh@blacksmith-consulting.ai

Name		Name	Last commit message	Last commit date
Latest commit History 2,652 Commits
.beads		.beads
.cargo		.cargo
.claude		.claude
.codex		.codex
.github		.github
.just		.just
.sqlx		.sqlx
.vscode		.vscode
.well-known		.well-known
config		config
crates		crates
data		data
docs		docs
examples/curation		examples/curation
keycloak		keycloak
schemas		schemas
scripts		scripts
tools		tools
web-client		web-client
.claudeignore		.claudeignore
.coderabbit.yaml		.coderabbit.yaml
.env.sample		.env.sample
.gitignore		.gitignore
.mcp.json		.mcp.json
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Justfile		Justfile
README.md		README.md
ROADMAP.md		ROADMAP.md
SECURITY.md		SECURITY.md
bacon.toml		bacon.toml
cargo		cargo
clippy.toml		clippy.toml
deny.toml		deny.toml
docker-compose.yml		docker-compose.yml
rustfmt.toml		rustfmt.toml

Folders and files

Latest commit

History

Repository files navigation

Apex Accelerator

Table of Contents

Elevator Pitch

The 80% you shouldn't build yourself

Key Features

Retrieval & Search

Observability & Tracing

Authentication & Security

Intelligence & Reasoning

Evidence Packs & Compliance

Cost Tracking

Licensing

Document Processing

Document Lifecycle Management

Production Ready

Quality Assurance

Architecture

Key Architectural Principles

Quick Start

Prerequisites

Installation

Ingest Your First Document

Query via API

Components

Core Crates

Desktop Tools

API Endpoints

Public Endpoints (No Authentication)

Protected Endpoints (Require x-tenant Header)

Prompt Catalog Endpoints

Agent Endpoints

Run Management & Evidence Packs

Metadata Generation

Admin Endpoints

Chat Response with Provenance Validation

Example: Execute Agent

Example: Ingestion with Headers

Configuration

Environment Variables

Key Variables

Observability

Stack Components

Quick Access

Pre-configured Grafana Dashboards

Key Metrics

Provenance Tracing

Authentication

Keycloak OIDC Provider

Quick Start

Pre-configured Realm

Test Users

JWT Token Claims

Web Client Authentication

Testing

Run All Tests

Smoke Tests

Integration Tests

SQLx Compile-Time DB Check (Troubleshooting)

Web Client E2E (Playwright)

Evaluation & Red-Team Testing

Documentation

Core Guides

Operational Guides

Specialized Guides

Crate READMEs

Vendor Tools

Desktop Tools

Development

Common Commands

Cargo Build Lock Contention (Local Dev)

Justfile Recipes

Security

Guardrails

Red-Team Testing

Deployment

Docker Compose

Protected Endpoints (Require `x-tenant` Header)