OpenWRT RAG Pipeline

A scalable, multimodal Retrieval-Augmented Generation (RAG) system specifically designed for processing OpenWRT documentation. This pipeline leverages the raganything (LightRAG) framework, Docling for semantic chunking, numpy for fast array operations, and incorporates Qdrant for vector storage, Neo4j for the knowledge graph, Redis for key-value caching, and MinIO for centralized media storage.

Features & Implemented Architecture

The pipeline has been thoroughly refactored to prioritize stability, efficiency, modular design, and strict static typing. Below are the core features and the reasoning behind their implementation. For a comprehensive technical deep-dive, please refer to ARCHITECTURE.md:

1. Typed Boundary & Strict Static Typing (CRITICAL)

What: The project employs a dual-mode static typing architecture using pyright. The core orchestration layer (main.py and src/core/) operates in # pyright: strict mode. Untyped external ML libraries (raganything and LightRAG) are strictly isolated behind the RAGPipelineAdapter pattern (src/adapters/). The adapter uses # pyright: basic to legally handle untyped logic while exporting only rigorously typed public methods (via @overload for dynamic streaming types). Why: To ensure 100% type safety in the core business logic without resorting to unsafe # type: ignore suppressions. It effectively walls off the "untyped chaos" of external ML libraries, preventing cascading reportUnknownMemberType errors.

2. Y-Shape Pipeline & Document Routing

What: The main.py orchestrator intelligently routes incoming documents based on their file extension. Native text and Markdown files (.md, .txt) are directed to Docling Core (via DoclingAdapter) for semantic chunking. Complex, rich formats like .pdf or .docx are passed directly into the RAGAnything pipeline for full multimodal OCR and vision extraction. Extracted media is automatically uploaded to MinIO, and references are replaced with portable rag-media:// URIs. Why: The initial architecture forced all files through RAGAnything, which resulted in unnecessary OCR overhead and conversion errors when dealing with native text like Markdown. The Y-Shape Pipeline drastically speeds up processing and avoids conversion failures by matching the right tool to the right format.

3. Pre-flight Health Checks (Fail-Fast)

What: Before heavy operations (like massive folder indexing or complex querying), the application performs asynchronous "pre-flight" checks (src/core/health.py). It validates API connectivity (LLM, VLM, and Embeddings), enforces strict dimensional requirements (e.g., verifying that embedding vectors are exactly 1024 dimensions), checks that the embedding adapter strictly returns numpy.ndarray, and tests database readiness (Qdrant, Redis, Neo4j). Why: In previous iterations, a pipeline processing thousands of files could fail hours into execution due to an invalid API key, network issue, rate-limit ban (401 or 429), or incorrect data types. Implementing a Fail-Fast mechanism saves time and computing resources while preventing corrupted states in the vector and graph databases.

4. Hierarchical Configuration

What: The flat configuration model has been upgraded to a strict, two-tier hierarchical system utilizing Pydantic v2.

• .env is exclusively used for sensitive secrets (API keys, URIs, passwords).
• config.yaml is structured into nested blocks (core, minio, multimodal_pipeline, markdown_pipeline), avoiding hardcoded defaults for models and endpoints. Supports Qwen thinking mode via LLM_ENABLE_QWEN_THINKING and VLM_ENABLE_QWEN_THINKING flags.
• Mandatory CLI Overrides: Critical resource and isolation parameters (WORKSPACE_NAME, MAX_ASYNC, MAX_WORKERS) have no defaults and must be explicitly provided via CLI for all commands.
• Configuration is accessed securely via a Facade pattern (ConfigAccessor). Why: As the project grew, a flat configuration file became disorganized and prone to silent failures. Nested Pydantic models enforce strict types, catch missing configuration immediately via ValidationError, and decouple secret management from behavioral logic.

5. Centralized Orchestrator & Law of Demeter

What: A single entry point (main.py) replaces multiple disjointed scripts. It provides an intuitive CLI using argparse. It also features a robust Graceful Shutdown mechanism using FastAPI Lifespan. It delegates storage teardown entirely to the Adapter, protecting the shutdown sequence with asyncio.shield() to prevent CancelledError from interrupting the database cleanup. Why: Adheres to the Law of Demeter by preventing the orchestrator from diving into pipeline.lightrag.finalize_storages(). Centralizes graceful shutdown routines, preventing corrupted connections or endless retries during critical exceptions.

6. Performance & Resilience Enhancements

What: Added exponential backoff retry mechanisms using tenacity for handling HTTP 429 (rate limit) and 502 (bad gateway) errors across all external API calls. Implemented configurable retry parameters via CoreConfig. Scoped asyncio.Semaphore correctly and utilized HTTPClientPool bounded by MAX_ASYNC.

7. Dual-Mode API Server & Semantic Reranking

What: Implemented a robust API server utilizing FastAPI for REST endpoints and an MCP Server (Model Context Protocol) powered by the official mcp Python SDK. The MCP server supports remote SSE transport (integrated into FastAPI, protected by JWT). It provides strict Pydantic schemas for tools, robust authentication, protection against Timing Attacks, and explicit DTO serialization. Includes a dynamically configurable semantic reranker (qwen3-rerank) and multiple search strategies (QA, SIMILARITY) exposed via /similar prefix for intelligent context retrieval. Multimodal files are exposed seamlessly to connected AI agents via MCP resources (@server.resource). Why: Enables the system to serve RAG queries securely, provide an Ollama-compatible interface, and operate as an highly optimized MCP server for both local and remote AI agents. Agents receive raw context (avoiding LLM "telephone game") and can request Base64 images directly via the resource protocol, with built-in graceful task cancellation.

Prerequisites

• Python: >= 3.11, < 3.13
• Dependency Manager: Poetry
• Infrastructure: Running instances of Redis, Neo4j, Qdrant, and MinIO (typically via Docker Compose).

Installation

Clone the repository and install the dependencies via Poetry:

poetry install

Configuration

The application requires two configuration files to run.

1. Secrets (.env): Copy the example template and fill in your actual credentials.

   cp .env.example .env

   Required Keys: REDIS_URI, NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD, QDRANT_URL, QDRANT_API_KEY, LLM_API_KEY, VLM_API_KEY, EMBED_API_KEY.

2. Application Settings (config.yaml): Copy the example configuration file. You can modify batch sizes, parsing engines, concurrency limits, reranking settings, and search strategies here.

   cp config.example.yaml config.yaml

Usage

A unified CLI is available via main.py. Place your OpenWRT markdown files in a directory (e.g., ./openwrt_docs_md).

Execute Pre-flight Health Checks

poetry run python main.py check --workspace openwrt --max-async 8 --max-workers 4

Run the Document Indexer

poetry run python main.py index --path ./openwrt_docs_md/ --workspace openwrt --max-async 8 --max-workers 4

Run a Query

poetry run python main.py query --question "How to configure VLAN in OpenWRT?" --workspace openwrt --max-async 8 --max-workers 4

Start the REST API Server (with WebUI)

poetry run python main.py serve --type rest --port 8000 --workspace openwrt --max-async 8 --max-workers 4 --webui

(Starts FastAPI on configured PORT with REST endpoints, Ollama compatibility, and serves the LightRAG WebUI on /) Note: If JWT authentication is enabled, you can log in to the WebUI by pasting the token generated via the trivialtoken command into the login screen.

Generate Trivial JWT Token

poetry run python main.py trivialtoken

(Generates a long-lived static JWT token for external clients or MCP agents, requires TOKEN_SECRET in .env)

List Active Workspaces

poetry run python main.py workspaces list

Delete Workspace Data

poetry run python main.py workspaces delete --name openwrt

(Validates workspace existence and prompts for interactive confirmation. Use --force to bypass the prompt)

Quality Assurance & Testing

This repository enforces an aggressive zero-tolerance policy for linting, formatting, and typing errors. For a complete overview of our isolated, mock-based testing strategy, please refer to TESTING.md. All AI Agents and contributors must verify their code against these tools before submitting.

1. Code Formatting (Black)

All code must be automatically formatted using Black.

poetry run black .

2. Linting (Ruff)

All code must pass Ruff checks. Fix auto-correctable errors with --fix and manually address the remaining (e.g., B904, SIM117). The goal is 0 warnings.

poetry run ruff check --fix .

3. Static Type Checking (Pyright)

The dual-mode strictness ensures safety. Core modules must be strict, adapters must be basic.

poetry run pyright

4. Unit, Integration & Security Tests (Pytest)

Tests are strictly isolated (no real external APIs) using unittest.mock. Ensure 100% test passing.

poetry run python -m pytest tests/ -v

AI Agent Guidelines

If you are an AI programming assistant working on this repository, please read AGENTS.md before making any code modifications. It contains critical instructions on our modular architecture, code style, dependency management, typing policies, and testing constraints.