A retrieval-augmented tool-selection layer for LLM agentic systems. Given a user query and a pool of hundreds or thousands of callable tools, it returns the small subset (typically 15–20) most relevant to that query — before those tool definitions ever reach the LLM's context window.
Includes a fully working reference implementation (example/) built on a simulated university management system with 191 tools across 8 MCP servers, used to validate the curator against a 282-case hand-labeled evaluation suite.
For the complete architecture, implementation details, evaluation methodology, and design decisions, visit:
https://yashwanth-r19.github.io/rag-tool-curator/
- Why This Exists
- How It Works
- Repository Structure
- Installation
- Quick Start
- Using the Curator in Your Own Project
- Configuration
- Improving Retrieval Accuracy
- Evaluation
- Architecture Notes
- Further Reading
LLM tool-calling accuracy degrades measurably once the number of available tools exceeds roughly 20–30, and degrades sharply beyond a few hundred. Independent of accuracy, every tool definition included in a prompt costs tokens — a pool of 3,000 tools at ~200 tokens per definition consumes 600,000 tokens per request, which exceeds the context window of essentially every production LLM.
This package sits between your tool registry and your LLM call. It performs retrieval — not generation — to narrow the candidate set down to a size the LLM can reason about reliably, using the same dense + sparse hybrid retrieval techniques used in production RAG document-search systems, applied here to tool descriptions instead of documents.
Measured impact (reference implementation, 282-case eval suite):
| Tool pool size | Without curator | With curator (top-15) | Token reduction |
|---|---|---|---|
| 87 tools (student persona) | ~17,400 tokens/query | ~3,000 tokens/query | 83% |
| 191 tools (admin persona) | ~38,200 tokens/query | ~3,000 tokens/query | 92% |
| 3,000 tools (enterprise scale) | ~600,000 tokens/query (exceeds context window) | ~4,000 tokens/query | 99.3% |
The curator (tool_curator/hybrid_rag_curator.py) runs two independent retrieval passes per query and fuses the results:
-
Dense retrieval —
BAAI/bge-small-en-v1.5(a retrieval-tuned sentence embedding model, CPU-only, 130 MB) encodes both the query and every tool description into 384-dimensional vectors. A FAISSIndexFlatIPindex performs exact cosine-similarity search. This pass captures semantic equivalence — e.g. matching "raise a formal complaint" to a tool namedsubmit_grievancewith no lexical overlap. -
Sparse retrieval —
BM25Okapiscores tools by exact keyword overlap with the query. This pass captures precise terminology matches — e.g. matching "CGPA" directly toget_cgpa— that dense retrieval alone can sometimes dilute. -
Fusion — results from both passes are combined via Reciprocal Rank Fusion (RRF, k=60), a scale-invariant rank-based fusion method that requires no manual weight tuning between the two retrieval scores.
Additional mechanisms layered on top of this core:
- Compound query decomposition — queries containing multiple distinct intents ("check my attendance and also raise a grievance") are split and retrieved independently, then merged, so neither intent is diluted by the other.
- Name-token anchoring — tools whose name closely matches query tokens are guaranteed inclusion regardless of their fused rank score.
- Adaptive result expansion — when retrieval confidence is low (ambiguous or very short queries), the result set automatically expands to reduce the risk of omitting the correct tool.
- Disk-cached embeddings — tool embeddings and the FAISS index are persisted to disk and invalidated automatically (via content hash) whenever the tool pool or domain synonym file changes. Cold start for ~200 tools is a few seconds; subsequent starts load in under 100 ms.
The curator package contains no domain-specific logic. All domain knowledge — synonyms, terminology, dependency relationships between tools — is supplied externally via JSON configuration files at runtime.
.
├── tool_curator/ # The reusable retrieval engine — domain-agnostic
│ ├── __init__.py # Public API: get_curator(), load_tool_context(),
│ │ # load_dependency_rules(), apply_dependency_rules()
│ ├── curator_interface.py # Abstract base class defining the curator contract
│ ├── hybrid_rag_curator.py # FAISS + BM25 + RRF implementation
│ ├── suggestion_validator.py # Detects LLM-hallucinated tool names in free-text responses
│ └── terminal_display.py # Minimal, dependency-light curator selection logging
│
├── accuracy_boosters/ # Annotated templates for domain-specific tuning
│ ├── domain_synonyms.template.json
│ ├── dependency_rules.template.json
│ └── equivalences.template.json
│
├── example/ # Full reference implementation
│ ├── main.py # Entry point — prompts for persona/user, starts the session
│ ├── settings.py # Configuration: model paths, curator parameters, server registry
│ ├── domain_synonyms.json # University-domain synonym set (see Improving Accuracy below)
│ │
│ ├── orchestrator/ # Application-level orchestration — NOT part of the reusable plugin
│ │ ├── agent_orchestrator.py # Main conversation loop: connects MCP servers, runs the
│ │ │ # curator per turn, dispatches Gemini tool calls
│ │ ├── multilingual_normalizer.py # Stage 0: translates any-language query to English before the curator
│ │ ├── persona_access_policy.py # Role-based tool visibility (student/faculty/admin/executive)
│ │ ├── mutation_guard.py # Confirmation gate for state-mutating tool calls
│ │ ├── hallucination_guard.py # Fuzzy correction for slightly-malformed tool names from the LLM
│ │ ├── usage_logger.py # Per-turn and per-session token usage tracking
│ │ └── terminal_display.py # Full rich-terminal UI for the example application
│ │
│ ├── mcp_servers/ # 8 FastMCP servers exposing 191 tools total
│ │ ├── academics_mcp_server.py
│ │ ├── student_services_mcp_server.py
│ │ ├── faculty_mcp_server.py
│ │ ├── finance_mcp_server.py
│ │ ├── infrastructure_mcp_server.py
│ │ ├── admin_analytics_mcp_server.py
│ │ ├── placement_mcp_server.py
│ │ ├── library_mcp_server.py
│ │ └── stdio_transport_patch.py # stdio transport compatibility shim
│ │
│ ├── mock_db/
│ │ └── university_mock_database.py # In-memory mock dataset shared by all 8 servers
│ │
│ ├── eval/
│ │ └── MCP Evaluation Sheet - BGE _ FAISS _ BM25 Tool Curator.xlsx
│ ├── tool_curator/
│ │
│ ├── requirements.txt
│ └── .env
│
├── docs/
│ ├── index.html
│ └── diagrams/
│
├── requirements-core.txt
├── README.md
├── GUIDE_ACCURACY.md # Step-by-step accuracy tuning methodology
└── GUIDE_INTEGRATION.md # Integration guide for existing MCP-based systems
pip install -r requirements-core.txtThis installs only what tool_curator/ needs: sentence-transformers, faiss-cpu, rank-bm25, numpy. No web framework, no LLM SDK, no example-specific dependencies.
cd example/
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txtRequirements: Python 3.12+. No GPU required — all retrieval runs on CPU.
cd example/
python main.pyYou will be prompted for a persona and user ID:
1. Student (e.g. 21CS045)
2. Faculty (e.g. FAC-CS-042)
3. Admin (ADMIN-001)
4. Executive (EXEC-001)
Once connected, every query you type will print a curator selection line before the model responds:
◈ Curator 87 → 15 tools
get_cgpa, get_sgpa_history, get_student_transcript, ...
This confirms the curator narrowed the persona's full tool pool down to the subset actually passed to the LLM for that turn.
Useful in-session commands: /tools (list all tools visible to the current persona), /stats (session token usage summary), exit.
The entire public surface area is three functions and one class, imported from tool_curator:
from tool_curator import get_curator, load_tool_context
curator = get_curator(
mode = "rag",
top_k = 20,
model_name = "BAAI/bge-small-en-v1.5",
index_dir = "/path/to/cache",
)
# Build the index once, when your tool pool is known (e.g. at session start)
synonyms = load_tool_context("/path/to/domain_synonyms.json")
curator.build_index(your_tools, extra_context=synonyms)
# your_tools: list[tuple[str, str]] — (tool_name, tool_description) pairs
# Per query
selected = curator.select(user_query, your_tools)
# selected: same format, trimmed to top_k entriesyour_tools should be the result of applying your own access-control/visibility filtering first — the curator never widens a tool pool, it only narrows it. See GUIDE_INTEGRATION.md for a worked example of wiring this into an existing FastMCP-based orchestration layer, including exact insertion points for session-scoped index builds and per-message selection.
All curator behavior is controlled through the get_curator() parameters:
| Parameter | Description | Default |
|---|---|---|
mode |
"rag" (hybrid retrieval) or "none" (bypass — all tools forwarded unfiltered) |
— |
top_k |
Number of tools returned per query | 15 |
model_name |
Embedding model. BAAI/bge-small-en-v1.5 recommended; all-MiniLM-L6-v2 for a smaller/faster alternative |
BAAI/bge-small-en-v1.5 |
index_dir |
Directory for the persisted FAISS index and embeddings. Empty string disables disk caching. | "" |
In the example application, these are set in example/settings.py as TOOL_CURATOR_MODE, TOOL_CURATOR_TOP_K, TOOL_CURATOR_EMBEDDING_MODEL, and TOOL_CURATOR_INDEX_DIR.
domain_synonyms.json is the primary lever for tuning retrieval to your domain's vocabulary. It maps tool names to additional keyword text that gets appended to the tool's description before indexing — without requiring any change to the tools themselves:
{
"submit_grievance": "raise complaint file issue lodge protest formal complaint",
"get_account_balance": "check balance how much money available funds statement"
}This file is loaded via load_tool_context() and passed to build_index() as extra_context. The curator does not interpret this content — it is purely text concatenated into the index.
For workflows where one tool's correctness depends on another being called alongside it (e.g. an eligibility check that requires both an attendance lookup and a balance lookup), dependency_rules.json force-includes companion tools whenever a trigger tool is selected:
{
"get_exam_eligibility": ["get_attendance_summary", "get_fee_dues"]
}from tool_curator import load_dependency_rules, apply_dependency_rules
rules = load_dependency_rules("/path/to/dependency_rules.json")
selected = apply_dependency_rules(curator.select(query, pool), pool, rules)Retrieval quality on a new domain is primarily a function of how well domain_synonyms.json covers the vocabulary your real users actually use — not the curator's internal logic, which requires no tuning. The full methodology, including the iterative evaluation loop used to reach the accuracy figures above, is documented in GUIDE_ACCURACY.md. In summary:
- Write a labeled test set mapping representative user queries to the exact tool(s) that should answer them.
- Run
eval/curator_eval.pyagainst that test set to get a quantified recall baseline. - For each failure, add the missing terminology to
domain_synonyms.json— including the inverse framing (e.g. both "eligible" and "barred" for an eligibility-check tool). - Re-run the evaluation and iterate until recall stabilizes.
Annotated starting templates for all three configuration files are provided in accuracy_boosters/.
The Tool Curator was evaluated using a manually curated ground-truth evaluation suite consisting of 281 user queries spanning multiple personas, tool categories, and single-tool as well as multi-tool workflows. For every query, the expected tool invocation(s) were predefined and compared against the actual tool(s) selected by the system.
Each evaluation case was classified into one of the following categories:
- Correct – The system selected the correct tool(s) in the correct order and produced the expected output.
- Partially Correct – The system selected some, but not all, of the expected tools or only partially matched the intended workflow.
- Wrong – The system invoked incorrect tool(s), resulting in an incorrect or irrelevant response.
- Skipped – The system failed to invoke an available tool and instead indicated that no suitable tool existed.
The overall Accuracy Score is computed as:
Accuracy Score = {Correct} + 0.5 * {Partially Correct} / {Total Queries}
| Metric | Value |
|---|---|
| Total Evaluation Queries | 281 |
| Correct | 262 |
| Partially Correct | 9 |
| Wrong | 3 |
| Skipped | 7 |
| Correct Response Rate | 93.2% |
| Overall Accuracy Score | 94.8% |
The evaluation demonstrates that the BGE + FAISS + BM25 based Tool Curator reliably retrieves the appropriate tools for a wide variety of user requests while maintaining high routing accuracy across diverse university MCP workflows.
The complete evaluation dataset is available in MCP Evaluation Sheet - Evaluation Sheet
- The curator operates purely on
(name, description)pairs and has no awareness of MCP, FastMCP, transport protocol, or any specific LLM SDK. It can be used identically whether tools are reached over stdio, HTTP, or any other transport. tool_curator/terminal_display.pyis a minimal, dependency-light logging helper intended for use directly within the plugin context (e.g. when embedding the curator in non-interactive services). It is distinct fromorchestrator/terminal_display.py, which is the full interactive terminal UI built for the example application and depends onrich.- Persona/role-based access control is intentionally kept outside the curator (
orchestrator/persona_access_policy.pyin the example). The curator should always receive an already-authorized tool pool and will never select a tool outside the pool it was given. orchestrator/multilingual_normalizer.pyimplements a Stage 0 pre-curator translation layer for the example application. It translates any-language user queries to English via a single Gemini call before they reach the curator, soBAAI/bge-small-en-v1.5and BM25 always operate on their training distribution. Pure-ASCII queries bypass the API entirely. Thetool_curator/package itself is language-agnostic and unchanged — the normalizer is an orchestration-layer concern, not a retrieval one.orchestrator/mutation_guard.pyandorchestrator/hallucination_guard.pyaddress two distinct failure modes commonly seen when integrating tool-calling LLMs: tools that mutate state being called without explicit confirmation, and tool names being called with minor spelling/formatting drift from their registered names. Neither is required for the curator to function — both are independent hardening layers in the example orchestrator.
GUIDE_ACCURACY.md— methodology for tuning retrieval accuracy on a new domainGUIDE_INTEGRATION.md— integrating the curator into an existing multi-server MCP orchestration system