doc-search-mcp

A local MCP server that indexes PDFs, EPUBs, HTML and text files and makes them searchable via MCP tools. Designed for LLM coding agents that need to query large local document collections without uploading them to the cloud.

Features

• Indexes PDFs (with TOC detection), EPUBs, HTML, Markdown, and plain text
• Keyword search via FTS5/BM25 and semantic search via vector embeddings
• Hybrid search merging both with Reciprocal Rank Fusion
• Documents organized into named categories
• Same content at multiple paths shares one index record (deduplication by checksum)
• Async indexing, one job per file, with progress tracking via job IDs
• Re-running an index is cheap and idempotent: unchanged files cost one stat
• Startup health checks with per-file change detection
• Accessible over a local network via SSE transport

Install

git clone git@github.com:nburns/doc-search-mcp.git
cd doc-search-mcp
uv sync
uv run doc-search-mcp

Requires Python 3.10+. All dependencies are installed by default — no extras needed.

Connect to Claude Code

Start the server, then register it:

claude mcp add --transport sse --scope user doc-search http://localhost:8080/sse

Or, to share it with everyone working in a repo, commit a .mcp.json at that repo's root:

{
  "mcpServers": {
    "doc-search": {
      "type": "sse",
      "url": "http://localhost:8080/sse"
    }
  }
}

Use the machine's hostname or IP instead of localhost if connecting from another machine.

In a new Claude Code session, run /mcp to confirm the server is connected and its tools are listed.

No auth or TLS. Only expose on trusted local networks.

Configuration

Config file lives at ~/.doc-search/config.toml. All settings are optional — defaults work out of the box.

[server]
transport = "sse"           # sse | stdio
host = "0.0.0.0"
port = 8080

[storage]
backend = "sqlite"          # sqlite is the only implementation
db_path = "~/.doc-search/index.db"

[embeddings]
backend = "auto"            # auto | fastembed | sentence-transformers | ollama | none
model = "nomic-ai/nomic-embed-text-v1.5"
ollama_url = "http://localhost:11434"
# cache_dir = "~/.doc-search/models"   # unset = fastembed's temp dir, re-downloaded on reboot
enable_cpu_mem_arena = false           # true trades ~30GB memory spikes for throughput

[search]
default_mode = "auto"       # auto | keyword | semantic | hybrid
default_limit = 10
rerank = false

[chunking]
target_tokens = 400
overlap_tokens = 50
max_toc_depth = 2
max_file_size_mb = 500

[performance]
extraction_workers = 0      # 0 = auto (CPU count)
embedding_batch_size = 32
embedding_queue_size = 256
max_concurrent_jobs = 3
embedding_max_padded_tokens = 16384    # batch size x longest chunk; this is what bounds peak memory
extraction_timeout = 1800   # seconds per file; covers large OCR jobs

[startup]
check_on_startup = true     # read-only integrity check, runs after the port opens

[profiling]
enabled = true              # one memory sample per interval to the log below
interval_s = 60
path = "~/.doc-search/mem-profile.jsonl"

Environment variable overrides:

Env var	Setting
DOC_SEARCH_DB	storage.db_path
DOC_SEARCH_BACKEND	storage.backend
DOC_SEARCH_POSTGRES_URL	storage.postgres_url (accepted, but no Postgres backend exists yet)
DOC_SEARCH_EMBEDDER	embeddings.backend
DOC_SEARCH_OLLAMA_URL	embeddings.ollama_url
DOC_SEARCH_EMBED_CACHE	embeddings.cache_dir
DOC_SEARCH_CPU_MEM_ARENA	embeddings.enable_cpu_mem_arena
DOC_SEARCH_DEFAULT_MODE	search.default_mode
DOC_SEARCH_PORT	server.port
DOC_SEARCH_MEMPROFILE	profiling.enabled
DOC_SEARCH_MEMPROFILE_INTERVAL	profiling.interval_s
DOC_SEARCH_MEMPROFILE_PATH	profiling.path

Embeddings

Backend auto-detection order:

1. DOC_SEARCH_EMBEDDER env var if set
2. CUDA available → sentence-transformers
3. Default → fastembed (CPU, no PyTorch required)
4. ollama — opt-in via config
5. none — keyword-only search, no vectors

Set backend = "none" to skip embedding entirely and use keyword search only. This makes indexing much faster and is a good default for technical documentation with precise terminology.

MCP Tools

Indexing

Tool	Args	Description
index_path	path, category	Scan a file or directory and queue one job per file needing work. Returns the scan summary and its job IDs. Safe to re-run — unchanged files are skipped, so this is also how you re-index after edits.
remove_document	path, category	Remove a path from the index. Also how you force a rebuild of one file.

Search

Tool	Args	Description
search	query, category, mode, limit, file_type, path_prefix	Search documents.
search_in_document	query, path, category, limit	Search within one file.
get_chunks	ids, category	Retrieve chunks by ID to expand context around a result.

Documents

Tool	Args	Description
list_documents	category	All indexed docs with metadata.
list_categories	—	Categories with doc/chunk counts and last indexed timestamp.
get_stats	category	DB size, document count, chunk count.

Jobs

Tool	Args	Description
get_job_status	job_id	Progress report for one file's job. The 8-character prefix is accepted anywhere a job_id is.
list_jobs	status, path_prefix, limit	Live jobs merged with DB history. Filter by path_prefix to follow a single import.
cancel_job	job_id	Cancel one queued or running job. There is no bulk cancel — restart the server to stop a large import.
resume_job	job_id	Re-run a failed or cancelled job. Files already indexed are skipped.

Maintenance

Tool	Args	Description
check_index	category	Full health report — changed, missing, and unindexed files.
acknowledge_warnings	ids	Suppress known warnings until next startup.
get_config	—	Current config and active backend. Read-only; edit config.toml and restart to change it.
inspect_pdf	path	Per-page extraction stats for one PDF: which pages carry text, character/word counts, TOC structure. Use to check coverage before or after indexing.

Supported Formats

Format	Notes
PDF	TOC-aware chunking; falls back to per-page if no TOC. Image-only PDFs are automatically OCR'd via tesseract if available.
EPUB	Per-chapter extraction
HTML	.html/.htm; one section per <h1>-<h6>. Scripts, styles and site furniture (nav, header, footer, aside, form) are dropped. Both suffixes store file_type = html
Markdown	Heading-aware splitting on ##
Plain text	Paragraph block splitting

OCR for image-only PDFs

If a PDF has no text layer, the extractor automatically falls back to OCR using tesseract. Install the system binary to enable it:

sudo apt install tesseract-ocr        # Debian/Ubuntu
brew install tesseract                 # macOS

Without tesseract, image-only PDFs fail with a clear error rather than silently producing an empty index entry.

Running it as a service

deploy/install.sh installs three system units (not user units — a user manager is torn down at logout, which took the server offline for 24 hours once):

Unit	Role
doc-search-mcp.service	The server, running as the doc-search system account
doc-search-sweep.timer	Fires every 30 minutes, Persistent=true
doc-search-sweep.service	Oneshot: calls index_path over the watched tree, exits once jobs are queued

To watch another tree, add an ExecStart= line to the sweep unit.

CLI helpers

mcp_call.py calls a single tool against a running server (DOC_SEARCH_URL, DOC_SEARCH_TIMEOUT override the target). The Makefile wraps the common ones:

make import SRC=~/books CATEGORY=reading   # index a tree (SRC=, never PATH= — Make inherits PATH)
make list CATEGORY=reading                 # list indexed documents
make jobs STATUS=running                   # list jobs; SRC=<prefix> filters to one import
make status JOB=1a2b3c4d                   # progress for one job (8-char prefix is fine)
make sweep                                 # run the incremental sweep now
make memprofile N=20                       # summarise the last N memory samples
make help                                  # all targets

Dev

uv run pytest                                # tests
uv run ruff check src                        # lint
uv run ruff format src                       # format
uv run --with mypy mypy src/doc_search_mcp/  # type check
uv run pyright                               # second type check

Both type checkers are clean; keep them that way.

Run the server from a checkout:

uv run doc-search-mcp              # production mode
uv run doc-search-mcp --reload     # auto-reload on source changes
uv run doc-search-mcp --port 9090  # override port

AGENTS.md covers the architecture: the job model, the shared embedding pipeline, and the memory findings behind the current defaults.