DocsChatAI

Self-Hosted Docs Assistant You Embed In Your Own Product

An open-source, self-hosted RAG (Retrieval Augmented Generation) chatbot that software vendors embed inside their own product to answer questions from their own documentation. Runs entirely on your infrastructure using local open-weight LLMs via Ollama or vLLM — no OpenAI, no Anthropic, no API keys, no per-token billing, and no customer data leaving your network.

Keywords: self-hosted docs chatbot · open source RAG · on-premise AI assistant · embed AI chat widget · documentation chatbot · private LLM · Ollama RAG · air-gapped AI

_{Answers stream in as they're written, then cite the page they came from.}

---

Contents

1. What it is
2. Quick start — one command
3. System requirements
4. Installation — Docker or Python
5. Usage — get docs in, index, search, ask
6. Choosing and changing the model
7. Configuration
8. Running without a GPU
9. Adding the widget to your product
10. Architecture
11. Project status
12. Paid help · Use cases · Comparison · FAQ

---

What it is

Your customers are inside your product, stuck on a settings screen, and the answer is in your manual somewhere. DocsChatAI puts that answer one question away — branded as your assistant, served from your server.

🏠	Fully self-hosted. You run the server. Docs, questions, and answers never leave your infrastructure.
🚫	No paid AI APIs. Open-weight models via Ollama or vLLM. No API keys, no per-token bills, no vendor lock-in.
🔌	Embedded, not bolted on. A script tag inside your product — not a third-party support bubble with someone else's branding.
🎯	Product-aware. The widget can tell the server which screen the user is on, narrowing retrieval to the docs that matter.
🛡	Safe to expose. Rate limits, a GPU concurrency cap and daily quotas ship on by default — and because the server refuses to answer anything outside your docs, nobody can use your endpoint as a free ChatGPT.
🔓	Your auth, not ours. The widget forwards an opaque context object to middleware you control. We never learn who installed anything — there is no identity database.
♻️	Cheap re-indexing. Per-chunk change detection means a re-crawl only re-embeds what actually changed.

Not a website support bot. No lead capture, no sales triggers, no human handoff. This is in-product documentation help for people who already bought your product.

---

System requirements

Software

Requirement	Version	Needed for
Python	3.11+	Everything (not needed if you use Docker)
Docker	24+	The Docker install path
Ollama or vLLM	current	Writing answers. Not needed to ingest, index or search
Redis	7+	Session memory + semantic cache (optional)
NVIDIA driver + CUDA	12+	GPU inference (optional)

Hardware — which model on which server

Only the answer-writing model cares about your hardware. Embeddings and search always run fine on plain CPU (measured: 5.5 ms per query) — never buy a GPU for them.

Your server	Run this model	Serve with	Answer arrives in	Verdict
No GPU, 8 GB RAM	llama3.2:3b	Ollama	~10 s	✅ Internal tools, evaluation
No GPU, 16 GB RAM	llama3.1:8b	Ollama	~25–40 s	⚠️ Too slow for customers
GPU 8–12 GB	llama3.1:8b	Ollama	~5 s	✅ What most vendors should run
GPU 16–24 GB	llama3.1:8b-instruct-q8_0 or a 14B	Ollama / vLLM	~5 s	Higher answer quality
GPU 24 GB+	llama3.1:8b with batching	vLLM	~5 s, many users at once	Thousands of installs

In one line: no GPU → 3B. Any GPU with 8 GB+ → 8B. Big GPU → still 8B, but switch to vLLM so one card serves many users.

Don't over-buy. A bigger model does not give better answers here, because the model is reading your documentation, not recalling facts. An 8B model given the right page beats a 70B model given the wrong one. If answers are poor, fix retrieval first — run docschatai search and look at what comes back.

Disk

	Size
Embedding model (bge-small, ONNX)	~130 MB
LLM weights (8B, 4-bit)	~5 GB
LLM weights (3B, 4-bit)	~2 GB
Index	~25 MB per 10,000 chunks
Corpus (normalized Markdown)	roughly your docs' text size

---

Quick start

Working docs search in about two minutes, no model server required yet:

git clone https://github.com/kapoordeepanshu/DocsChatAI.git
cd DocsChatAI

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .

docschatai quickstart ./manuals        # a folder of docs, or a URL to crawl

[1/3] Reading docs from ./manuals
      normalized 24 file(s)
[2/3] Building the index (first run downloads a ~130MB model)
      24 document(s), 389 chunk(s) embedded
[3/3] Checking retrieval
      top match: Integrations — Webhooks > Adding an endpoint  (score 0.812)

Ready. Next:
  docschatai search "a question your users actually ask"

Then, any time something isn't working:

docschatai doctor

It checks every moving part and prints the exact command to fix each one — including which model your hardware should run:

  [OK  ] Embeddings     fastembed (ONNX) available
  [WARN] GPU            none detected, CPU only — recommended model: llama3.2:3b
                        -> Not a problem. llama3.2:3b answers in about 10s on CPU.
  [WARN] Model server   not reachable at http://localhost:11434
                        -> ollama serve   (ingest/index/search work without it)
  [OK  ] Corpus         24 document(s) in data/docs
  [OK  ] Index          24 document(s), 389 chunk(s)

Written answers need a model server — that's step 4 below. Everything up to and including search works without one.

---

Installation

The quick start above is the Python path. Full options below — Docker is faster to a working system; Python is better if you're going to modify the code.

Option A — Docker (recommended)

Nothing to install on the host but Docker itself.

CPU (no GPU):

git clone https://github.com/kapoordeepanshu/DocsChatAI.git
cd DocsChatAI

docker compose up -d ollama redis
docker compose exec ollama ollama pull llama3.2:3b     # 3B for CPU

GPU:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d ollama redis
docker compose exec ollama ollama pull llama3.1:8b     # 8B for GPU

GPU needs the NVIDIA Container Toolkit on the host. Verify with:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

Then tell it which model you pulled, and put your docs where it can see them:

cp .env.example .env
# edit .env:  APP_LLM_MODEL=llama3.2:3b   (or llama3.1:8b)

mkdir -p manuals        # drop your docs here; mounted read-only at /manuals

Every command in Usage then runs as:

docker compose run --rm docschatai docschatai <command>

Three named volumes hold state outside the image, so rebuilding never discards your index or re-downloads model weights: app-data, ollama-models, redis-data. Neither Ollama nor Redis is published to the host — they're reachable only on the compose network, which is what you want for a model server.

Option B — Python

git clone https://github.com/kapoordeepanshu/DocsChatAI.git
cd DocsChatAI

python -m venv .venv
source .venv/bin/activate         # Windows: .venv\Scripts\activate

pip install -e .
cp .env.example .env

Then install a model server:

curl -fsSL https://ollama.com/install.sh | sh    # Windows: winget install Ollama.Ollama
ollama pull llama3.2:3b                          # no GPU — see the table above

Embeddings use fastembed (ONNX, ~50 MB, no PyTorch). If you need a model fastembed doesn't carry, pip install -e ".[torch-embeddings]" adds the sentence-transformers backend — but it pulls ~2.5 GB and most deployments never need it.

---

Usage

Four commands, in order. They're identical on both install paths — prefix with docker compose run --rm docschatai if you're using Docker.

1. Get your docs in

docschatai ingest folder ./manuals                    # docs you already maintain
docschatai ingest url https://docs.yourproduct.com    # or crawl a live site

Accepts .md, .html, .pdf, .txt. Navigation, sidebars and footers are stripped automatically. Both routes just fill one folder with Markdown — nothing downstream knows or cares which was used.

2. Build the index

docschatai index

Chunks by heading, embeds locally, writes one data/index.db. Safe to re-run — unchanged chunks keep their vectors, so a nightly re-crawl costs almost nothing.

3. Check retrieval before wiring up a model

docschatai search "how do I verify a webhook signature?"

[1] Integrations — Integrations > Webhooks > Signing Secrets  (score 0.847)
    ./manuals/integrations.md
    Every webhook is signed with your endpoint's secret. Compare the ...
    … +180 more characters (--full to see all)

Do not skip this step. If the right chunk comes back on top, the assistant will answer well. If it doesn't, no model will rescue it — fix ingestion or chunking first. Retrieval sets the ceiling on quality.

4. Ask

docschatai ask "how do I add a webhook endpoint?"

1. Open Settings > Integrations > Webhooks.
2. Click Add Endpoint.
3. Paste your destination URL and select the events to subscribe to.
4. Copy the signing secret shown on creation — it is displayed only
   once. [1]

Sources:
  [1] Integrations — Webhooks
      https://docs.example.com/webhooks

Steps first, then the sources. Streams by default (--no-stream to wait for the whole answer), which matters a lot on CPU — first words appear in about a second instead of after 30 seconds of silence.

Two safeguards you should know about:

• It refuses before calling the model. If nothing retrieves above APP_RETRIEVAL_MIN_SCORE, it answers "not in the documentation" without spending a model call. A model handed weak context still writes something confident.
• It won't invent a procedure. When your docs describe a feature but don't list the clicks, it says so rather than fabricating plausible steps. For a support assistant, invented steps are the worst possible output — users follow them.

5. See what's indexed

docschatai stats

---

Choosing and changing the model

Change the answer-writing model

ollama pull llama3.2:3b          # 1. download it
ollama list                      # 2. confirm the exact tag

Then set it — either in .env:

APP_LLM_MODEL=llama3.2:3b

or as an environment variable for one run:

APP_LLM_MODEL=llama3.2:3b docschatai ask "how do I add a webhook?"

No reindex needed. Swap the answer model as often as you like — it only reads chunks, it doesn't produce them.

Known-good tags: llama3.2:3b, llama3.1:8b, qwen2.5:7b, mistral:7b.

Use vLLM, TGI or llama.cpp instead of Ollama

Any OpenAI-compatible server works:

APP_LLM_BACKEND=openai-compatible
APP_LLM_URL=http://localhost:8000
APP_LLM_MODEL=meta-llama/Llama-3.1-8B-Instruct

Change the embedding model

APP_EMBEDDING_MODEL=BAAI/bge-base-en-v1.5

⚠️ This one requires a full reindex. Vectors from different models aren't comparable, so the store refuses to mix them rather than silently returning nonsense. Delete data/index.db and run docschatai index again.

Worth doing if retrieval is missing obvious matches. bge-base is more accurate than bge-small at ~4× the size — still tiny next to an LLM.

---

Configuration

Every setting is an APP_* environment variable, or a line in .env. Copy .env.example to .env to start. All values below are defaults.

Model / answers

Variable	Default	What it does
APP_LLM_BACKEND	ollama	ollama or openai-compatible
APP_LLM_URL	http://localhost:11434	Where the model server listens
APP_LLM_MODEL	llama3.1:8b	The model that writes answers
APP_LLM_TEMPERATURE	0.1	Low on purpose — faithful reading, not creativity
APP_LLM_MAX_TOKENS	800	Longest answer
APP_LLM_TIMEOUT_S	120	Give up on the model server after this
APP_PRODUCT_NAME	this product	Used in the prompt: "assistant for {name}"

Retrieval

Variable	Default	What it does
APP_RETRIEVAL_TOP_K	5	Chunks used to ground an answer
APP_RETRIEVAL_MIN_SCORE	0.35	Below this, refuse instead of guessing
APP_RETRIEVAL_MIN_CHARS	40	Ignore near-empty stub pages

Typo handling

Variable	Default	What it does
APP_SPELLCHECK_ENABLED	true	Corpus-aware spelling correction
APP_SPELLCHECK_TRIGGER_SCORE	0.70	Only retry queries that scored below this
APP_SPELLCHECK_MIN_LENGTH	4	Never "correct" short words
APP_SPELLCHECK_MIN_FREQ	2	Ignore terms appearing once — they may be typos too

API server and abuse protection

Variable	Default	What it does
APP_ALLOWED_ORIGINS	[]	Origins allowed to embed the widget. Required for browsers. Never *
APP_RATE_LIMIT_ENABLED	true	Turn off only if something in front already limits
APP_RATE_LIMIT_PER_MINUTE	30	Per key, or per IP when there's no key
APP_RATE_LIMIT_PER_DAY	2000	Cost ceiling
APP_MAX_CONCURRENT_GENERATIONS	4	Protects the GPU — how many answers generate at once
APP_MAX_QUEUE_DEPTH	20	Past this, shed load with 503 rather than time everyone out
APP_REQUIRE_SIGNATURE	false	Require an HMAC from your middleware
APP_SHARED_SECRET	—	The HMAC secret
APP_REDIS_URL	redis://localhost:6379/0	Required — sessions and rate limits
APP_SESSION_TTL_S	86400	24 hours, refreshed per turn, then gone
APP_SESSION_MAX_TURNS	20	Turns remembered per conversation

Documents and indexing

Variable	Default	What it does
APP_DOCS_ROOT	data/docs	Where normalized Markdown lands
APP_INDEX_PATH	data/index.db	The vector index file
APP_EMBEDDING_BACKEND	fastembed	fastembed or sentence-transformers
APP_EMBEDDING_MODEL	BAAI/bge-small-en-v1.5	⚠️ Changing needs a reindex
APP_CHUNK_MAX_CHARS	1200	Largest chunk
APP_CHUNK_OVERLAP_CHARS	150	Overlap so facts spanning a break stay findable
APP_CRAWL_MAX_PAGES	500	Crawl cap
APP_CRAWL_TIMEOUT_S	20	Per-page fetch timeout

---

Running without a GPU

Yes, an ordinary server works — as long as you understand which half is slow.

	Plain CPU?	Runs how often
Ingest, chunk, embed, index, search	✅ Production-grade	Once per docs change
Writing the answer	⚠️ Depends on model size	Every question

Measured on an ordinary 8-core CPU, no GPU: ~38 chunks/sec indexing, 5.5 ms per query embedding. A 10,000-chunk docs set indexes in about four minutes. The retrieval half is simply not your problem.

Four ways to handle the answer-writing half, cheapest first:

1. Ship retrieval-only. Return the matching sections with links instead of a written paragraph. Instant, runs anywhere, and cannot hallucinate — the user reads your actual docs. docschatai search already does this. For many products this is genuinely the better feature.
2. Use 3B instead of 8B. ~10 seconds versus ~30, with little quality loss, because the model is reading rather than recalling.
3. Keep streaming on (the default). First words in about a second reads as fast; the same answer after 30 silent seconds reads as broken.
4. Enable the semantic cache. Docs questions repeat heavily; near-duplicates skip retrieval and generation entirely.

Deployment recipes

Your situation	Setup
Evaluating	Docker, CPU, retrieval-only. Prove your docs extract well before spending on hardware.
Internal staff docs bot	CPU server, llama3.2:3b, streaming on. No GPU cost.
Customer-facing, hundreds of installs	One 8–12 GB GPU, llama3.1:8b via Ollama, semantic cache on.
Customer-facing, thousands	24 GB+ GPU, still 8B, served by vLLM for request batching.
Air-gapped / regulated	Any of the above with weights baked into your deployment artifact. Nothing calls out once models are local.
Docs extract badly	Fix ingestion before buying hardware. A bigger GPU can't rescue bad retrieval.

---

Adding the widget to your product

You ship one file — widget/docschatai.js. No HTML, no CSS, no build step, no dependencies.

First run the server (docschatai serve, or the Docker path — it needs Redis), then drop the script tag into your product:

<script src="https://your-server.com/widget/docschatai.js"></script>
<script>
  DocsChatAI.init({
    serverUrl: "https://chat.yourproduct.com",   // or your own middleware
    title:     "Ask Acme Assistant",

    // Where the user is. Usually maps onto a docs page, so it biases
    // retrieval toward that area before falling back to a full search.
    appName:   "Admin Console",
    pageName:  () => currentScreenId,

    // Opaque. Forwarded verbatim for YOUR middleware to check — the server
    // never parses, logs or stores it.
    context:   () => ({ tenant: currentTenant, userToken: sessionToken() }),
  });
</script>

Two ways to deploy it

	Widget points at	Who authenticates	Use when
Direct	The server	Nobody, or a shared secret	The server is only reachable from inside your network
Proxied	Your middleware	You — using context	The endpoint is public

If your middleware does the authenticating, the server must not be publicly reachable, or the middleware is simply skipped. Bind it to a private interface or allowlist your middleware's address.

There is no identity database. We never learn who installed anything. You already know who your users are — context is where you plug that in.

It renders in a Shadow DOM, so your CSS can't break it and it can't break your product. That also means normal selectors won't style it — theming goes through CSS custom properties, which are the one thing that crosses the shadow boundary:

docs-chat-ai {
  --dca-primary: #7c3aed;
  --dca-radius: 4px;
  --dca-font: "Inter", system-ui, sans-serif;
}

Full guide — all 17 variables, React/Vue/Angular examples, and the server API contract: docs/WIDGET.md. Live theming playground you can open right now: widget/demo.html.

---

Architecture

INDEXING (when docs change)
  docs ──► Markdown ──► chunks ──► embeddings ──► index.db
 (URL or folder)      (by heading)   (local)

ANSWERING (per question)
  question ──► embedding ──► nearest chunks ──► LLM ──► answer + citations

One vendor runs one server. Every install of your product queries it. Ingestion sources have exactly one job — put Markdown in the corpus folder — so the indexer never knows whether content came from a crawl, a folder, or an upload.

Layer	Default	Alternative
LLM serving	Ollama	vLLM, TGI, llama.cpp
Model	Llama 3.1 8B	Qwen 2.5 7B, Mistral 7B, Llama 3.2 3B
Embeddings	bge-small-en-v1.5 (fastembed/ONNX)	bge-base, bge-large, nomic-embed-text
Vector store	sqlite-vec	Qdrant
Session memory	Redis	—

Every layer is a swappable provider behind one interface — including auth, which is what makes vendor-specific SSO or license-system integration possible without a rewrite.

More detail, written for people new to RAG: docs/HOW-IT-WORKS.md.

---

Project status

Working today:

• Ingestion — URL crawl (SSRF-guarded) and existing-folder sources

• Normalization — HTML/PDF/Markdown/text → Markdown, boilerplate stripped

• Heading-aware chunking with heading breadcrumbs

• Local embeddings (fastembed/ONNX), no external API

• sqlite-vec store with per-chunk change detection

• Corpus-aware typo correction and special-character handling

• LLM serving — Ollama + any OpenAI-compatible server

• Answer generation — numbered steps, citations, streaming, refuses on weak retrieval, never invents procedures

• CLI — quickstart, doctor, serve, ingest, index, search, ask, stats

• Embeddable themeable widget with opaque context pass-through

• Chat API — /api/chat, /api/health, CORS allowlist, optional HMAC

• Abuse protection — rate limits, daily quotas, GPU concurrency cap, load shedding. On by default, fails closed

• 24-hour conversation memory in Redis

• App/page retrieval hints

• Docker — CPU and GPU profiles

• Streaming over HTTP — SSE, with proxy-buffering headers and in-band error events

Not built yet:

• Semantic cache (skip retrieval + generation on near-duplicate questions)
• Extraction quality-gate CLI

---

Paid help

Everything above is free and meant to work out of the box, and for most documentation sites it does. The one thing that reliably needs a human is extraction.

The crawler pulls the main content out of a docs page and throws away navigation, sidebars and footers. That works on most sites. On some — unusual themes, JavaScript-rendered pages, heavily customised layouts — it pulls in menu clutter or misses the body entirely. If your docs don't extract cleanly, send us the link and we'll tune the extraction script for your site by hand. That's the service: a manual fix for your specific documentation, not a plugin or a platform.

How to tell whether you need it — run this and read the output:

docschatai search "a question you know the answer to"

If the right section comes back on top, you're fine and you don't need us. If you get navigation text, page furniture, or nothing sensible, that's an extraction problem and we can fix it.

📧 deepanshukapoor [at] live [dot] in — send the link to your documentation and what you're seeing. You'll get a straight answer on whether it needs work at all. Most sites don't.

---

Use cases

In-product documentation help (primary). A SaaS admin dashboard, developer platform, or enterprise application embeds the widget so users get answers without leaving the screen they're stuck on.

On-premise and air-gapped deployments. Regulated industries — healthcare, finance, defence, government — where sending user questions to a third-party AI API is prohibited outright.

Internal employee knowledge base. Point it at your wiki, runbooks, or handbook and give staff a private search assistant.

Developer documentation portal. Conversational search over your API docs, so developers can ask "how do I paginate this endpoint?" instead of grepping a sidebar.

Reducing support ticket volume. Deflect repetitive "how do I…" tickets already answered in the manual, with citations so users can verify.

---

How it compares

	DocsChatAI	Hosted AI docs SaaS	Generic website chatbot
Hosting	Self-hosted, your servers	Their cloud	Their cloud
AI cost	Free — local models	Per-message or per-seat	Per-message
Data leaves your network	Never	Yes	Yes
Branding	Entirely yours	Often "powered by"	Often "powered by"
Air-gap capable	Yes	No	No
Source available	Yes, Apache 2.0	No	No
Built for	In-product help	Docs sites	Sales and lead capture

Related open-source projects: DocsGPT, Danswer, Verba and PrivateGPT solve adjacent problems. DocsChatAI differs by targeting vendors embedding an assistant into a product they ship to customers, as well as to their own staff — hence the opaque context pass-through for your own middleware, per-key rate limiting, and a Shadow-DOM widget built to survive being dropped into someone else's codebase.

---

FAQ

Does this require an OpenAI or Anthropic API key?

No. It runs open-weight models locally through Ollama or vLLM. There are no API keys, no per-token costs, and no external AI service involved at any point.

What hardware do I need?

No GPU? Run llama3.2:3b — answers land in about ten seconds. Any GPU with 8 GB or more? Run llama3.1:8b and answers arrive in about five. That second setup is what most vendors should deploy.

Embeddings and search always run fine on plain CPU (5.5 ms per query), so never buy a GPU for those. Full table: System requirements.

How do I change which model writes the answers?

ollama pull llama3.2:3b, then set APP_LLM_MODEL=llama3.2:3b in .env. No reindex needed. See Choosing and changing the model.

Can it run fully offline / air-gapped?

Yes. Once the embedding model and LLM weights are downloaded, no component makes an outbound network call.

Why RAG instead of fine-tuning a model on my docs?

Fine-tuning teaches style well and facts poorly, must be redone on every docs change, and still hallucinates specifics. RAG looks up the relevant passage first and asks the model to answer from it — so docs updates take effect immediately and every answer can be cited.

What document formats can it ingest?

Markdown, HTML, PDF and plain text — from a crawled site, a folder you already maintain, or an upload. Everything normalizes to Markdown internally, and site navigation, sidebars and footers are stripped automatically.

What if users misspell things or paste junk?

Questions are Unicode-normalized and stripped of control characters, then — only if retrieval came back weak — retried with corpus-aware spelling correction, using your indexed documentation as the dictionary. So your product's own vocabulary counts as correctly spelled with no wordlist to maintain.

Measured, all three retrieving the same correct section:

Question	Score	What happened
how do I add a webhook?	0.819	Retrieved well — left completely untouched
how ot add a webhok ?@@	0.790	Embeddings absorbed the typos unaided
how do i creat an endpoit ?!<>	0.616 → 0.758	Too weak, so retried with corrections

Punctuation and markup are deliberately preserved, not stripped — someone asking why --dca-primary: red doesn't apply is asking a real question.

Can I restyle the widget to match my product?

Yes — 17 CSS custom properties covering colour, typography, radius, size and position, with automatic dark mode. See the Widget Guide.

How do I stop one customer from abusing the server?

Every product install gets its own key. Rate limiting is per key, so one noisy install exhausts only its own budget, and revoking a key has zero effect on any other install. Only a hash of each key is stored server-side.

Is it free for commercial use?

Yes. Apache 2.0, including an explicit patent grant. You can embed it in a commercial product and are not required to open-source your own code.

---

License

Apache 2.0 — includes an explicit patent grant.

Use it only for documentation you own or control. This is not a general-purpose scraper. Because deployments are self-hosted, GDPR/CCPA responsibility sits with the deploying vendor — see SECURITY.md.

Contributing: CONTRIBUTING.md covers local setup and the two extension points most changes touch — docs sources and auth providers. Changes: CHANGELOG.md.