Local RAG based Document Q&A System

A local Retrieval-Augmented Generation (RAG) application that allows users to ask questions about a collection of documents. The application uses Ollama for local AI models, ChromaDB for vector storage, and LangChain for building the RAG pipeline.

The project supports both Markdown (.md) and PDF (.pdf) documents.

---

Features

• Load multiple Markdown and PDF documents
• Split documents into smaller chunks
• Generate embeddings locally using Ollama
• Perform semantic similarity search using ChromaDB
• Generate answers using a locally running LLM
• No paid API keys required
• Runs completely locally using Ollama

---

Project Architecture

Documents (.md / .pdf)
          ↓
   Document Loaders
          ↓
   Text Chunking
          ↓
   Ollama Embeddings
   (nomic-embed-text)
          ↓
      ChromaDB
   Vector Database
          ↓
    User Question
          ↓
   Question Embedding
          ↓
  Similarity Search
          ↓
 Relevant Document Chunks
          ↓
      Ollama LLM
    (llama3.2:3b)
          ↓
      Final Answer

---

Project Structure

rag-application/
│
├── data/
│   └── documents/
│       ├── 01_document_ingestion.md
│       ├── 02_text_chunking.md
│       ├── 03_embeddings_and_vector_storage.md
│       └── 04_retrieval_and_generation.md
│
├── create_database.py
├── query_data.py
├── requirements.txt
├── .gitignore
└── chroma/

The chroma/ directory is generated automatically and is excluded from version control using .gitignore.

---

Technologies Used

• Python
• LangChain
• Ollama
• nomic-embed-text — Embedding model
• llama3.2:3b — Local Large Language Model
• ChromaDB — Vector database
• PyPDF — PDF document processing

---

Setup

1. Clone the Repository

git clone <your-repository-url>
cd rag-application

2. Create and Activate a Python Environment

For example:

python -m venv venv

On Windows:

venv\Scripts\activate

Alternatively, a Conda environment can also be used.

3. Install Dependencies

pip install -r requirements.txt

---

Install Ollama Models

Install Ollama and make sure it is running.

Download the embedding model:

ollama pull nomic-embed-text

Download the language model:

ollama pull llama3.2:3b

Check the installed models:

ollama list

---

Add Documents

Place your documents inside:

data/books/

Supported formats:

.md
.pdf

---

Create the Vector Database

Run:

python create_database.py

This will:

1. Load the documents
2. Split them into smaller chunks
3. Generate embeddings using nomic-embed-text
4. Store the chunks and embeddings in ChromaDB

The vector database will be created in:

chroma/

---

Ask Questions

Run:

python query_data.py

The application will ask:

Ask a question about your documents:

For example:

What is Retrieval-Augmented Generation and how does this application work?

The application will:

1. Convert the question into an embedding
2. Search ChromaDB for relevant document chunks
3. Retrieve the most relevant context
4. Send the context and question to the local LLM
5. Generate an answer

---

Updating the Knowledge Base

If you add or modify documents:

1. Delete the Existing ChromaDB Folder

On Windows PowerShell:

Remove-Item -Recurse -Force .\chroma

2. Rebuild the Database

python create_database.py

This creates a fresh vector database using the updated documents.

---

RAG Workflow

Indexing

Documents
    ↓
Document Loading
    ↓
Text Splitting
    ↓
Embedding Generation
    ↓
Vector Storage

Querying

User Question
    ↓
Question Embedding
    ↓
Similarity Search
    ↓
Relevant Context
    ↓
LLM
    ↓
Generated Answer

---

Why Ollama?

This project uses Ollama to run AI models locally.

Advantages:

• No OpenAI API key required
• No API usage charges
• Documents remain on the local machine
• Works offline after the models are downloaded
• Easy to experiment with different open-source models

---

Example Questions

The application can answer questions based on the uploaded documents, such as:

What is the purpose of a document loader?

Why is text chunking necessary in a RAG application?

What chunk size and chunk overlap are used in this project?

Which embedding model is used to generate document embeddings?

What is the role of ChromaDB in this project?

How is the final answer generated using the retrieved context?

---

Important Notes

• Ollama must be installed and running before querying the application.
• The embedding model used during indexing and querying should be the same.
• If documents are changed, the ChromaDB database should be rebuilt.
• The chroma/ directory contains generated vector database files and is excluded from Git.
• The quality of answers depends on the content and quality of the documents provided to the system.

---

Future Improvements

Possible future improvements include:

• Streamlit web interface
• Upload documents directly through the UI
• Chat history and conversational memory
• Support for DOCX, TXT, and other file formats
• Improved retrieval and filtering
• Source citations in generated answers
• Docker containerization
• Cloud deployment

---

Author

Shubhi Joshi

Built as a hands-on project to understand:

• Generative AI
• Retrieval-Augmented Generation
• Vector databases
• Embeddings
• Local LLMs
• LangChain
• Ollama