Legal RAG System - Document Q&A Platform

A comprehensive Retrieval-Augmented Generation (RAG) system for legal document question answering, built with FastAPI, Pinecone vector database, and Voyage AI embeddings.

📋 Table of Contents

• Overview
• Features
• Architecture
• Quick Start
• Installation
• Configuration
• Usage
• API Reference
  • Authentication
  • HackRx Run API
• Testing
• Troubleshooting
• Project Structure
• Contributing
• License

🎯 Overview

The Legal RAG System is a sophisticated document question-answering platform designed specifically for legal documents. It combines advanced text processing, semantic search, and AI-powered responses to provide accurate answers to legal queries.

Key Capabilities

• Multi-format Document Support: PDF, DOCX, TXT, Images (with OCR)
• Legal-aware Chunking: Intelligent text segmentation for legal documents
• Hybrid Search: Dense embeddings + sparse retrieval for better results
• Context-aware Responses: Legal document context preservation
• Admin Interface: Document management and system monitoring
• Sample Dataset: Pre-loaded legal documents for testing

🚀 Features

Document Processing

• OCR Support: Extract text from images and scanned documents
• PDF Processing: Advanced PDF text extraction with layout preservation
• Legal Chunking: Intelligent segmentation based on legal document structure
• Metadata Extraction: Automatic extraction of document metadata

Search & Retrieval

• Semantic Search: Vector-based similarity search using Voyage AI embeddings
• Hybrid Retrieval: Combines dense and sparse retrieval methods
• Context Preservation: Maintains document context across chunks
• Relevance Scoring: Advanced similarity scoring and ranking
• Spell Correction: Automatic correction of misspelled legal and insurance terms

AI Integration

• Voyage AI Embeddings: High-quality vector embeddings
• Response Generation: AI-powered answer generation
• Context Awareness: Legal document-specific response formatting
• Confidence Scoring: Response confidence indicators

System Management

• Admin Dashboard: System monitoring and statistics
• Document Management: Upload, delete, and manage documents
• Health Monitoring: System health checks and status
• Performance Metrics: Processing time and accuracy tracking

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Document      │    │   Processing    │    │   Vector        │
│   Ingestion     │───▶│   & Chunking    │───▶│   Storage       │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   OCR & Text    │    │   Embedding     │    │   Pinecone      │
│   Extraction    │    │   Generation    │    │   Database      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                       │
                                                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Query         │◀───│   Retrieval     │◀───│   Semantic      │
│   Interface     │    │   & Reranking   │    │   Search        │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │
         ▼                       ▼
┌─────────────────┐    ┌─────────────────┐
│   LLM           │    │   Response      │
│   Generation    │───▶│   Formatting    │
└─────────────────┘    └─────────────────┘

⚡ Quick Start

Prerequisites

• Python 3.8 or higher
• Voyage AI API key
• Pinecone API key

1. Clone and Setup

# Clone the repository
git clone <repository-url>
cd legal-rag-system

# Install dependencies
pip install -r requirements.txt

2. Configure Environment

# Copy environment template
cp env_template.txt .env

# Edit .env with your API keys
VOYAGE_API_KEY=your_voyage_api_key_here
PINECONE_API_KEY=your_pinecone_api_key_here
PINECONE_ENVIRONMENT=us-east-1

3. Initialize Vector Database

# Create Pinecone index
python create_pinecone_index.py

4. Start the Application

# Start the server
python -m uvicorn api.main:app --reload --host 127.0.0.1 --port 8000

5. Access the Application

• API Documentation: http://localhost:8000/docs
• Health Check: http://localhost:8000/health

📦 Installation

System Requirements

• Python: 3.8 or higher
• Memory: Minimum 4GB RAM
• Storage: 1GB free space
• Network: Internet connection for API calls

Quick Setup

Option 1: Automated Setup (Recommended)

# Run the setup script
python setup.py

# Or use make
make setup

Option 2: Manual Setup

# Install dependencies
pip install -r requirements.txt

# Create environment file
cp env_template.txt .env

# Initialize database
python create_pinecone_index.py

Option 3: Docker Setup

# Build and run with Docker
docker-compose up --build

# Or build manually
docker build -t legal-rag-system .
docker run -p 8000:8000 legal-rag-system

Environment Setup

1. Create Environment File

   cp env_template.txt .env

2. Configure API Keys

   # Voyage AI Configuration
   VOYAGE_API_KEY=your_voyage_api_key_here
   
   # Pinecone Configuration
   PINECONE_API_KEY=your_pinecone_api_key_here
   PINECONE_ENVIRONMENT=us-east-1
   PINECONE_INDEX_NAME=legal-rag-index
   PINECONE_DIMENSION=1024
   
   # System Configuration
   CHUNK_SIZE=1000
   CHUNK_OVERLAP=200
   TOP_K_RESULTS=5
   SEARCH_SIMILARITY_THRESHOLD=0.7

⚙️ Configuration

Pinecone Setup

1. Get Pinecone API Key

   • Sign up at https://app.pinecone.io/
   • Navigate to API Keys section
   • Copy your API key

2. Create Index

   python create_pinecone_index.py

3. Verify Setup

   python -c "from vectordb.pinecone_client import get_index_stats; print(get_index_stats())"

Voyage AI Setup

1. Get Voyage AI API Key

   • Sign up at https://platform.voyageai.com/
   • Generate API key from dashboard
   • Add to your .env file

2. Test Integration

   python test_voyage_integration.py

System Configuration

Key settings in config/settings.py:

# Document Processing
CHUNK_SIZE = 1000          # Words per chunk
CHUNK_OVERLAP = 200        # Overlap between chunks

# Search Configuration
TOP_K_RESULTS = 5          # Number of search results
SEARCH_SIMILARITY_THRESHOLD = 0.8  # Similarity threshold
ENABLE_SPELL_CORRECTION = True    # Enable automatic spell correction

# API Configuration
VOYAGE_MODEL = "voyage-3-large"    # Embedding model
PINECONE_DIMENSION = 1024          # Vector dimension

📚 Usage

Document Upload

Using API

# Single document upload
curl -X POST "http://localhost:8000/ingest/upload" \
  -H "accept: application/json" \
  -H "Content-Type: multipart/form-data" \
  -F "file=@path/to/your/document.pdf" \
  -F "doc_type=employment_agreement" \
  -F "doc_title=My Employment Contract"

# Multiple documents upload
curl -X POST "http://localhost:8000/ingest/upload-multiple" \
  -H "accept: application/json" \
  -H "Content-Type: multipart/form-data" \
  -F "files=@document1.pdf" \
  -F "files=@document2.pdf" \
  -F "doc_types=employment_agreement,nda" \
  -F "doc_titles=Employment Contract,Non-Disclosure Agreement"

Using Python

import requests

# Upload a document
url = "http://localhost:8000/ingest/upload"
files = {'file': open('path/to/document.pdf', 'rb')}
data = {
    'doc_type': 'employment_agreement',
    'doc_title': 'My Employment Contract'
}

response = requests.post(url, files=files, data=data)
print(response.json())

Query Interface

Using API

# Ask a question
curl -X POST "http://localhost:8000/query/ask?question=What%20is%20the%20employee%27s%20base%20salary?"

# Search documents
curl -X GET "http://localhost:8000/query/search?query=termination%20provisions"

# The system automatically corrects misspelled terms
curl -X GET "http://localhost:8000/query/search?query=what%20is%20my%20deductable%20amount"

Using Python

import requests

# Ask a question
response = requests.post(
    'http://localhost:8000/query/ask',
    params={'question': 'What is the employee\'s base salary?'}
)
print(response.json())

Web Interface

1. Open http://localhost:8000/docs
2. Use the interactive API documentation
3. Test endpoints directly from the browser

🔌 API Reference

Authentication

POST /auth/register

Register a new user.

Parameters:

• username (required): User's username
• password (required): User's password
• email (required): User's email address

Response:

{
  "username": "user123",
  "email": "user@example.com",
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "token_type": "bearer"
}

POST /auth/token

Get authentication token.

Parameters:

• username (required): User's username
• password (required): User's password

Response:

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "token_type": "bearer"
}

HackRx Run API

POST /hackrx/run

Process a document URL and answer questions about it.

Parameters: This endpoint accepts both JSON and form data inputs:

JSON Input:

• url (required): URL of the document to process
• question (required): Question to ask about the document

Form Data Input:

• url (required): URL of the document to process
• question (required): Question to ask about the document

Example Usage:

Using JSON:

curl -X POST "http://localhost:8000/hackrx/run" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/document.pdf","question":"What are the key points?"}'

Using Form Data:

curl -X POST "http://localhost:8000/hackrx/run" \
  -H "Content-Type: multipart/form-data" \
  -F "url=https://example.com/document.pdf" \
  -F "question=What are the key points?"

Response:

{
  "answer": "Based on the document, the key points are...",
  "sources": [
    {
      "doc_id": "document_001",
      "chunk_id": "section_1",
      "text": "The document states...",
      "similarity": 0.92
    }
  ],
  "processing_time": 2.5
}

Document Ingestion Endpoints

POST /ingest/upload

Upload a single document.

Parameters:

• file (required): Document file (PDF, DOCX, TXT, images)
• doc_type (required): Document type (e.g., employment_agreement, nda)
• doc_title (required): Human-readable document title
• author (optional): Document author
• date (optional): Document date
• description (optional): Document description
• tags (optional): Comma-separated tags

Response:

{
  "filename": "document_20250801_123456_abc123.pdf",
  "status": "accepted",
  "warnings": []
}

POST /ingest/upload-multiple

Upload multiple documents.

Parameters:

• files (required): Array of document files
• doc_types (required): Comma-separated document types
• doc_titles (required): Comma-separated document titles

GET /ingest/status/{doc_id}

Check document processing status.

Query Endpoints

POST /query/ask

Ask a question about uploaded documents.

Parameters:

• question (required): The question to ask

Response:

{
  "answer": "Based on the employment agreement...",
  "sources": [
    {
      "doc_id": "contract_001",
      "chunk_id": "section_2_1",
      "text": "The Company shall pay...",
      "similarity": 0.95
    }
  ],
  "confidence": 0.95,
  "warnings": [],
  "query_processing": {
    "original_query": "What is the employee's base salery?",
    "processed_query": "what is the employee's base salary",
    "spell_corrections": [
      {
        "original": "salery",
        "corrected": "salary",
        "method": "fuzzy_match"
      }
    ],
    "corrections_applied": true
  }
}

GET /query/search

Search documents for specific terms.

GET /query/suggest

Get suggested questions.

Admin Endpoints

GET /admin/health

System health check.

GET /admin/stats

System statistics and metrics.

DELETE /admin/documents/{doc_id}

Delete a document.

POST /admin/cleanup

System cleanup operations.

🧪 Testing

Comprehensive Test Suite

The project includes a unified test suite that covers all aspects of the system:

# Run the complete test suite
python test_comprehensive.py

This single test file covers:

• Environment & Configuration: API keys, dependencies, configuration files
• Core Components: Chunking, metadata building, validation
• Document Processing: Text extraction, file utilities
• API Integration: Voyage AI, Pinecone connectivity
• Server Functionality: Health checks, API documentation
• Upload & Query: Document upload and query functionality
• Advanced Features: Query enhancement, response formatting
• Error Handling: Invalid inputs, edge cases

Individual Component Testing

# Test server health
curl http://localhost:8000/health

# Test document upload
curl -X POST "http://localhost:8000/ingest/upload" \
  -H "accept: application/json" \
  -H "Content-Type: multipart/form-data" \
  -F "file=@data/legal_docs/sample_contract.txt" \
  -F "doc_type=employment_agreement" \
  -F "doc_title=Sample Employment Contract"

# Test query interface
curl -X POST "http://localhost:8000/query/ask?question=What%20is%20the%20employee%27s%20base%20salary?"

Test Results

The comprehensive test suite provides detailed results for each category:

LEGAL RAG SYSTEM - COMPREHENSIVE TEST SUITE
================================================================================

📋 Environment & Configuration
--------------------------------------------------
✅ Environment & Configuration: 3/3 tests passed

📋 Core Components
--------------------------------------------------
✅ Core Components: 3/3 tests passed

📋 Document Processing
--------------------------------------------------
✅ Document Processing: 2/2 tests passed

📋 API Integration
--------------------------------------------------
✅ API Integration: 3/3 tests passed

📋 Server Functionality
--------------------------------------------------
✅ Server Functionality: 3/3 tests passed

📋 Upload & Query
--------------------------------------------------
✅ Upload & Query: 2/2 tests passed

📋 Advanced Features
--------------------------------------------------
✅ Advanced Features: 2/2 tests passed

📋 Error Handling
--------------------------------------------------
✅ Error Handling: 2/2 tests passed

================================================================================
TEST SUMMARY
================================================================================
Total Tests: 20
Passed: 20
Failed: 0
Success Rate: 100.0%

🎉 ALL TESTS PASSED! The system is working correctly.

Sample Test Data

The system includes sample legal documents:

• data/legal_docs/sample_contract.txt - Employment agreement
• data/legal_docs/sample_nda.txt - Non-disclosure agreement

Sample Questions to Test

1. Salary Questions:

   • "What is the employee's base salary?"
   • "What is the annual salary?"

2. Termination Questions:

   • "What are the termination provisions?"
   • "How much notice is required for termination?"

3. Benefits Questions:

   • "What benefits is the employee eligible for?"
   • "What insurance coverage is provided?"

4. Non-Compete Questions:

   • "What is the non-competition period?"
   • "What are the restrictions after termination?"

5. Confidentiality Questions:

   • "What are the confidentiality obligations?"
   • "What is considered confidential information?"

🐛 Troubleshooting

Common Issues

1. Missing API Keys

Symptoms: API key errors in logs Solution:

# Check .env file exists
ls -la .env

# Verify API keys are set
cat .env | grep API_KEY

2. Pinecone Connection Issues

Symptoms: "Invalid API Key" or "Environment Not Found" Solution:

# Test Pinecone connection
python -c "
from pinecone import Pinecone
pc = Pinecone(api_key='your_key')
print('Available indexes:', pc.list_indexes().names())
"

3. Voyage AI API Errors

Symptoms: 429 errors or quota exceeded Solution:

• Check Voyage AI API key and billing
• Verify API usage limits
• Add credits to your account

4. Server Not Starting

Symptoms: "uvicorn: command not found" Solution:

# Install uvicorn
pip install uvicorn

# Use Python module
python -m uvicorn api.main:app --reload

5. Import Errors

Symptoms: ModuleNotFoundError Solution:

# Install dependencies
pip install -r requirements.txt

# Check Python version
python --version

Debug Steps

1. Check Server Status:

   curl http://localhost:8000/health

2. View Application Logs:

   tail -f legal_rag.log

3. Test Individual Components:

   # Test Voyage AI
   python test_voyage_integration.py
   
   # Test Pinecone
   python test_pinecone_setup.py
   
   # Test document processing
   python test_document_processing.py

4. Verify Environment Variables:

   from config.settings import settings
   print("Voyage API Key:", settings.VOYAGE_API_KEY[:10] + "...")
   print("Pinecone API Key:", settings.PINECONE_API_KEY[:10] + "...")

📁 Project Structure

legal-rag-system/
├── api/                    # FastAPI application
│   ├── main.py            # Main application entry
│   ├── auth.py            # Authentication utilities
│   └── routes/            # API endpoints
│       ├── ingest.py      # Document ingestion
│       ├── query.py       # Q&A interface
│       └── admin.py       # Admin operations
├── ingestion/             # Document processing
│   ├── pdf_extractor.py   # PDF text extraction
│   ├── ocrProcessor.py    # OCR for images
│   └── textCleaner.py     # Text cleaning utilities
├── chunking/              # Text segmentation
│   ├── chunker.py         # Legal-aware chunking
│   └── metadata_builder.py # Chunk metadata
├── embeddings/            # Vector generation
│   └── embed_client.py    # Embedding service
├── vectordb/              # Vector database
│   ├── pinecone_client.py # Pinecone integration
│   └── schema.sql         # Database schema
├── llm_service/           # LLM integration
│   ├── llm_client.py      # Voyage AI client
│   ├── prompt_template.j2 # Prompt templates
│   └── response_formatter.py # Response formatting
├── data/                  # Sample datasets
│   ├── legal_docs/        # Sample legal documents
│   └── sample_queries.txt # Test queries
├── config/                # Configuration
│   └── settings.py        # System settings
├── utils/                 # Utilities
│   ├── file_utils.py      # File operations
│   └── validation.py      # Input validation
├── tests/                 # Test suite
├── requirements.txt       # Dependencies
├── .env.example          # Environment template
└── README.md             # This file

📊 Performance

System Metrics

• Document Processing: ~100 pages/minute
• Query Response Time: <2 seconds
• Search Accuracy: >85% on legal queries
• Vector Storage: Pinecone serverless index

Optimization Tips

1. Chunk Size: Adjust CHUNK_SIZE based on your documents
2. Batch Processing: Use /ingest/upload-multiple for multiple files
3. Caching: Consider adding Redis for response caching
4. Indexing: Monitor Pinecone index performance

🔒 Security

Security Features

• API Key Management: Environment variable-based configuration
• Input Validation: Comprehensive file and query validation
• Rate Limiting: API endpoint rate limiting
• Secure File Upload: File type and size validation

Best Practices

1. API Keys: Never commit API keys to version control
2. File Validation: All uploaded files are validated
3. Rate Limiting: Implement rate limiting for production
4. CORS: Configure CORS appropriately for your domain

🛠️ Development Tools

Available Commands

# Setup and installation
make setup          # Complete project setup
make install        # Install dependencies
make init-db        # Initialize database

# Development
make run            # Start development server
make test           # Run comprehensive tests
make lint           # Run code linting
make format         # Format code with black

# Maintenance
make clean          # Clean temporary files
make docs           # View documentation
make upload         # Upload sample documents

# Docker
make docker-build   # Build Docker image
make docker-run     # Run Docker container

# Utilities
make check-env      # Check environment setup
make check-deps     # Check dependencies
make status         # System status

Code Quality

• Linting: flake8 for code style checking
• Formatting: black for consistent code formatting
• Type Checking: mypy for static type analysis
• Testing: pytest for comprehensive testing

Professional Standards

• Type Hints: All functions include proper type annotations
• Documentation: Comprehensive docstrings and comments
• Error Handling: Robust exception handling throughout
• Logging: Structured logging with appropriate levels
• Security: Non-root Docker containers, secure defaults

🤝 Contributing

Development Setup

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

Testing Guidelines

• Run all tests before submitting: make test
• Add tests for new features
• Ensure code coverage is maintained
• Follow the existing test patterns

Code Style

• Follow PEP 8 guidelines
• Use type hints where appropriate
• Add docstrings to functions and classes
• Run make format before committing
• Run make lint to check code quality

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Getting Help

1. Check the troubleshooting section above
2. Review the API documentation at /docs
3. Check the logs in legal_rag.log
4. Run the test scripts to verify functionality

Reporting Issues

• Use GitHub Issues for bug reports
• Include logs and error messages
• Provide steps to reproduce the issue

Feature Requests

• Submit feature requests via GitHub Issues
• Include use cases and requirements
• Consider contributing the feature yourself

---

Version: 1.0.0