chore: added bg task for chucking

[blackbox24]

Changelog

All notable changes to LexAI are documented here.

---

[Unreleased] — In Progress

Fixed

• Uncommented and restored ingest_pdf() call inside run_ingest_job which was accidentally left commented out, causing all PDF uploads to silently do nothing
• Restored DigitalOcean Spaces upload inside run_ingest_job as a non-fatal operation — Spaces failure now logs a warning and continues rather than blocking the ingestion pipeline

Changed

• Spaces upload and FAISS ingestion are now correctly separated into independent try/except blocks inside the background job — a Spaces outage can no longer prevent a document from being indexed

---

[1.1.0] — 2026-03-12

Fixed — 504 Gateway Timeout on /ingest

• Root cause: DigitalOcean App Platform enforces a 30-second load balancer timeout. Embedding 1600+ chunks from large PDFs took 60-90 seconds, causing every large document upload to fail in production while working fine locally
• Fix: /ingest now returns a job_id immediately (< 1 second) and processes the document in a FastAPI BackgroundTask. The frontend polls /ingest/status/{job_id} every 2 seconds until status is done or error
• Added /ingest/status/{job_id} endpoint returning job status, full result on completion, and error message on failure
• Added request_timeout=25 to ChatOpenAI in agent.py to prevent /chat from hitting the same 30-second wall on slow LLM responses

Changed

• uploadDocument() in client.js updated to two-phase flow: POST to /ingest for job submission, then poll /ingest/status/{job_id} until completion
• DocumentUpload.jsx now shows live elapsed time during processing (Analysing document... 12s) so users know the upload is progressing
• In-memory jobs dict added to main.py to track background job state per job_id

---

[1.0.1] — 2026-03-12

Fixed — fastembed model downloading at runtime (exit code 128 / OOM)

• Root cause: Multi-stage Dockerfile copied the fastembed model cache to /root/.cache but the app ran as myuser, whose cache is at /home/myuser/.cache. Cache miss on every startup triggered a 90MB HuggingFace download, exhausting the 512MB container RAM and crashing with exit code 128
• Fix: Switched to single-stage Dockerfile running as root with explicit FASTEMBED_CACHE_PATH=/app/.cache/fastembed and HF_HOME=/app/.cache/huggingface environment variables set before the model download step — build and runtime now use identical paths, guaranteed cache hit
• Model cache now bakes into the Docker image at build time (list(model.embed(['test'])) forces all 5 model files to download)
• Increased DO App Platform instance from 512MB to 1GB RAM to handle fastembed model + PDF parsing + FAISS operations concurrently

Fixed — CORS blocking frontend requests

• Added production frontend URL (https://lexai-frontend-43cn4.ondigitalocean.app) to allow_origins in main.py
• FRONTEND_URL now read from environment variable so CORS origins update without code changes

---

[1.0.0] — 2026-03-11

Added — Initial release

Backend

• POST /ingest — PDF upload pipeline: text extraction (pdfplumber + pytesseract OCR fallback), section-aware chunking, fastembed embeddings, FAISS indexing, and proactive risky clause detection
• POST /chat — RAG question answering scoped to uploaded document via LangGraph retrieve_node → generate_node pipeline
• POST /ticket — Lawyer review request with mock ticket ID (LEX-XXXXXX)
• GET /health — Health check endpoint for DO App Platform and CI/CD
• PII redaction (Ghana Card numbers, TINs, passports, emails, phone numbers) applied to all queries before model calls
• Legal disclaimer appended to every generated answer
• DigitalOcean Spaces integration for persistent raw PDF storage

AI Pipeline

• Generation via DigitalOcean Gradient AI serverless inference (llama3-8b-instruct) using OpenAI-compatible endpoint (https://inference.do-ai.run/v1)
• Local embeddings via fastembed (BAAI/bge-small-en-v1.5, 384-dim) — no torch, no CUDA, no external embedding API
• FAISS IndexFlatIP with L2 normalisation for cosine similarity search, persisted to disk
• LangGraph StateGraph wrapped with gradient-adk @entrypoint decorator for ADK compatibility and automatic trace capture

Frontend

• React 18 + Tailwind CSS chat interface
• Drag-and-drop PDF upload with instant risky clause summary
• Cited source cards showing retrieved clause text and relevance score
• Suggested questions bar
• Lawyer review request modal

Infrastructure

• Dockerfile (single-stage, Python 3.12-slim) with fastembed model baked in at build time
• docker-compose.yml for single-command local development
• DO App Platform deployment for both backend (Docker) and frontend (static site)
• GitHub Actions CI/CD: test → deploy-backend → deploy-frontend via doctl apps create-deployment
• DigitalOcean Spaces for raw contract storage