RFPilot

An agentic RFP evaluation system. Upload an RFP and a set of vendor proposals; a LangGraph-orchestrated pipeline extracts structured requirements, scores every proposal against every requirement using retrieval-augmented matching, flags low-confidence results for human review, ranks proposals head-to-head, and generates a downloadable report — with the pipeline's execution surfaced live in the UI as it runs.

Live demo: https://rfpilot-1072960699477.asia-south1.run.app

---

Why this exists

Most "LLM wrapper" demos are a single prompt call with a chat box around it. RFPilot is built to demonstrate the opposite: a real orchestrated, stateful, multi-step agentic pipeline — with conditional branching, concurrent execution, retrieval, structured output validation, and a human-in-the-loop escape hatch for when the model isn't confident. The product surfaces that structure directly, instead of hiding it behind a spinner.

The pipeline

flowchart TD
    A["extract_requirements\n(structured extraction, Pydantic-validated)"] -->|ok| B["extract_proposal_content\n(chunk + embed, parallel per proposal)"]
    A -->|failed twice| X(["END — job failed"])
    B --> C["match_and_score\n(RAG similarity search + scoring,\nparallel per proposal×requirement)"]
    C -->|any low-confidence| D[mark_needs_review]
    C -->|all confident| E[mark_done]
    D --> F["rank_and_compare\n(cross-proposal reasoning)"]
    E --> F
    F --> G["report_generator\n(JSON + rendered PDF)"]
    G --> Y(["END"])

Loading

Every node emits progress events consumed live by the frontend — while a job is running, you see "Scoring Proposal 2 against 6 requirements...", not a static loading state.

Engineering highlights

A few things in here that go beyond "call the API and hope":

• Retry-then-flag, not trust-then-crash. Every structured LLM call is Pydantic-validated; a schema failure gets one corrective retry with the error fed back into the prompt, and a second failure routes that specific item to a human-review queue instead of failing the whole job. One bad response never takes down an evaluation.
• Parallelized the actual bottleneck, safely. The scoring stage was a sequential loop over every (proposal × requirement) pair — the dominant cost of a job. Rewrote it around a bounded ThreadPoolExecutor, with all shared-state aggregation kept on the main thread (workers return immutable results, never touch shared dicts) — no locks, no races. Real measured result: ~45s → ~13s for a 3-proposal evaluation.
• Live execution view backed by a real ordering guarantee. Progress events are polled from an append-only log. Initial design ordered by timestamp — a test caught a real reordering bug where rapid-fire events emitted with no work between them landed on the same wall-clock tick. Fixed with a Postgres IDENTITY sequence as the actual ordering key, not a band-aid retry.
• Found and fixed a systemic auth bug via testing, not luck. JWT verification assumed a shared HMAC secret; the identity provider actually signed asymmetrically (JWKS/ES256). Every real user token had been silently rejected — invisible for days because the test suite self-signed tokens under the same wrong assumption. Root-caused by decoding a real token's header offline, fixed with cached JWKS key verification, and the test suite was rebuilt to sign against a throwaway keypair through the real verification path instead of a shortcut.
• Cost governance as a first-class feature, not an afterthought. Every evaluation makes real, billed LLM calls, and this is a publicly-reachable endpoint — so there's a global rate limit (not per-account, since accounts are free to create) enforced before any model call is made, backed by nothing more than a COUNT(*) against existing data — no new infrastructure required.

Architecture

flowchart LR
    U[Browser] <-->|upload / evaluate / poll| API[FastAPI]
    API -->|BackgroundTasks| Graph[LangGraph pipeline]
    Graph -->|structured generation + embeddings| Gemini[Google Gemini]
    Graph <-->|relational + vector + blob storage| PG[(Postgres + pgvector)]
    Graph -->|trace every node/LLM call| LS[LangSmith]
    API -.->|Docker image| CR[Cloud Run]

Loading

One Postgres database does relational data, vector search (pgvector), and file storage (PDFs as bytea) — deliberately no separate object store or vector DB, to keep the moving parts down for a project at this scale.

Tech stack

Layer	Choice
API	FastAPI + Uvicorn
Orchestration	LangGraph + LangChain (explicit state graph, conditional routing — not a linear chain)
LLM	Google Gemini (gemini-3.5-flash-lite for generation, gemini-embedding-001 for embeddings)
Database	PostgreSQL + pgvector — relational data, vector search, and file storage in one place
Background jobs	FastAPI BackgroundTasks (deliberately not Celery/Redis at this scale)
Tracing	LangSmith — every node and LLM call traced automatically
Reporting	WeasyPrint (structured JSON + rendered PDF)
Frontend	Server-rendered Jinja2 + vanilla JS — no build step, no framework
Deployment	Docker on Google Cloud Run, secrets via Secret Manager, migrations via Alembic
Testing	pytest — mocked-LLM unit/integration tests plus real end-to-end verification (Docker + browser automation) before every deploy

Running it locally

git clone https://github.com/apoorv890/RFPilot.git
cd RFPilot
cp .env.example .env   # fill in a real GEMINI_API_KEY at minimum

docker compose up -d          # Postgres + pgvector
alembic upgrade head          # apply migrations
pip install -r requirements.txt
uvicorn app.main:app --reload

Then open http://localhost:8000. No login required — upload an RFP, upload one or more proposals, click Evaluate.

Tests

pytest

46 tests covering structured-output retry/validation logic, the parallelized scoring aggregation (including its event-emission and edge cases), API endpoints, and the monthly cost cap — all against mocked LLM calls and a real local Postgres instance, no network calls to Gemini in the automated suite.

Project structure

app/
  api/routes.py          # REST endpoints
  graph/                 # LangGraph nodes, graph assembly, live-event emission
  services/               # LLM client, PDF parsing/rendering, vector store, file storage
  models/                  # SQLModel schemas
  web/                       # Jinja2 templates + static JS/CSS
alembic/                      # database migrations
tests/

Deployment

Single-stage Docker image, deployed to Cloud Run (asia-south1) with --no-cpu-throttling (required so background pipeline execution keeps running after the HTTP response is sent) and --min-instances=1 to avoid cold starts. Secrets are injected via Secret Manager — never baked into the image or committed to the repo.