Real-time multi-agent claims processing with voice, vision, and fraud detection.
Built for the Gemini Live Agent Challenge hackathon.
ClaimSight transforms insurance claims from a weeks-long ordeal into a single live conversation. A policyholder opens the app, shows damage via their camera, and AI agents handle everything — from identity verification to damage assessment, cost estimation, fraud analysis, and FNOL submission.
- Triage Agent greets the caller, verifies their policy, and routes to the right specialist
- Specialist Agent (Maya for property, Alex for auto, Jordan for liability) conducts the damage assessment via live video
- Fraud Sentinel runs in parallel — analyzing narrative consistency, checking for AI-generated images, verifying content provenance (C2PA), and cross-referencing financial records via Plaid
- Weather Verifier confirms weather conditions at the claim location/date
- Everything documented, estimated, triaged, and submitted — in one session
- Multi-Agent Architecture — 5 specialized agents with 24 tools, orchestrated by Google ADK
- Live Video Damage Assessment — Camera feed analyzed in real-time for damage documentation
- AI Image Generation — Annotated damage photos, repair visualizations, cost infographics (Gemini image gen)
- Deepfake Detection — 3-layer approach: Gemini vision forensics + C2PA content credentials + linguistic voice analysis
- Financial Verification — Plaid API integration to verify purchase transactions and repair receipts
- Weather Correlation — Confirms weather events match claimed damage dates
- Sentiment Analysis — Adapts agent tone based on policyholder emotional state
- Instant Triage — Auto-approves simple claims, routes complex ones to human adjusters with AI-generated briefs
- Real-Time Agent Brain — Visual dashboard showing active agent, tool calls, and 9-step claim progress
┌─────────────────────────────────────────────────────────┐
│ React Frontend │
│ ┌──────────┐ ┌──────────┐ ┌───────────┐ ┌───────────┐ │
│ │ Video │ │Transcript│ │Agent Brain│ │ Gallery │ │
│ │ Stream │ │ Panel │ │ Dashboard │ │ Viewer │ │
│ └────┬─────┘ └────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
│ └─────────────┴─────────────┴─────────────┘ │
│ WebSocket │
└─────────────────────────┬───────────────────────────────┘
│
┌─────────────────────────┴───────────────────────────────┐
│ FastAPI Backend │
│ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Google ADK Runner │ │
│ │ │ │
│ │ ┌───────────┐ │ │
│ │ │ Triage │──┬── Maya (Property, 18 tools) │ │
│ │ │ Agent │ ├── Alex (Auto, 6 tools) │ │
│ │ │ │ ├── Jordan (Liability, 5 tools) │ │
│ │ │ │ ├── Fraud Sentinel (11 tools) │ │
│ │ │ │ └── Weather Verifier (1 tool) │ │
│ │ └───────────┘ │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌────────┐ ┌──────────────┐ │
│ │ Gemini │ │ C2PA │ │ Plaid │ │ Weather │ │
│ │ Vision │ │Provenance│ │Finance │ │ Verify │ │
│ └──────────┘ └──────────┘ └────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────┘
| Layer | Technology |
|---|---|
| AI Framework | Google ADK 1.26+ (Agent Development Kit) |
| LLM | Gemini 2.5 Flash |
| Image Gen | Gemini 3.1 Flash Image Preview |
| Backend | FastAPI + WebSocket (bidirectional) |
| Frontend | React 19 + Vite + TailwindCSS v4 |
| Fraud Detection | Gemini Vision + C2PA (c2pa-python) |
| Financial | Plaid API (Sandbox) |
| Deployment | Google Cloud Run + Firestore + Cloud Storage |
- Python 3.11+
- Node.js 18+
- Google AI API key (Get one)
cd backend
pip install -r requirements.txt
# Create .env with your GOOGLE_API_KEY
python -m uvicorn main:app --host 0.0.0.0 --port 8003cd frontend
npm install
npm run dev # Runs on http://localhost:3002GOOGLE_API_KEY=your_gemini_api_key
PLAID_CLIENT_ID=your_plaid_client_id # Optional
PLAID_SECRET=your_plaid_sandbox_secret # Optional
| Name | Policy | Type |
|---|---|---|
| John Smith | HO-2024-78901 | Homeowners ($350K) |
| Sarah Johnson | AU-2024-45678 | Auto ($100K) |
| Mike Davis | GL-2024-11223 | Liability ($1M) |
Core Claims: policy_lookup · damage_assessment · photo_capture · estimate_repair · submit_fnol · send_summary
Image Gen: annotate_damage · generate_repair_viz · generate_cost_infographic
Fraud & Deepfake: analyze_narrative_consistency · check_photo_metadata · detect_claim_patterns · calculate_fraud_risk_score · detect_ai_generated_image · verify_content_provenance · detect_synthetic_voice · calculate_media_authenticity_score
Financial (Plaid): verify_purchase_transaction · validate_repair_receipt · get_financial_verification_summary
Assessment: verify_weather_conditions · analyze_sentiment · triage_claim_decision · generate_adjuster_brief
Insurance claims cost carriers $40+ per claim in processing and take days to weeks. Fraud costs the industry $80B+/year. ClaimSight handles the entire FNOL in a single live session while running multi-layered fraud detection — reducing cost, time, and fraud simultaneously.
Live Agents — Real-time bidirectional voice + video agent with multi-agent handoff
Created for the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge
MIT