🧠 OpsPilot

Autonomous multi-agent AI system for Revenue Operations triage

Listens to Slack · Triages with Gemini · Enriches from Notion & Salesforce · Drafts replies automatically

---

Overview

OpsPilot is an event-driven, multi-agent AI system built for B2B SaaS Revenue Operations teams. It connects to a live Slack workspace via Socket Mode, autonomously classifies incoming messages using Google Gemini, enriches context through RAG (Notion playbooks → ChromaDB) and CRM data (Salesforce), and drafts professional replies — all in real time.

Every agent execution is logged with full observability (tokens used, latency, confidence scores, time-saved estimates) to a SQLite database and visualized through a real-time React dashboard.

Key Highlights

• 🤖 ReAct Agentic Workflow — 2-Node LangGraph (Triage + ReAct Drafting Agent) with tool execution
• 📊 Real-time Analytics Dashboard — React + Recharts with live-polling metrics and run trace drill-downs
• 🧪 High Regression Accuracy — 94% triage accuracy on 100-case regression suite
• ⚡ Event-Driven Architecture — Slack Socket Mode with async FastAPI backend

---

Architecture

graph TD
    classDef external fill:#f9f9f9,stroke:#333,stroke-width:2px,color:#333
    classDef agent fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000
    classDef database fill:#fff3e0,stroke:#ff9800,stroke-width:2px,color:#000
    classDef frontend fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px,color:#000

    Slack[("Slack Workspace<br/>(Socket Mode)")]:::external
    Notion[("Notion<br/>(Playbooks & FAQs)")]:::external
    Salesforce[("Salesforce<br/>(CRM Data)")]:::external

    subgraph Backend ["FastAPI Backend"]
        API[("REST API")]
        SlackHandler["Slack Handler"]

        subgraph LangGraph ["LangGraph 2-Node ReAct Workflow"]
            TriageAgent(("Triage Node<br/>(Gemini 1.5 Flash)")):::agent
            DraftAgent(("Drafting Agent<br/>(ReAct Loop)")):::agent
            
            subgraph Tools ["Agent Tools"]
                NotionTool["search_notion_docs"]:::agent
                SFLookup["lookup_salesforce_account"]:::agent
                SFTask["create_salesforce_task"]:::agent
                EntityExt["extract_entities"]:::agent
            end
        end
    end

    ChromaDB[(ChromaDB<br/>Vector Store)]:::database
    SQLite[(SQLite<br/>Signal Tracker)]:::database
    ReactUI["React Dashboard<br/>(Vite + Tailwind 4)"]:::frontend

    Slack -- "Live Events" --> SlackHandler
    SlackHandler -- "Triggers" --> LangGraph

    TriageAgent -- "Conditional Routing" --> DraftAgent
    DraftAgent -- "Invokes Tools" --> Tools

    Notion -- "Ingested via Seed Script" --> ChromaDB
    NotionTool -- "Retrieves" --> ChromaDB
    SFLookup -- "Queries" --> Salesforce
    SFTask -- "Creates Tasks" --> Salesforce
    EntityExt -- "Structured Parsing" --> GeminiLLM[("Gemini LLM")]

    DraftAgent -- "Saves Draft (HITL)" --> SQLite
    API -- "Approves / Edits / Rejects" --> SQLite
    SlackHandler -- "Posts Approved Reply" --> Slack

    API -- "Reads Data" --> SQLite
    ReactUI -- "Fetches Metrics" --> API
    ReactUI -- "SSE Real-Time updates" --> API

Loading

Agent Workflow

The LangGraph state machine processes each Slack message through a streamlined conditional pipeline:

Step	Agent / Node	Description
1	Triage Node	Classifies the message into escalation, deal_question, info_request, or noise using Gemini with structured JSON output.
2	Conditional Router	If classified as noise, routes directly to END. Otherwise, routes to the autonomous Drafting Agent.
3	Drafting Agent	Runs an autonomous ReAct loop (up to 5 iterations) to gather context via bound tools, then drafts a thread reply.
4	Human-in-the-Loop	Saves the finalized reply as a pending draft in SQLite. Admins review, edit, or approve the draft in the dashboard before it posts.

Agent Tool Execution Matrix (ReAct Loop):

• Escalations (Customer at risk): Dynamically uses extract_entities ➔ lookup_salesforce_account ➔ search_notion_docs ➔ create_salesforce_task (writes follow-up ticket in CRM).
• Deal Questions: Dynamically uses extract_entities ➔ lookup_salesforce_account (verifies pricing/contract details) ➔ search_notion_docs (pulls playbooks).
• Info Requests: Searches Notion documents directly using RAG.

---

Tech Stack

Layer	Technology
Backend	Python 3.12+, FastAPI, uv (package manager)
Agent Framework	LangGraph, LangChain
LLM / Embeddings	Google Gemini (gemini-1.5-flash & gemini-embedding-001)
Vector Store	ChromaDB (local persistence)
Observability	LangSmith (full trace logging)
Integrations	Slack Bolt (Socket Mode), simple-salesforce, notion-client
Database	SQLite + aiosqlite (async ORM via SQLAlchemy)
Frontend	React 19, Vite 8, Tailwind CSS 4, Recharts, Lucide Icons
Linting	OxLint (frontend)

---

Project Structure

OpsPilot/
├── backend/
│   ├── agents/                    # LangGraph agent nodes
│   │   ├── graph.py               # StateGraph definition & compilation
│   │   ├── state.py               # AgentState TypedDict schema
│   │   ├── triage.py              # Triage classification node (Gemini)
│   │   ├── drafting_agent.py      # Autonomous drafting agent (ReAct Loop) [CURRENT]
│   │   ├── tools.py               # RAG, Salesforce, and Extraction Tools [CURRENT]
│   │   ├── crm.py                 # Deprecated (Legacy 4-node flow)
│   │   ├── enrichment.py          # Deprecated (Legacy 4-node flow)
│   │   ├── notification.py        # Deprecated (Legacy 4-node flow)
│   │   └── prompts/               # Prompt templates
│   │       └── triage_prompt.py   # Classification prompt
│   ├── api/
│   │   ├── routes.py              # REST API endpoints (runs, metrics, weekly reports, approval)
│   │   └── schemas.py             # Pydantic validation schemas
│   ├── integrations/
│   │   ├── slack_handler.py       # Slack event listener + slash commands + App Home
│   │   ├── salesforce.py          # Salesforce client (real + mock)
│   │   ├── notion.py              # Notion client (real + mock)
│   │   └── vectorstore.py         # ChromaDB manager + RAG setup
│   ├── tracker/
│   │   ├── models.py              # SQLAlchemy AgentRun & PendingDraft models
│   │   └── logger.py              # Async DB logging & feedback updates
│   ├── scripts/
│   │   ├── seed_all.py            # Master seed orchestrator
│   │   ├── seed_salesforce.py     # Salesforce demo data seeder
│   │   ├── seed_notion.py         # Notion demo data seeder
│   │   ├── seed_rag.py            # ChromaDB ingestion script
│   │   ├── backdate_runs.py       # Populates 90 days of historical runs for dashboard metrics [NEW]
│   │   ├── simulate_load.py       # Simulates high volume message load for dashboard testing [NEW]
│   │   └── seed_slack_messages.py # Slack test message sender
│   ├── main.py                    # FastAPI app entrypoint + lifespan
│   ├── config.py                  # Pydantic settings (env-based)
│   └── pyproject.toml             # Python dependencies
├── frontend/
│   └── src/
│       ├── App.jsx                # Dashboard routing, layout & SSE updates
│       ├── api.js                 # Backend API client
│       ├── components/
│       │   ├── ROIDashboard.jsx    # Executive desk, ROI calculators, and automation rates
│       │   ├── ApprovalQueue.jsx   # HITL Review workspace (approve, reject, edit)
│       │   ├── RunTraceDetails.jsx # Complete execution trace timeline and LangSmith links
│       │   ├── EvaluationHub.jsx  # Regression charts, category F1s, and E2E judge reports
│       │   ├── MetricCards.jsx    # KPI summary cards
│       │   ├── TimeSeriesChart.jsx # Performance metrics over time
│       │   ├── ToolUsageRadar.jsx  # Radar chart of tool calls
│       │   └── VolumeChart.jsx    # Category distribution chart
│       └── index.css              # Global styles
├── evals/
│   ├── run_all_evals.py           # Evaluation runner script
│   ├── run_triage_eval.py         # 100-case triage classification evaluator
│   ├── run_rag_eval.py            # 20-case similarity retrieval evaluator
│   ├── run_e2e_eval.py            # 10-case LLM-as-a-judge quality evaluator
│   ├── fixtures/
│   │   └── triage_test_cases.json # Labeled test cases
│   └── results/
│       ├── full_eval_report.json  # Compiled evaluations JSON
│       └── run_output.json        # Evaluation output
└── README.md                      # Project documentation

---

Getting Started

Prerequisites

• Python 3.12+
• Node.js 18+ and npm
• uv — Python package manager
• API keys and credentials for Gemini, Slack, Salesforce, and Notion (specified in .env)

1. Clone the Repository

git clone https://github.com/Jethva-Parthiv/OpsPilot.git
cd OpsPilot

2. Configure Environment

cp .env.example .env

Edit .env with your API credentials (see env vars below).

3. Start the Backend

cd backend
uv sync                              # Install dependencies
uv run uvicorn main:app --reload      # Start FastAPI server on :8000

4. Start the Frontend

cd frontend
npm install                           # Install dependencies
npm run dev                           # Start Vite dev server on :5173

5. Seed Demo Data & Generate Simulation Logs (Optional — for dashboard metrics preview)

When running locally, you can generate historical logs to demonstrate dashboard trends:

# Seed local vector database
cd backend/scripts
uv run python seed_rag.py

# Simulate high volume message traffic
uv run python simulate_load.py --count 150 --fast

# Backdate runs across a 90-day window to showcase metrics trend lines
uv run python backdate_runs.py

---

API Reference

The backend exposes a comprehensive REST API for the React dashboard:

Method	Endpoint	Description
GET	/api/health	Health check
GET	/api/runs	Paginated list of agent runs
GET	/api/runs/{run_id}	Detailed run metrics (trace, tokens, latency, LangSmith links)
GET	/api/metrics/summary	Aggregated KPIs (runs, saved time, cost, net ROI)
GET	/api/metrics/roi?days=7	Period-specific ROI analysis, automation success metrics & tool statistics
GET	/api/metrics/weekly-report	Weekly email report highlights, successes & anomalies
GET	/api/metrics/feedback	Approved/rejected rates, thumbs reactions, edit distance
GET	/api/drafts/pending	Fetch all active drafts awaiting human review
POST	/api/drafts/{id}/approve	Send a pending draft to the Slack workspace
POST	/api/drafts/{id}/edit	Modify the message content and post to Slack
POST	/api/drafts/{id}/reject	Archive/dismiss a pending draft
GET	/api/evals/latest	Retrieve latest compiled evaluation run results

Example Response — /api/metrics/summary

{
  "total_runs": 142,
  "total_time_saved_seconds": 28640,
  "avg_time_saved_seconds": 201.6,
  "escalations_handled": 32,
  "categories": {
    "escalation": 32,
    "deal_question": 45,
    "info_request": 30,
    "noise": 35
  },
  "total_llm_cost_usd": 0.0425,
  "net_roi_usd": 397.75
}

---

Evaluation

Eval Type	Method	Result
Triage Classification	100 labeled messages, 4 categories	94% accuracy
RAG Retrieval	20 query-document pairs, top-3 hit rate	90%
Reply Quality	LLM-as-judge scoring completions across 5 axes	4.2 / 5.0

Run Evaluations

To run the complete evaluation suite and compile the combined results:

# From the project root
cd backend
uv run python ../evals/run_all_evals.py

Individual evaluations can also be run:

uv run python ../evals/run_triage_eval.py
uv run python ../evals/run_rag_eval.py
uv run python ../evals/run_e2e_eval.py

Detailed results are saved in evals/results/ and compiled in full_eval_report.json which dynamically updates the frontend Agent Evaluation Hub tab.

---

Dashboard Tab Navigations

The React dashboard provides granular observability into the agent system:

1. Executive Desk (ROI Dashboard): Financial value metrics (Net ROI, hours saved, avg cost per run) alongside automation success rate funnels and tool usage radars.
2. Run Trace Logs: Real-time run logs. Selecting a run displays a step-by-step trace timeline showing active agent state changes, execution parameters, inputs/outputs, and LangSmith deep links.
3. HITL Approvals (Human Desk): Real-time list of pending drafts. Reviewers can approve drafts directly, modify draft contents, or reject them. Actions immediately sync with the SQLite database and SSE clients.
4. Agent Evaluation (Eval Hub): Dynamically renders precision, recall, F1 metrics, RAG hit rates, and LLM judge quality scores of E2E runs.

---

Environment Variables

Variable	Description	Default
GEMINI_API_KEY	Google Gemini API key	mock-key
LLM_MODEL_NAME	LLM model for triage & drafting	gemini-1.5-flash
EMBEDDING_MODEL_NAME	Embedding model for RAG	gemini-embedding-001
SLACK_BOT_TOKEN	Slack bot OAuth token	mock-token
SLACK_SIGNING_SECRET	Slack signing secret	mock-secret
SLACK_APP_TOKEN	Slack app-level token (Socket Mode)	mock-app-token
SF_USERNAME	Salesforce username	—
SF_PASSWORD	Salesforce password	—
SF_SECURITY_TOKEN	Salesforce security token	—
SF_DOMAIN	Salesforce domain (login or test)	login
NOTION_API_KEY	Notion integration API key	—
NOTION_DATABASE_ID	Notion database ID for playbooks	—
CHROMA_PERSIST_DIR	ChromaDB persistence directory	./data/chroma
DATABASE_URL	SQLite database URL	sqlite+aiosqlite:///./data/agent_runs.db
LOG_LEVEL	Logging level	INFO
HOURLY_RATE_USD	Estimated team hourly wage for ROI calculation	50.0
LLM_COST_PER_MILLION_TOKENS	Gemini API dollar rate per million tokens	0.15
LANGSMITH_API_KEY	LangSmith API Key (for observability)	—
LANGSMITH_TRACING_ENABLED	Toggle LangSmith trace forwarding	true

---

Slack App Setup (Slash Commands & App Home)

To register and use the slash commands and App Home tab, configure your Slack App manifest or settings at api.slack.com/apps:

1. Register Slash Commands

Add these commands in Features -> Slash Commands:

• /opspilot-triage — Classify any message with OpsPilot's AI triage
• /opspilot-status — Show today's operational stats
• /opspilot-lookup — Look up a Salesforce account by company name

Note: Since the app uses Socket Mode, no Request URLs are needed; leave them blank/empty.

2. Enable App Home

• Go to Features -> App Home
• Toggle Home Tab to enabled
• Optionally toggle Messages Tab to allow DM actions

3. Required Scopes & Events

• Bot Token Scopes (Features -> OAuth & Permissions):
  • commands — required for slash commands
  • chat:write — required to send replies
• Bot Events (Features -> Event Subscriptions):
  • app_home_opened — required to trigger rendering the home tab

---

License

This project is for portfolio and demonstration purposes.