Meeting Copilot

A local-first agent that listens to your meetings, extracts the decisions, and ships the follow-ups before the call ends. Everything runs on your machine. Nothing leaves it.

---

Why local-first

Every other meeting assistant sends your audio to a third party. This one doesn't. The whole stack — speech-to-text, speaker diarization, the LLM that extracts action items, the database, the dashboard — runs on a single host. The only infrastructure is Postgres in Docker.

That tradeoff gets you:

• No rate limits. A two-hour call is a two-hour call, not a metered event.
• No data residency theatre. The audio never leaves the box.
• Cheap to run. A laptop is enough; no monthly bill.
• Inspectable. Every model prompt and every LLM response is persisted in runs.llm_call for audit.

The price is hardware: the defaults are tuned for ~6.7 GB RAM and a CPU. The hardware tiers section covers what to change for a beefier host.

---

What it does

1. Capture — A Chrome MV3 extension grabs the current tab's audio (chrome.tabCapture → offscreen AudioContext → 16 kHz mono Int16 PCM) and streams it over a binary WebSocket to the backend.
2. Transcribe + diarize — Every 3-second window goes through faster-whisper (incremental) and pyannote.audio (speaker turns), in series, behind a per-meeting asyncio.Semaphore so peak RAM stays bounded.
3. Extract notes — A real langgraph.graph.StateGraph (not a while True loop) calls a local Ollama model (phi-4-mini-reasoning Q4_K_M) with a JSON schema and merges results into the four buckets: action items, decisions, blockers, unresolved questions, plus a rolling summary.
4. Stream to the dashboard — A second WebSocket fans LiveEvent deltas to a Next.js 15 dashboard. The transcript types itself in, the notes pulse when they update, and the speaker map reshuffles.
5. Finalize — On stop, a second-pass transcription runs (medium int8 on CPU / large-v3 on GPU), then the finalize node produces an executive summary, key topics, and per-person follow-up email drafts.

---

Quick start

1. Clone and configure

git clone https://github.com/Sane219/meeting-copilot.git
cd meeting-copilot
cp .env.example .env
# Optional: edit HUGGINGFACE_TOKEN in .env (see "Hugging Face setup" below)

2. Bring up the data + model layers

make up              # docker compose up -d (postgres, ollama, backend, frontend)
make ollama-pull     # downloads phi-4-mini-reasoning Q4_K_M (~2.4 GB)
make db-migrate      # alembic upgrade head
make db-seed         # creates a demo meeting so the dashboard isn't empty

3. Build the Chrome extension

make extension-install
make extension-build

Then load chrome-extension/dist/ in chrome://extensions (Developer mode → "Load unpacked").

4. Use it

1. Open a meeting tab (Google Meet, Zoom, Teams, anything with audio).
2. Click the extension icon → Start capture. Leave the popup; the offscreen document keeps recording.
3. Open http://localhost:3000/meetings/{id}/live to see the transcript, speakers, and structured notes appear in real time.
4. When the call ends, hit Stop, then Run final pass to get the executive summary and follow-up drafts.

---

Hugging Face setup (pyannote is gated)

pyannote/speaker-diarization-3.1 requires a one-time license acceptance:

1. Create a token at https://huggingface.co/settings/tokens.
2. Accept the license at https://huggingface.co/pyannote/speaker-diarization-3.1.
3. Put the token in .env as HUGGINGFACE_TOKEN=hf_....

Without it, the backend will start, but every meeting will fail at the diarization step with a clear error.

---

Stack

Layer	Choice
Browser capture	Chrome MV3 · tabCapture · offscreen AudioContext + ScriptProcessor (AudioWorklet-swappable)
Backend	FastAPI · async SQLAlchemy 2 · asyncpg
Agent	LangGraph StateGraph (compiled, typed reducers, real graph)
LLM	Ollama serving phi-4-mini-reasoning (Unsloth GGUF, Q4_K_M)
STT	faster-whisper (small int8 incremental, medium int8 final)
Diarization	pyannote.audio 3.1
Realtime	Two WebSockets: ingest (binary PCM) and live (JSON events)
DB	Postgres 16 + pgvector (speaker embeddings)
Dashboard	Next.js 15 (App Router) · TS strict · TanStack Query · Tailwind
Python tooling	uv · Ruff · mypy strict · pytest
Frontend lint	ESLint next/core-web-vitals · Prettier

No paid APIs. No OpenAI, no Deepgram, no AssemblyAI, no hosted services. The only paid thing is your own electricity.

---

Hardware tiers

The defaults target a low-end host (6.7 GB RAM, AMD APU, no GPU). Bump the models for a beefier host.

Tier	RAM	GPU	LLM	Whisper incremental	Whisper final	Notes
Minimum	6 GB	none	phi-4-mini Q4_K_M	small int8	medium int8	Default
Low memory	4 GB	none	phi-4-mini Q3_K_M	base int8	small int8	LOW_MEMORY_MODE=1 in .env
GPU host	16 GB+	CUDA	phi-4-mini Q4_K_M	small int8	large-v3	Auto-bumps on CUDA

The model selection is env-driven, not per-meeting. See plan.md for the per-model sizing math.

---

Documentation hub

Deep dives for contributors and integrators:

• Architecture — system diagram, audio pipeline sequence diagram, LangGraph state machine, ER diagram.
• API reference — REST endpoints, WebSocket frames, Pydantic schemas, security roadmap.
• Contributing — local dev loop, style guide, testing, PR checklist.
• Code of Conduct — Contributor Covenant 2.1.
• Product register — the design constraints that shaped the dashboard UI.
• Original plan — the full design + hardware reasoning.

---

Development

make dev              # up + pull models + migrate + seed
make test             # pytest (fakes for ML/LLM, finishes in <30s)
make lint             # ruff + eslint
make typecheck        # mypy + tsc --noEmit
make docker-config    # validates docker-compose.yml

The test suite uses in-memory SQLite and fakes for Ollama / Whisper / pyannote, so CI never needs the heavyweight stack. See backend/app/tests/conftest.py.

---

Known limitations / roadmap

• Single-user, localhost. No auth, permissive CORS. The security roadmap covers the next iteration (bearer token, origin checks, retention policy).
• No email sending. Per-person follow-up drafts are rendered in the UI with one-click clipboard copy.
• No Postgres-backed LangGraph checkpointer. v0.1.0 keeps state in-process (lost on restart). The agent graph is a real StateGraph; the swap to langgraph.checkpoint.PostgresSaver is mechanical.
• Cross-meeting speaker resolution is implemented at the data layer (pgvector) but the UI only renames within a meeting. The matcher ships next.
• ScriptProcessor resampler. A real AudioWorkletProcessor module is the planned drop-in.
• No calendar/email integration. v0.1.0 is a listener + a writer. Calendar invites and email drafts are local-only.

---

License

MIT.

---

Contact

Open an issue for bugs and feature requests. For security disclosures, see CONTRIBUTING.md § Security.