Voice Engine - AI Voice Assistant

A custom, end-to-end conversational voice agent built from scratch without using any third-party voice orchestration frameworks (e.g., Pipecat, LiveKit). It leverages Sarvam AI's APIs for STT, LLM, and TTS, delivering a real-time, low-latency conversational experience.

🎥 [Demo Video]

https://www.loom.com/share/430769d6c5fb423eb754e71b4c2d70fb

✨ Features Implemented

Core Requirements:

• From-Scratch Pipeline: Zero usage of open-source voice frameworks. Custom handling of WebSocket connections, audio chunking, and streaming.
• Sarvam Stack: Integrates saaras:v3 (STT), sarvam-30b (LLM), and bulbul:v3 (TTS).
• Web UI: React-based frontend to initiate and interact with the voice bot.

Optional Enhancements & Advanced Features:

• Live Transcripts: Real-time chat UI displaying both user and agent messages.
• Tool Calling (with Visuals): The LLM is equipped with a get_weather tool, automatically rendering visual indicators (🔧) in the UI when executed.
• Real-Time Latency Metrics: Live dashboard tracking STT Latency, LLM Time-to-First-Token (TTFT), LLM Total time, TTS First Audio, and End-to-End latency.
• Barge-in / Interruption Handling: Detects START_SPEECH events from the STT engine, instantly aborting ongoing LLM generation and flushing TTS frontend playback buffers for natural interruptions.

🏗️ Architecture Summary

• Frontend (React): Captures raw PCM audio via the Web Audio API and streams it over WebSockets. Handles incoming audio chunks dynamically using Media Source Extensions (MSE) for true streaming playback rather than waiting for complete files.
• Backend (FastAPI): Orchestrates the pipeline using asyncio.TaskGroup for true concurrency.
  • Wraps incoming raw PCM into WAV in-memory for STT consumption.
  • Streams STT transcripts directly into the LLM context.
  • Sentence-buffers LLM streams to chunk output to the TTS engine for minimal Time-To-First-Audio.

🚀 Setup Instructions

1. Backend (Python/FastAPI)

cd backend
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Create a .env file in the backend directory:

SARVAM_API_KEY=your_sarvam_api_key_here

Start the server:

uvicorn main:app --host 0.0.0.0 --port 8000

2. Frontend (React/Vite)

cd frontend
npm install
npm run dev

Open http://localhost:5173 in your browser. Click Start, allow microphone permissions, and begin speaking!