Open-source framework for developing real-time multimodal conversational AI agents.
The AI Agent SDK is a Python framework built on top of the VideoSDK Python SDK that enables AI-powered agents to join VideoSDK rooms as participants. This SDK serves as a real-time bridge between AI models (like OpenAI and Gemini) and your users, facilitating seamless voice and media interactions.
- Real-time voice and media: agents can listen, speak, and interact live in meetings
- SIP and telephony integration: seamlessly connect agents to phone systems via SIP for call handling, routing, and PSTN access
- Virtual avatars: add lifelike avatars to enhance interaction and presence
- Multi-model support: integrate with OpenAI, Gemini, AWS NovaSonic, and more
- Cascading pipeline: integrates with different providers of STT, LLM, and TTS seamlessly
- Conversational flow: VAD, turn detection, RAG, and memory for smooth, context-aware conversations
- Function tools: extend agent capabilities with event scheduling, expense tracking, and more
- Client-side function calling: trigger actions directly from the client side for low-latency interaction
- MCP integration: connect agents to external data sources and tools using Model Context Protocol
- A2A protocol: enable agent-to-agent interactions for complex workflows
- SDK support: available for web, mobile, gaming, and IoT
- Recording and transcription: capture and analyze conversations
- In-built observability: access real-time and historical traces, logs, and metrics for monitoring and debugging
- Deploy and scale: run on FlyIO, Kubernetes, or Agent Cloud (deploy close to your users)
- Global coverage: infrastructure optimized for international availability and low latency
- Scale and deploy on your cloud: flexible deployment to meet enterprise compliance and privacy needs
- Security and compliance covered: enterprise-grade security protocols
- End-to-end encryption of streams: ensure secure communication across the board
Important
Star VideoSDK Repositories ⭐️
Get instant notifications for new releases and updates. Your support helps us grow and improve VideoSDK!
- 🖥️ Your Backend: Hosts the Worker and Agent Job that powers the AI agents
- ☁️ VideoSDK Cloud: Manages the meeting rooms where agents and users interact in real time
- 📱 Client SDK: Applications on user devices (web, mobile, or SIP) that connect to VideoSDK meetings
- 📝 Register: Your backend worker registers with the VideoSDK Cloud
- 📲 Initiate to join Room: The user initiates joining a VideoSDK Room via the Client SDK on their device
- 📡 Notify worker for Agent to join Room: The VideoSDK Cloud notifies your backend worker to have an Agent join the room.
- 🤖 Agent joins the room: The Agent connects to the VideoSDK Room and can interact with the user.
Before you begin, ensure you have:
- A VideoSDK authentication token (generate from app.videosdk.live)
- A VideoSDK meeting ID (you can generate one using the Create Room API or through the VideoSDK dashboard)
- Python 3.12 or higher
- Third-Party API Keys:
- API keys for the services you intend to use (e.g., OpenAI for LLM/STT/TTS, ElevenLabs for TTS, Google for Gemini etc.).
-
Create and activate a virtual environment with Python 3.12 or higher.
💻 macOS / Linux
python3 -m venv venv source venv/bin/activate🪟 Windows
python -m venv venv venv\Scripts\activate
-
Install the core VideoSDK AI Agent package
pip install videosdk-agents
-
Install Optional Plugins. Plugins help integrate different providers for Realtime, STT, LLM, TTS, and more. Install what your use case needs:
# Example: Install the Turn Detector plugin pip install videosdk-plugins-turn-detector👉 Supported plugins (Realtime, LLM, STT, TTS, VAD) are listed in the Supported Libraries section below.
Now that you've installed the necessary packages, you're ready to build!
- For detailed guides, tutorials, and API references, check out our official VideoSDK AI Agents Documentation.
- To see the framework in action, explore the code in the Examples directory. It is a great place to quickstart.
This architecture shows how AI voice agents connect to VideoSDK meetings. The system links your backend with VideoSDK's platform, allowing AI assistants to interact with users in real-time.
The framework supports integration with various AI models and tools, including:
| Provider | Real-time Models | Speech-to-Text (STT) | Language Models (LLM) | Text-to-Speech (TTS) | Voice Activity Detection (VAD) | Turn Detection Model |
|---|---|---|---|---|---|---|
| OpenAI | OpenAIRealtime | OpenAISTT | OpenAILLM | OpenAITTS | ✖️ | ✖️ |
| GeminiRealtime | GoogleSTT | GoogleLLM | GoogleTTS | ✖️ | ✖️ | |
| AWS | AWSNovaSonic | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ |
| Sarvam | ✖️ | SarvamSTT | SarvamLLM | SarvamTTS | ✖️ | ✖️ |
| Deepgram | ✖️ | DeepgramSTT | ✖️ | ✖️ | ✖️ | ✖️ |
| ElevenLabs | ✖️ | ✖️ | ✖️ | ElevenLabsTTS | ✖️ | ✖️ |
| Silero VAD | ✖️ | ✖️ | ✖️ | ✖️ | SileroVAD | ✖️ |
| Hugging Face | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | Turn Detection |
The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in VideoSDK's Discord community.
When contributing, consider developing new plugins or enhancing existing ones to expand the framework's capabilities. Your contributions can help integrate more AI models and tools, making the framework even more versatile.
We love our contributors! Here's how you can contribute:
- Open an issue if you believe you've encountered a bug.
- Follow the documentation guide to get your local dev environment set up.
- Make a pull request to add new features/make quality-of-life improvements/fix bugs.