Vision Agents is an open-source Python framework by Stream for building multimodal AI agents that process video and voice in real time. The project enables developers to create intelligent agents that combine computer vision models, large language models, and audio processing with ultra-low latency by leveraging Stream's edge network infrastructure. The framework is designed to work with any model provider and video infrastructure, making it flexible for diverse use cases ranging from sports coaching to security monitoring.
The core strength of Vision Agents lies in its real-time capabilities. The framework achieves join times of 500 milliseconds and maintains audio and video latency under 30 milliseconds, which is critical for interactive applications. It provides native API access to major LLM providers including OpenAI, Gemini, and Claude, ensuring developers can always use the latest model capabilities. The framework includes a pluggable video processing pipeline that allows integration of computer vision models like YOLO and Roboflow before or after LLM calls, enabling frame-by-frame understanding of video streams.
The feature set is comprehensive for production deployment. Vision Agents supports real-time WebRTC for direct video streaming to model providers, natural conversation flow through voice activity detection and diarization, tool calling and Model Context Protocol support for executing code and APIs mid-conversation, phone integration via Twilio and Telnyx for inbound and outbound calls, retrieval-augmented generation with vector search backends, persistent memory across conversation turns and sessions, and a text back-channel for silent messaging during calls. The framework includes production-ready components such as a built-in HTTP server, Prometheus metrics, horizontal scaling support, and Kubernetes deployment guidance.
The integration ecosystem is extensive. Vision Agents supports multiple speech-to-text providers including Deepgram, AssemblyAI, and Fast-Whisper, text-to-speech services from ElevenLabs, Cartesia, and AWS Polly, vision models from Ultralytics and Roboflow, and realtime AI models from OpenAI, Gemini, AWS Nova, and others. The framework also integrates with avatar systems, turn detection services, and specialized tools like TurboPuffer for RAG operations.
Bug reports and dependency updates represent the most common issue types. The repository shares contributors with related projects including LobHub, Google's Gemini CLI, and Hugging Face Hub, indicating cross-pollination within the AI agent ecosystem.
The project provides extensive documentation and examples covering voice agents with low-latency RAG, realtime coaching with pose tracking, video restyling with avatars, custom computer vision models for security, and phone-based workflows with tool integration. Installation is straightforward via the uv package manager, with optional integrations available for specific providers. Stream offers free API credentials with 333,000 participant minutes monthly plus additional credits through their Maker Program, lowering barriers to entry for developers building production video and voice AI applications.
