Delayed Streams Modeling is a framework for building speech-to-text and text-to-speech models that handle streaming audio and text tasks in real time.
The framework addresses the challenge of processing continuous audio or text streams with minimal latency while maintaining accuracy. It formalizes an approach used in prior models like Moshi and Hibiki, enabling efficient streaming inference across multiple modalities. The Kyutai STT models built on this framework process audio in chunks, return word-level timestamps, and include semantic voice activity detection in the smaller variant, making them suitable for interactive voice applications.
The speech-to-text models come in two sizes: a 1 billion parameter bilingual model supporting English and French with 0.5 second latency, and a 2.6 billion parameter English-only model with 2.5 second latency. Both support real-time streaming and efficient batching. Developers should choose this tool if they need production-grade speech recognition with streaming capabilities and word-level timing information. The project provides three separate implementations tailored to different deployment contexts: PyTorch for research and experimentation, Rust for production servers with websocket streaming, and MLX for on-device inference on Apple hardware. The Rust server implementation can serve 64 simultaneous connections on consumer GPU hardware at real-time performance factors.
Development activity shows consistent focus on practical deployment scenarios. The project maintains multiple implementation paths rather than a single codebase, indicating responsiveness to different user needs across research, production, and edge deployment. Documentation includes interactive notebooks and clear guidance on selecting the appropriate implementation for specific use cases. The team has built a complete voice AI system called Unmute using these models, demonstrating confidence in the framework's production readiness.
