The speech-to-speech repository is a Python-based framework for building local voice agents using open-source models. It provides a fully modular, low-latency pipeline that chains together voice activity detection, speech-to-text, language model inference, and text-to-speech synthesis. The entire pipeline is exposed through an OpenAI Realtime-compatible WebSocket API, making it compatible with existing OpenAI Realtime clients while allowing users to swap out any component with alternatives. Every stage of the pipeline runs in its own thread connected by queues, enabling concurrent processing and streaming responses.
The project overlaps with other major Hugging Face ecosystem repositories including chat-ui, langchain, and llama_index, suggesting it serves as a bridge between conversational AI frameworks and voice interaction capabilities.
The default configuration uses Silero VAD v5 for voice activity detection, Parakeet TDT for speech-to-text transcription, OpenAI-compatible APIs for the language model component, and Qwen3-TTS for speech synthesis. However, the architecture supports multiple interchangeable backends for each stage. Speech-to-text options include Whisper, Faster Whisper, Lightning Whisper MLX, MLX Audio Whisper, and Paraformer. For the language model slot, users can point to hosted providers, Hugging Face Inference Providers, or self-hosted servers running vLLM or llama.cpp. Text-to-speech alternatives include Kokoro-82M, Pocket TTS, ChatTTS, and MMS TTS. Platform-specific optimizations are built in, with automatic resolution of macOS and non-macOS dependencies through pyproject.toml markers.
The repository supports multiple run modes beyond the default realtime server. A local mode allows direct interaction through the machine's microphone and speakers without requiring a separate client. Raw WebSocket mode streams raw PCM audio for minimal custom clients. TCP socket mode provides a simple interface for remote server deployments, though it omits some Realtime API features like interruption handling and tool-call events. Docker support is included with automatic NVIDIA Container Toolkit integration for GPU acceleration.
The realtime server implements the core OpenAI Realtime event set, supporting live transcription, low-latency turn-taking, and tool calls. An optional LLM proxy feature exposes the configured language model as a standard OpenAI-compatible endpoint, allowing concurrent side tasks like summaries and background agents without interruption from voice input. The implementation is designed for production use, currently serving as the conversation backend for thousands of Reachy Mini robots.
Installation requires Python 3.10 or later. The default installation covers the standard realtime pipeline with platform-specific optimizations. Optional backends are available through pip extras, with deprecated implementations archived separately. The codebase emphasizes modularity and ease of modification, focusing on models available through Transformers and the Hugging Face Hub. Multi-language support is documented, and the CLI provides extensive configuration options for selecting specific backends and adjusting model parameters.
