VoiceCraft is a neural codec language model that performs zero-shot speech editing and text-to-speech synthesis on real-world audio.
The tool addresses the challenge of editing speech and generating natural-sounding voices without requiring extensive training data for each speaker. It works by using a token infilling approach: given a few seconds of reference audio from a target voice, the model can either edit existing speech by replacing portions while preserving the speaker's characteristics, or generate new speech in that voice from text. The approach treats speech as tokens in a neural codec and uses language modeling to predict missing or new tokens conditioned on the reference audio context.
Developers should choose this tool if they need to perform speech editing or text-to-speech on diverse, uncontrolled audio sources like audiobooks, podcasts, or internet videos rather than clean studio recordings. The project suits applications requiring speaker cloning with minimal reference material and inference flexibility. Multiple deployment options are available: a Gradio interface for interactive use, Docker containerization for reproducibility, standalone command-line scripts for integration into other projects, and Jupyter notebooks for experimentation. The tool provides different model sizes to balance quality and computational requirements.
The project maintains active development with recent improvements to inference sampling strategies that substantially enhanced editing and synthesis quality. Multiple model variants are available on the model hub, including versions specifically optimized for text-to-speech tasks. The codebase includes comprehensive documentation for inference workflows and training procedures, with clear guidance on environment setup and model selection based on use case requirements.
