MuseTalk is a real-time audio-driven lip-syncing model that modifies facial video to match input audio with high fidelity.
The tool addresses the problem of creating convincing lip-synchronized video for virtual humans and dubbed content. It works by operating in the latent space of a frozen VAE encoder, taking audio input processed through a Whisper model and fusing audio embeddings with image features via cross-attention mechanisms borrowed from Stable Diffusion's UNet architecture. The approach enables real-time inference at 30 frames per second on standard GPU hardware while maintaining visual quality and synchronization accuracy.
MuseTalk suits projects requiring virtual human generation, video dubbing, or lip-sync correction where real-time performance matters. It handles multilingual audio including Chinese, English, and Japanese, and works with unseen faces at 256x256 resolution. The tool integrates with other virtual human pipelines, particularly MuseV for complete end-to-end solutions. Developers can train custom models using the provided training scripts and configurations, with checkpoints available trained on public and private datasets.
The project maintains active development with regular releases introducing significant improvements. Training code was open-sourced alongside version 1.5, which incorporated perceptual loss, GAN loss, and sync loss to enhance overall performance. A two-stage training strategy and spatio-temporal data sampling approach were implemented to balance visual quality against lip-sync accuracy. Pretrained model weights are available, and the team provides both inference and training code alongside a Gradio demo for experimentation.
