SAM-Audio is a foundation model for isolating sounds in audio using text, visual, or temporal prompts.
The tool addresses the problem of separating specific sounds from complex audio mixtures. It accepts three types of prompts: natural language descriptions in lowercase noun-phrase or verb-phrase format, visual cues from video frames with masks, or time spans indicating where target sounds occur. The model relies on the Perception-Encoder Audio-Visual backbone for its multimodal capabilities. For text prompting, the tool can optionally predict time spans automatically when enabled, which helps separate non-ambience sound events. It supports generating multiple candidate separations and ranking them using CLAP for text-audio similarity, a Judge model that scores separation quality across precision, recall, and faithfulness dimensions, or ImageBind embeddings for visual prompting scenarios.
Developers should note that checkpoint access requires prior approval through the Hugging Face model repository and authentication via access token. The tool requires Python and a CUDA-compatible GPU is recommended for practical use. It suits projects involving audio source separation, audio-visual content analysis, or applications requiring fine-grained control over which sounds to isolate from mixed audio. The approach of supporting multiple prompt modalities distinguishes it from single-modality separation tools.
The project provides example notebooks demonstrating usage patterns and maintains an active codebase with inference implementation, model checkpoint links, and documentation of the three prompting methods and reranking capabilities.
