The stable-audio-metrics repository provides a collection of metrics specifically designed for evaluating music and audio generative models, with particular emphasis on assessing long-form, full-band, and stereo audio generations. Developed by Stability AI, this Python-based toolkit addresses the practical evaluation needs of realistic audio generation scenarios that existing metrics were not optimized for.
The repository implements three primary evaluation metrics, each operating at different sampling rates and based on established audio analysis frameworks. The Fréchet Distance metric operates at 48kHz and is built upon Openl3, a deep learning-based audio feature extraction system. The Kullback-Leibler divergence metric runs at 32kHz and leverages PaSST, a pre-trained audio spectrogram transformer. The CLAP score metric also operates at 48kHz and is derived from CLAP-LAION, a contrastive learning framework for audio and language. A key design principle across all three metrics is their ability to handle variable-length audio inputs, making them suitable for evaluating generations of different durations without requiring preprocessing constraints.
The toolkit is optimized exclusively for GPU computation, as CPU execution would be prohibitively slow for practical use. Installation requires cloning the repository, setting up a Python virtual environment, and installing dependencies from a requirements file. The documentation notes that GPU support may require CUDA 11.8 due to dependencies from the Openl3 framework, addressing a potential compatibility issue users might encounter.
The repository provides comprehensive documentation through source files for each metric implementation and includes example scripts demonstrating usage with multiple datasets. Three primary datasets are supported through example implementations: MusicCaps, AudioCaps, and Song Describer. The examples folder contains both standard evaluation scripts and specialized "no-audio" variants that allow users to run evaluations without downloading full datasets, as reference statistics and embeddings are precomputed and included in the load directory. The CLAP score does not include precomputed embeddings due to its computational efficiency.
The toolkit includes built-in functionality for fair model comparison, particularly when benchmarking against Stable Audio. The metrics automatically handle resampling and mono-to-stereo conversion, ensuring that models outputting audio at different sampling rates or channel configurations can be evaluated on equal terms. This normalization capability is critical for meaningful cross-model comparisons.
Usage requires organizing generated audio files according to dataset-specific naming conventions. For MusicCaps, files are named using the ytid identifier from the prompts file, with 5,521 expected generations. For AudioCaps, files use the audiocap_id, with 4,875 expected generations. The repository demonstrates extensibility by including examples for additional datasets like Song Describer, allowing users to adapt the framework to their own datasets by following the established data structure patterns.
The repository is classified across multiple domains including audio evaluation, sound metrics, generative audio, model assessment, audio quality, perceptual metrics, deep learning, audio synthesis, performance measurement, and sound analysis, reflecting its comprehensive scope in the audio generation evaluation space.
