Whisper Diarization is a speech recognition and speaker identification tool that combines OpenAI Whisper's transcription capabilities with speaker diarization to attribute each transcribed segment to its speaker.
The tool addresses the problem of identifying who spoke each part of a multi-speaker audio recording. It works by first extracting vocals from the audio to improve speaker identification accuracy, then generating transcription using Whisper. The timestamps from transcription are corrected and aligned using ctc-forced-aligner to minimize timing errors. The audio is processed through MarbleNet for voice activity detection and segmentation to exclude silence, then TitaNet extracts speaker embeddings to identify speakers for each segment. These embeddings are associated with the aligned timestamps to determine the speaker for each word, with final realignment using punctuation models to compensate for minor time shifts.
Adopters should note that the tool requires Python 3.10 or higher, with FFMPEG and Cython as prerequisites. Systems with at least 10GB of VRAM can use the parallel processing variant, which runs NeMo and Whisper concurrently for potentially faster results, though this mode is marked as experimental. The project is suitable for applications requiring speaker-attributed transcription of multi-speaker audio, such as meeting transcription, interview analysis, or podcast processing. The tool's approach of combining multiple specialized models for VAD, speaker embedding, and alignment differs from simpler single-model approaches that may struggle with accurate speaker attribution.
The project maintains active testing infrastructure with automated test workflows. Development appears focused on core functionality, with the README noting that command-line argument support for adjusting Whisper and NeMo parameters is planned for future addition.
