NeMo Curator is a scalable data preprocessing and curation toolkit for large language models that runs GPU-accelerated pipelines on single machines or multi-node Ray clusters.
The tool addresses the challenge of preparing large multimodal datasets for AI training. It provides a unified framework for loading, filtering, deduplicating, and transforming text, image, video, and audio data at scale. The approach centers on Ray-based distributed pipelines that can execute the same workflow locally or across HPC clusters via Slurm, enabling teams to develop and test on smaller hardware before scaling to production workloads.
Teams building LLM training datasets should consider this tool when they need to process large volumes of multimodal data with quality filtering and deduplication. It suits projects requiring repeatable, production-grade pipelines that can run across different infrastructure scales. The toolkit supports text operations including deduplication, classification, and quality filtering; image operations such as aesthetic filtering and NSFW detection; video capabilities like scene detection and clip extraction; and audio workflows including ASR transcription and quality assessment. The project includes an inference server component that provides an OpenAI-compatible LLM endpoint for synthetic data generation and classification tasks within pipelines.
The project maintains active development across multiple modalities with recent work expanding audio curation capabilities including composite quality filtering, audio tagging, and speaker diarization. Development includes ongoing infrastructure improvements such as Slurm deployment support for HPC environments and API refinements to the Resources configuration system. The codebase receives regular updates with new modality support and runtime upgrades integrated into the pipeline architecture.
