RedPajama-Data is a dataset preparation toolkit for training large language models on large-scale text corpora.
The project addresses the challenge of preparing raw web-crawled text into high-quality training data for language models. It processes documents from CommonCrawl snapshots through a three-stage pipeline: artifact creation, quality signal computation, and deduplication. The approach uses the CCNet pipeline for initial processing and applies multiple filtering and weighting techniques including quality classifiers, bag-of-ngram generative models for importance weighting, bad-word filtering, and URL blacklist checking. The toolkit supports multilingual data across English, German, French, Italian, and Spanish.
Organizations building large language models should consider this toolkit if they need to process CommonCrawl data at scale and want reproducible, open-source preparation code. The project is most suitable for teams with infrastructure to run containerized workloads and access to S3 storage, as the pipeline assumes Docker and Apptainer installations. The toolkit provides pre-computed quality signals and deduplication for a substantial portion of its dataset, reducing the computational burden for users who can work with these pre-processed versions rather than running the full pipeline from scratch.
The project maintains active development with regular updates to its codebase and documentation. The pipeline implementation is modular, allowing users to run individual steps independently or skip early stages if they have alternative data sources. Configuration is managed through environment files, and the toolkit includes detailed setup instructions for both containerized and non-containerized execution paths. The developers emphasize reproducibility by requiring consistent hash seed settings across distributed processing steps.
