Modded-NanoGPT is a language model training framework that optimizes the speed of training a 124-million-parameter model to reach a target validation loss on the FineWeb dataset.
The project addresses the problem of slow language model training by implementing a competitive speedrun to discover the fastest algorithms for training on eight NVIDIA H100 GPUs. The approach combines architectural innovations with training optimizations, including rotary embeddings, QK-Norm, ReLU-squared activations, the Muon optimizer, mixed-precision training with FP8, Flash Attention 3 with sliding window patterns, and numerous other techniques such as learnable skip connections, batch size scheduling, and multi-token prediction. The framework achieves its target performance in under 75 seconds on the specified hardware, a dramatic improvement over the 45-minute baseline from prior work.
This tool suits researchers and practitioners interested in efficient language model training who have access to high-end GPU resources. The project is descended from NanoGPT and llm.c, making it relevant for those already familiar with that lineage. It represents a collaborative effort to push the boundaries of training speed rather than a production-ready framework, so adoption makes sense for those exploring optimization techniques or benchmarking their own implementations against state-of-the-art results.
The maintainers typically respond to new issues and pull requests within a day.
