Megatron-LM is a GPU-optimized library developed by NVIDIA for training transformer models at scale, containing two complementary components. Megatron-LM serves as a reference implementation with pre-configured training scripts designed for research teams and quick experimentation with distributed training. Megatron Core is a composable library providing GPU-optimized building blocks for developers constructing custom training frameworks, offering transformer building blocks, advanced parallelism strategies including tensor parallelism, pipeline parallelism, data parallelism, expert parallelism, and context parallelism, along with mixed precision support for FP16, BF16, FP8, and FP4 formats.
The repository demonstrates substantial production capability, successfully training models ranging from 2 billion to 462 billion parameters across thousands of GPUs. Performance benchmarking on H100 clusters achieves up to 47 percent Model FLOP Utilization on weak scaling tests, with superlinear scaling improvements as model size increases from 41 percent to 47-48 percent MFU. Strong scaling tests of a 175 billion parameter GPT-3 model scale from 96 to 4,608 H100 GPUs while maintaining consistent batch sizes, demonstrating practical scalability for production workloads.
The codebase is written in Python and maintains active development with significant community engagement.
Recent developments highlight expanding model architecture support and optimization techniques. The repository includes initial DeepSeek-V4 implementation on the development branch, support for emerging optimizers through the Emerging-Optimizers library, Falcon-H1 hybrid transformer-Mamba architecture contributions from Technology Innovation Institute, and dynamic context parallelism achieving up to 1.48x speedup for variable-length sequence training. Megatron Bridge provides bidirectional checkpoint conversion between Hugging Face and Megatron formats with production-ready recipes for popular models.
Installation options include PyPI packages or source compilation, with documentation available through NVIDIA's official guides covering installation, first training runs, parallelism strategies, and contribution procedures. The development process moved to open GitHub in December 2025, enabling transparent community contributions and continuous integration in public view.
Benchmark configurations specify vocabulary size of 131,072 tokens and sequence length of 4,096 tokens with varied model dimensions to achieve target parameter counts. Communication optimizations include fine-grained overlapping for data parallelism, tensor parallelism, and pipeline parallelism. The codebase includes comprehensive checkpointing and fault tolerance mechanisms for production training pipelines. Upcoming releases will drop Python 3.10 support, requiring downstream applications to adopt Python 3.12 or later for compatibility with Megatron Core 0.17.0 and subsequent versions.
