GPT-NeoX is a library for training large-scale autoregressive transformer language models on GPUs, built on NVIDIA's Megatron and augmented with DeepSpeed techniques and novel optimizations.
The library addresses the challenge of training billion-parameter language models by implementing model parallelism across distributed GPU systems. It combines techniques from Megatron and DeepSpeed while adding its own optimizations to make large-scale training more accessible. The tool supports distributed training with ZeRO and 3D parallelism, and includes architectural innovations such as rotary and alibi positional embeddings, parallel feedforward attention layers, and flash attention.
GPT-NeoX is designed specifically for researchers and practitioners training models from scratch at scale, not for generic inference tasks. The README explicitly recommends Hugging Face's transformers library for inference needs. The tool distinguishes itself through broad hardware and system support, having run successfully on AWS, various national laboratory supercomputers including ORNL Summit and Frontier, and other HPC environments. It supports multiple job schedulers including Slurm, MPI, and IBM Job Step Manager. The library includes predefined configurations for popular architectures like Pythia, PaLM, Falcon, and LLaMA variants, and integrates with the open-source ecosystem through Hugging Face tokenizers and transformers, experiment monitoring via WandB, Comet, and TensorBoard, and evaluation through the Language Model Evaluation Harness. It has seen adoption across academic, industry, and government institutions.
The project maintains active development with recent additions including Transformer Engine integration, support for preference learning methods like DPO and KTO, and reward modeling capabilities. The codebase reflects ongoing refinement of distributed training techniques and integration with evolving ecosystem tools.
