Train LLM From Scratch is a tutorial and implementation framework that teaches how to build and train a language model end-to-end using PyTorch, from raw text through model alignment.
The project addresses the challenge of understanding how large language models work by implementing a transformer architecture from scratch based on the Attention is All You Need paper. Rather than relying on high-level libraries, all components are written directly in PyTorch, including the transformer blocks, attention mechanisms, and post-training algorithms. The approach walks through the complete pipeline: data preparation, model construction from individual pieces like multi-head attention and MLPs, pretraining on raw text, text generation, and post-training alignment using techniques like supervised fine-tuning, reward modeling, DPO, ORPO, KTO, PPO, and GRPO.
This tool suits students learning how transformers function, developers who want to train small language models on a single GPU, and researchers interested in post-training algorithms. The project explicitly targets different audiences: students get step-by-step explanations with expected outputs, developers get ready-to-run commands and file paths, and researchers can focus on the post-training implementations. The codebase demonstrates training models ranging from million to billion parameter scales on consumer hardware. The documentation site and Streamlit control panel provide interfaces for interacting with trained models.
The project maintains a complete, self-contained implementation where every algorithm is hand-written rather than delegated to external libraries like transformers, trl, or peft. The repository includes a structured progression through seven steps covering the entire training journey, with code organized to support both learning and practical application. Documentation is available both in the README and on a dedicated documentation site, supporting multiple entry points for different user types.
