The rasbt/llms-from-scratch repository is the official code repository for the book "Build a Large Language Model (From Scratch)" and provides a comprehensive implementation of a ChatGPT-like language model in PyTorch developed entirely from scratch. The repository is written primarily in Jupyter Notebooks and serves as both an educational resource and a practical guide for understanding how large language models work at a fundamental level.
The repository is structured around seven main chapters plus appendices, each building progressively on previous concepts. Chapter 2 covers working with text data and includes notebooks on data loading. Chapter 3 focuses on coding attention mechanisms, including multihead attention implementations. Chapter 4 guides users through implementing a complete GPT model from scratch with both notebook and Python script formats. Chapter 5 addresses pretraining on unlabeled data with training and generation scripts. Chapter 6 demonstrates finetuning for text classification tasks. Chapter 7 covers instruction-tuning to make models follow specific directives. Appendix A provides an introduction to PyTorch fundamentals, while Appendix E covers parameter-efficient finetuning using LoRA techniques. The repository also includes exercise solutions throughout and references to further reading materials.
The codebase emphasizes accessibility and practical learning. The implementation is designed to run on conventional laptops without requiring specialized hardware, though it automatically utilizes GPUs when available. The repository explicitly avoids external LLM libraries, instead building everything from PyTorch primitives to ensure learners understand each component. The code mirrors the approach used in creating large-scale foundational models like ChatGPT, making it suitable for both educational purposes and understanding production-level techniques.
According to GitGenius activity tracking, the repository maintains active engagement with a median issue and pull request response latency of 0.0 hours and a mean latency of 64.1 hours across 208 tracked items. The primary maintainer rasbt has logged 670 events, with significant contributions from d-kleine (135 events) and labdmitriy (44 events). Bug reports represent the most active issue category with 76 tracked items, followed by questions (44 items) and enhancement requests (9 items). The repository shares overlapping contributors with major projects including pytorch/pytorch, tensorflow/tensorflow, and huggingface/transformers, indicating its integration within the broader deep learning ecosystem.
The repository covers essential LLM concepts including tokenization, attention mechanisms, transformer architectures, pretraining strategies, and finetuning approaches for both classification and instruction-following tasks. It includes supplementary materials such as a 17-hour video course that mirrors the book structure and a companion sequel titled "Build A Reasoning Model (From Scratch)" that extends the concepts to reasoning and reinforcement learning. The repository also provides a free 170-page PDF with approximately 30 quiz questions per chapter to help users test their understanding of the material.
