GuppyLM is a small language model that demonstrates how to train a working LLM from scratch without specialized infrastructure or expertise.
The project addresses the perception that language model development is inaccessible and mysterious. It solves this by providing a complete, minimal implementation that covers the entire pipeline: synthetic data generation, tokenizer training, transformer architecture, training loop, and inference. The model itself is a nine million parameter transformer with six layers, 384 hidden dimensions, and a four thousand token vocabulary, trained on sixty thousand synthetic conversations about fish life. The approach prioritizes clarity and simplicity over performance, using vanilla transformer components without modern optimizations, so every piece remains understandable to someone learning how language models work.
Someone considering this project should understand it is educational rather than practical. It suits developers who want to understand how language models function end-to-end, or who want a template for training custom models on domain-specific data. The model is intentionally limited in scope—it speaks only as a fish character about water, food, and tank life—which keeps the training data manageable and the behavior predictable. The project includes browser-based inference, making it accessible without GPU access, and provides Colab notebooks for both training and inference that run in minutes.
The project maintains active documentation through written articles explaining the approach, and provides both the trained model and the training dataset publicly. The codebase is structured to be readable and educational rather than optimized for scale or performance.
