nanochat is an experimental training harness for large language models designed to run on a single GPU node with minimal, hackable code.
The tool addresses the challenge of training capable language models affordably by automating hyperparameter selection across the full LLM pipeline: tokenization, pretraining, finetuning, evaluation, and inference. Rather than requiring manual tuning of transformer width, number of heads, learning rates, and other settings, nanochat calculates optimal hyperparameters automatically based on a single control parameter—the depth, or number of transformer layers. This approach lets developers train models of varying capability levels by adjusting one dial. The tool comes preconfigured to train compute-optimal models and includes a simple CLI for interacting with trained models.
The project suits developers and researchers who want to experiment with LLM training without large budgets or complex infrastructure. It is particularly valuable for those interested in understanding the full training pipeline or benchmarking training efficiency. The tool enables training a GPT-2 capability model in approximately two hours on an eight-GPU H100 node, reducing what historically cost tens of thousands of dollars to under one hundred dollars. The project maintains a leaderboard tracking wall-clock time to GPT-2 grade capability, reflecting ongoing optimization efforts in the pretraining stage.
Almost all open issues are raised by users rather than the core team, indicating a substantial base of real-world adopters. Maintainers typically respond to new issues and pull requests within a day. Work in the issue tracker centers on feature requests, potential bugs, and confirmed bugs.
