The llama2.c repository provides a minimal, educational implementation for running Llama 2 language model inference in pure C. The core functionality is contained in a single 700-line C file called run.c that requires no external dependencies. The project enables users to train small Llama 2 models in PyTorch and then deploy them for inference using only C, making it accessible for educational purposes and resource-constrained environments.
The repository demonstrates that surprisingly capable language models can be built at small scales when the problem domain is sufficiently narrow. It includes pre-trained models ranging from 15 million to 110 million parameters trained on the TinyStories dataset, which can generate coherent short stories at interactive speeds. On an M1 MacBook Air, the 15M parameter model achieves approximately 110 tokens per second, while the 42M model maintains interactive performance with more diverse and coherent outputs. The project also supports loading and running Meta's official Llama 2 models, though current float32 inference limits practical use to models around 7 billion parameters due to memory and speed constraints.
The implementation includes several sampling strategies for text generation, supporting temperature-based sampling and top-p nucleus sampling. Users can control output diversity through command-line arguments and provide custom prompts to guide generation. The repository provides conversion tools via an export.py script that transforms Meta's Llama 2 checkpoints into the llama2.c format, though this conversion process takes approximately ten minutes and produces large 26GB files for the 7B model in float32 precision.
Beyond basic inference, the repository includes int8 quantization support through the Q8_0 quantization scheme, which reduces checkpoint file sizes and accelerates computation by using integer arithmetic for matrix multiplications while keeping sensitive parameters like RMSNorm scales in float32. This quantization approach represents standard practice in LLM inference optimization and provides meaningful speedups without sacrificing model accuracy significantly.
The project originated as a weekend project building on the author's earlier nanoGPT work, adapted to implement the Llama 2 architecture instead of GPT-2. It was inspired by the llama.cpp project but deliberately prioritizes simplicity and educational value by hard-coding the Llama 2 architecture and avoiding external dependencies.
The repository shares contributors with several major projects including Microsoft's VSCode and TypeScript implementations as well as the Rust language project, indicating crossover interest from developers working on large-scale systems.
The codebase supports additional features including chat mode for interactive conversations with Llama Chat models and experimental support for Meta's Code Llama models, though Code Llama support remains incomplete due to hyperparameter differences in the RoPE layer. Compilation options and OpenMP support enable performance tuning for different hardware configurations, with reported speeds ranging from 4 tokens per second on multi-threaded CPU systems to 30 seconds per token on single-threaded MacBook Air execution.
