gemma.cpp is a lightweight, standalone C++ inference engine for Google's Gemma models.
The project addresses the gap between deployment-oriented C++ inference runtimes that lack experimentation features and Python-centric ML research frameworks that abstract away low-level computation. It provides a minimalist implementation of Gemma-2, Gemma-3, and PaliGemma-2 models with a small core of approximately two thousand lines of code, designed to be straightforward to embed in other projects with minimal dependencies and easily modifiable. The engine uses the Google Highway Library to leverage portable SIMD for CPU inference and includes mixed-precision GEMM optimizations with weight compression, sampling with TopK and temperature, and backward pass support with an Adam optimizer for research purposes.
The tool targets experimentation and research use cases rather than production deployments. It suits developers who want to understand and modify model inference at a low level without the abstraction overhead of larger frameworks, and those seeking to co-design high-level algorithms with low-level computation. The README explicitly recommends standard deployment pathways using Python frameworks for production-oriented edge deployments, positioning this project as complementary to rather than competitive with those tools. The implementation supports tensor parallelism, multiple weight compression formats including custom fp8, and runs on any CPU across Linux, Windows, and macOS.
The project maintains a substantial base of adopters reporting real-world use, as evidenced by most open issues being raised by outside users rather than the core team. Maintainers typically respond to new issues and pull requests within a day. Work in the issue tracker is dominated by support requests, feature requests, and issues awaiting user response, reflecting active engagement with a community of users.
