LiteRT-LM is Google's production-ready orchestration layer for running Large Language Models on edge devices, built as a high-performance, cross-platform inference framework written primarily in C++. The project serves as the foundation for on-device GenAI experiences across Google's products, including Chrome, Chromebook Plus, and Pixel Watch, demonstrating its maturity and real-world deployment at scale.
The framework provides comprehensive cross-platform support spanning Android, iOS, Web, Desktop, and IoT devices like Raspberry Pi. It incorporates hardware acceleration capabilities through GPU and NPU accelerators to achieve peak performance on diverse hardware. The project supports multiple programming language APIs at varying maturity levels: Python and Kotlin are marked as stable, C++ is stable for high-performance native applications, while Swift, JavaScript for web, and Flutter represent early preview or community-supported options. This multi-language approach enables developers to integrate LiteRT-LM across different application contexts and platforms.
LiteRT-LM's model support is broad, encompassing Gemma, Llama, Phi-4, Qwen, and other architectures. The most recent version 0.13 added support for Gemma4 12B models and introduced an OpenAI API compatible server through the LiteRT-LM CLI, expanding interoperability options. The framework includes multi-modality support for vision and audio inputs, enabling more sophisticated on-device AI applications. Tool use and function calling capabilities are built in, supporting agentic workflows that allow models to interact with external functions and services.
The release history demonstrates steady feature expansion and platform coverage improvements. Version 0.12 introduced early preview Swift and Web JavaScript APIs with community Flutter support. Version 0.11 added Single Position Multi-token Prediction for Gemma 4 and expanded Windows support. Version 0.10 marked the introduction of the LiteRT-LM CLI tool. Earlier versions progressively added NPU acceleration, multi-modality support, desktop GPU support, and function calling capabilities. The project provides a CLI tool that enables users to run models immediately without writing code, using tools like uv for quick experimentation. Documentation is extensive, with guides covering technical overviews, performance benchmarks, model support, language-specific implementations, and build-from-source instructions, reflecting the project's focus on developer accessibility and production readiness.
