LiteRT is Google's on-device runtime framework designed as the successor to TensorFlow Lite, enabling high-performance machine learning and generative AI deployment on edge platforms. Written primarily in C++, the project focuses on efficient model conversion, runtime execution, and optimization specifically tailored for resource-constrained devices including mobile phones, embedded systems, and IoT hardware.
The framework addresses the complete pipeline from model to on-device deployment, supporting PyTorch, TensorFlow, and JAX models. LiteRT V2 introduces a Compiled Model API that streamlines development through automated accelerator selection without requiring explicit delegate configuration, true asynchronous execution capabilities, simplified NPU distribution, and efficient I/O buffer handling. The runtime features unified NPU acceleration providing seamless access to neural processing units from major chipset providers including Google Tensor, Intel, MediaTek, and Qualcomm through a single consistent API.
Key capabilities include LiteRT-LM for deploying large language models directly on-device, LiteRT.js for secure client-side ML inference in browsers via WebGPU and WebAssembly, a Tensor API offering a lightweight C++ library for high-performance tensor manipulation, and the LiteRT CLI command-line toolkit for streamlining AI coding agent workflows. The framework supports GPU acceleration across multiple APIs including OpenCL, OpenGL, Metal, and WebGPU, with particular emphasis on minimizing latency through new buffer interoperability features designed to support generative AI inference workloads.
Platform support spans Android, iOS, Linux, macOS, Windows, web browsers, and IoT devices. The project maintains nightly wheel builds for Linux, macOS, and Windows alongside continuous integration builds for various architectures including macOS ARM64, Linux x86_64, Windows x86_64, and CMake Android builds.
The project roadmap emphasizes broadening NPU support and improving performance across hardware accelerators, introducing generative AI-specific optimizations, developing better debugging and profiling utilities, and enhancing platform support for emerging ecosystems. LiteRT integrates within Google's broader AI-Edge ecosystem of on-device machine learning tools and maintains official sample applications and code examples covering use cases including automatic speech recognition, image segmentation, and real-time inference tasks.
