esp32-ai is a framework for running a 28.9 million parameter language model on an ESP32-S3 microcontroller.
The project solves the problem of fitting a large language model onto a device with only 512KB of SRAM by using Per-Layer Embeddings, an approach from Google's Gemma 3n. Most of the model's parameters live in a lookup table stored in flash memory rather than in fast RAM. During inference, only the few rows needed for each token are pulled from flash, keeping memory usage low enough to run on the microcontroller itself. The model generates text at 9.88 tokens per second end-to-end with no network connectivity, displaying output on a small screen wired directly to the chip.
This tool suits developers interested in on-device inference for embedded systems where network access is unavailable or undesirable. The project is designed for story generation and simple question answering rather than instruction-following or factual reasoning. Adopters should understand that the model's capabilities are limited by its small reasoning component; the memory optimization technique does not expand what the model can learn or do, only where it can run. The architecture itself is the primary point of interest rather than the model's conversational abilities.
Development activity shows careful attention to deployment safety and reproducibility. The fetch_model.sh script validates downloads against pinned SHA-256 hashes and byte sizes before installation, leaving existing artifacts untouched if any check fails. The deploy.sh tool operates only on already-downloaded models and manages its own dependencies through pinned wheel versions rather than fetching them during deployment. The project provides two trained models available through external model repositories.
