Needle is a 14MB foundation model for on-device inference on phones, wearables, smart home devices, and robots.
The tool solves the problem of running capable language models on severely resource-constrained devices. Needle 2 is a 45-million-parameter model compressed to 14MB that performs tool calling, device control, and structured data extraction while using only about 28MB of RAM during inference. It uses a Simple Attention Network architecture with a Hadamard MLP, grouped query attention, and engram key-value memory, then applies CQ2-bit quantization. The model constrains token generation with byte-level grammars compiled from tool schemas, ensuring outputs are valid structured data. A learned confidence head provides calibrated confidence scores for each response, and a built-in retrieval mechanism selects the five most relevant tools from a large catalogue per turn. A 256-token sliding window with pinned tool key-value sinks keeps memory bounded regardless of conversation length.
Needle suits developers building applications for mobile and edge devices where model size and memory footprint are hard constraints. It is particularly valuable for scenarios requiring tool calling and structured extraction on devices that cannot run larger models. The project trades performance wins with other small models like FunctionGemma 270M, LFM2.5 230M, and Apple FM while being five to seventy times smaller and using two-bit quantization against their float16. The Python package provides inference, LoRA fine-tuning, and export capabilities; the inference engine is fetched once from Hugging Face and cached with no additional build steps required.
The maintainers respond to issues and pull requests within a day. The project is growing steadily.
