Cactus is an inference engine for running AI models on mobile devices, wearables, and edge devices.
The tool addresses the challenge of deploying large language models and vision models to resource-constrained hardware by combining quantization techniques with optimized kernels and a runtime designed specifically for edge deployment. It supports multiple model families including Gemma, Liquid, Qwen, Whisper, and Parakeet, with the ability to convert models from HuggingFace format. The engine handles large language model inference, vision-language model processing, and speech transcription on devices ranging from iPhones to iPads and Mac computers.
Developers should choose this tool if they need to run AI inference directly on mobile or edge hardware without cloud dependencies. It suits projects requiring on-device language models, vision processing, or speech recognition where latency and privacy are priorities. The tool provides a command-line interface for benchmarking, downloading pre-converted models, and running inference, making it accessible for rapid prototyping and deployment. The README demonstrates performance across multiple device tiers, from high-end Mac systems down to iPhone 15 Pro, showing how inference speed and memory usage scale with hardware capabilities.
The project maintains active development with regular updates to model support and quantization strategies. The codebase is written primarily in C++ to optimize performance on constrained devices. Documentation includes detailed quantization analysis showing accuracy tradeoffs across different bit widths, helping developers choose appropriate compression levels for their use cases. The tool includes a specialized 26-parameter model called Needle designed specifically for on-device tool calling, indicating ongoing refinement of capabilities beyond general inference.
