PrismML-Eng/Bonsai-demo

Bonsai Demo is a demonstration repository for running quantized language models locally across multiple platforms and hardware accelerators.

The project addresses the challenge of running capable language models on consumer hardware by providing ready-to-use implementations of Bonsai models, which use extreme quantization techniques to reduce model size while maintaining quality. Bonsai models employ 1-bit and ternary (roughly 1.7-bit) quantization schemes that pack weights into minimal storage, allowing models to run on devices from iPhones to standard laptops. The demo provides shell scripts to download models and launch a local inference server with a web interface, supporting Mac via Metal acceleration, Linux and Windows via CUDA, Vulkan, or ROCm, and CPU-only setups.

Developers should choose this tool if they need to run language models entirely locally without cloud dependencies and have limited hardware resources. The repository suits projects requiring vision capabilities, tool calling, or extended context windows, as the newest Bonsai 27B family adds vision-language support, native OpenAI-style tool calling, and reasoning models alongside the smaller quantized variants. The tool integrates with llama.cpp, with both 1-bit and ternary quantization formats landing in mainline llama.cpp, meaning models can eventually be used with standard llama.cpp tooling rather than remaining tied to this demo repository.

The project maintains active development with regular updates to support new model sizes and quantization formats. The repository includes detailed documentation for specific use cases, such as guides for AI coding agents and vision capabilities. Community benchmarking infrastructure is present, allowing users to contribute performance results across different hardware configurations. The codebase provides hardware-specific configuration options and defaults, indicating attention to making the tool accessible across diverse setups.