Shimmy is a single-binary inference server that provides OpenAI-compatible API access to GGUF models running on local GPUs.
Shimmy solves the problem of running large language models locally without requiring Python runtimes or C++ toolchains. It works by wrapping Airframe, a pure-Rust WebGPU transformer engine that compiles compute shaders for any GPU—NVIDIA, AMD, Intel, integrated, or Apple Silicon. Model specifications are automatically derived from GGUF metadata, eliminating the need for hardcoded per-model configuration. The tool supports extended context windows through YaRN RoPE scaling and guarantees deterministic output: the same model, seed, and parameters always produce identical results.
Shimmy suits developers who want to run language models locally without external dependencies or complex setup. It works well for projects that need OpenAI-compatible endpoints but want to avoid Python environments or C++ compilation. The tool is particularly valuable for those working with GGUF-quantized models and needing GPU acceleration across diverse hardware. The project certifies 26 model and quantization combinations across 12 model families through a three-part regimen covering mathematics, inference, and determinism.
The maintainers respond to new issues and pull requests within hours. Work in the issue tracker centers on bug reports, enhancement requests, and user questions.
