jundot/omlx

oMLX is an LLM inference server optimized for Apple Silicon Macs that runs from the macOS menu bar.

The tool addresses the friction of running large language models locally on Apple Silicon by implementing continuous batching and tiered key-value caching across both in-memory and SSD storage. This two-tier caching strategy allows the server to keep frequently used models pinned in memory while automatically swapping larger models to disk on demand. Crucially, KV cache persists across context changes within conversations, meaning past context remains cached and reusable even when requests change, making it practical for extended coding sessions with tools that maintain state across multiple interactions.

oMLX suits developers who want to run local LLMs on their Mac without sacrificing either convenience or control. The menu bar interface provides straightforward management without requiring terminal expertise, while the CLI shim enables integration with terminal commands and Apple Shortcuts for automation. The project targets users who need context limits, model pinning, and intelligent memory management—particularly those working with coding assistants that benefit from persistent context across requests. The macOS app distribution with built-in auto-update lowers the barrier to staying current.

Development activity shows consistent engagement with the project. The maintainer actively responds to user feedback and maintains the codebase. The tool receives regular updates that address both performance improvements and user-reported issues. Documentation is maintained across multiple languages, indicating attention to accessibility for a broader user base. The project includes benchmarking infrastructure to track performance characteristics, suggesting a data-driven approach to optimization decisions.