edge0 is an open-source streaming mixture-of-experts inference framework designed to run large sparse-MoE models on consumer hardware by bounding peak memory to the active expert set rather than total parameter count.
The framework addresses the challenge of deploying large MoE models on resource-constrained devices through three core mechanisms. SSD expert offload streams expert weights from storage on demand, keeping memory usage proportional only to the active experts being used. Prerouter is a trained routing prediction head that forecasts expert assignments one step ahead, allowing expert loads to overlap with the forward pass rather than stalling it, yielding throughput gains up to 59 percent that increase with storage latency and model size. Recover-LoRA uses int4 quantization with frozen base weights and trained LoRA adapters recovered through distillation from a full-precision teacher, preserving quality at 4-bit precision while allowing a single base model to serve multiple adapter sets.
The tool suits developers targeting edge deployment across multiple platforms who need to run 35-billion or 8-billion parameter MoE models on consumer devices. The framework provides platform-specific implementations: Python with MLX for macOS and Apple Silicon, native Rust for macOS applications, Swift with MLX Swift for iOS, Kotlin with a native engine for Android, and C++ with Vulkan for Windows. Two model tiers ship as complete end-to-end releases, each including the checkpoint, trained LoRA adapters, and prerouter heads designed to work together.
The project maintains active development across multiple platform implementations, with Python and macOS runtimes available and iOS, Android, and Windows engines open-sourced. A technical report documenting the approach has been published on arXiv. The framework is positioned as extensible, allowing the production-proven recipe to generalize across different hardware targets and use cases.
