The exo repository is a Python-based project that enables users to run frontier AI models locally by connecting multiple devices into a distributed AI cluster.
The core functionality of exo centers on distributed inference across heterogeneous devices. The system automatically discovers devices on the network without requiring manual configuration, eliminating setup friction for users wanting to pool computational resources. A key differentiator is day-0 support for RDMA over Thunderbolt 5, which the project claims enables a 99 percent reduction in latency between devices. This capability allows models to run faster as additional devices are added to the cluster, addressing a fundamental challenge in distributed AI inference where communication overhead typically limits scaling benefits.
The project implements topology-aware automatic parallelization, meaning exo analyzes the real-time device topology including available resources and network characteristics between devices to determine optimal model distribution strategies. The system supports tensor parallelism with reported speedups of up to 1.8x on two devices and 3.2x on four devices. The inference backend uses MLX and MLX distributed for distributed communication, providing a foundation for efficient computation across the cluster.
API compatibility is a significant design consideration. exo supports multiple API standards including OpenAI Chat Completions API, Claude Messages API, OpenAI Responses API, and Ollama API, allowing users to leverage existing tools and clients without modification. The project also supports loading custom models from the HuggingFace hub, expanding the range of available models beyond built-in options.
A built-in dashboard provides cluster management and direct model interaction capabilities, accessible at localhost:52415. The project includes benchmark data demonstrating performance on multi-device setups, including configurations running models like Qwen3-235B and DeepSeek v3.1 671B on clusters of M3 Ultra Mac Studios.
Installation options span macOS and Linux platforms, with a native macOS app available for background operation. The project follows the XDG Base Directory Specification on Linux for configuration and data storage. Current limitations include CPU-only execution on Linux with GPU support under development, while macOS benefits from Metal acceleration on Apple Silicon. The system supports coordinator-only nodes for machines with good network connectivity but limited computational resources, and includes legacy daemon mode for compatibility with older init systems.
