ZML is a production inference stack purpose-built to decouple AI workloads from proprietary hardware.
The problem ZML solves is the tight coupling between AI models and specific hardware platforms, which forces developers to rewrite code when deploying to different accelerators. The tool addresses this by compiling a single codebase directly to multiple hardware targets including NVIDIA, AMD, Intel, TPU, Trainium, and CPU, achieving peak performance on each without requiring model rewrites. It accomplishes this through a foundation built on Zig, MLIR, and Bazel, which enable hardware-agnostic compilation and optimization.
ZML suits teams deploying inference workloads across heterogeneous hardware environments or those wanting to avoid vendor lock-in. It is particularly valuable for organizations running large language models or other computationally intensive inference tasks where hardware flexibility matters. The tool includes examples for LLM inference supporting Llama, Qwen, and LFM models, as well as smaller examples like MNIST for getting started. It supports loading models from multiple sources including local directories, Hugging Face repositories, S3, and HTTPS endpoints through a unified virtual filesystem layer.
The project maintains active development with regular example additions and documentation updates covering tutorials, concepts, and deployment guidance. The codebase includes comprehensive examples demonstrating sharding, benchmarking, and model loading across different platforms, suggesting ongoing refinement of core capabilities. Build infrastructure relies on Bazel with platform-specific compilation flags, indicating a mature approach to managing multi-target compilation complexity.
