pytorch/ao

TorchAO is a PyTorch native quantization library that optimizes models for both training and inference through techniques including float8 quantization, int4 quantization, and sparsity.

The library addresses the challenge of reducing model size and computational cost while maintaining accuracy during both training and serving phases. It provides quantization-aware training to recover accuracy lost during quantization, float8 training to accelerate pre-training workloads, and post-training quantization for inference optimization. The tool integrates directly with PyTorch's native operations, allowing developers to apply these optimizations without leaving the PyTorch ecosystem.

Developers should choose this tool if they are working with large language models or transformer architectures and need to optimize for either training speed or inference efficiency. It suits projects where reducing memory footprint and latency are critical, particularly when fine-tuning or serving models like Llama or Gemma. The library is designed for practitioners who want production-ready quantization without complex external dependencies, as it leverages PyTorch's native capabilities.

The project shows active development with recent integrations into complementary tools and ongoing work on specialized techniques like mixture-of-experts training optimization. Contributors are actively expanding the library's scope to cover emerging model architectures and training scenarios, as evidenced by prototype implementations for new quantization methods. The team maintains comprehensive documentation and makes quantized model checkpoints and training recipes publicly available, indicating a commitment to reproducibility and practical usability.