intel/neural-compressor

Intel Neural Compressor is a model compression library that provides quantization, sparsity, and knowledge distillation techniques for PyTorch, TensorFlow, and ONNX Runtime.

The tool addresses the challenge of reducing model size and inference latency by implementing multiple compression strategies including static and dynamic quantization, quantization-aware training, mixed precision, and weight-only quantization. It supports advanced low-bit quantization for large language models and vision-language models through techniques like SmoothQuant and integration with AutoRound, enabling INT8, FP8, MXFP8, INT4, MXFP4, and NVFP4 precision formats. The library handles both post-training compression and training-time optimization approaches.

Developers should choose this tool if they need to compress models for deployment on Intel hardware, including Gaudi AI Accelerators, Core Ultra Processors, Xeon Scalable Processors, and Intel Data Center GPUs, where it receives extensive testing and optimization. It also supports AMD CPUs, ARM CPUs, and NVIDIA GPUs with limited testing. The project suits teams working with large language models or vision-language models that require aggressive quantization while maintaining accuracy, and those already invested in PyTorch, TensorFlow, or ONNX ecosystems.

The project maintains active development with recent additions of experimental FP8 quantization support for Keras and JAX, NVFP4 quantization, MXFP8 and MXFP4 quantization formats, and FP8 dynamic quantization for Intel Gaudi accelerators. The codebase shows ongoing expansion of model coverage, including support for DeepSeek and other contemporary large language models. Development includes both experimental features and production-ready techniques, with documentation and examples covering multiple hardware targets and framework combinations.