LLM Compressor is a Python library designed to optimize large language models for deployment with vLLM through various compression techniques. The repository serves as a Transformers-compatible tool that enables researchers and practitioners to apply quantization algorithms, pruning methods, and other compression strategies to reduce model size and improve inference efficiency while maintaining compatibility with the vLLM inference engine.
The library provides comprehensive support for multiple quantization approaches across different model components. It supports weight and activation quantization in formats including W8A8 (both int8 and fp8), W4AFP8, and microscale formats like NVFP4 and MXFP4. Mixed precision quantization options include W4A16, W8A16, and various microscale variants. Additionally, the library handles attention and KV cache quantization in FP8 and NVFP4 formats. The supported quantization algorithms encompass Simple PTQ, GPTQ, AWQ, SmoothQuant, AutoRound, and rotation-based methods like SpinQuant and QuIP.
Recent developments highlight the library's expanding capabilities. The REAP Expert Pruning Modifier was introduced to reduce VRAM requirements for Mixture-of-Experts models by structurally removing less-relevant experts based on saliency metrics. Support for Transformers v5 was added with improved MoE calibration workflows. The library now supports day-zero quantization for models like DiffusionGemma, Nemotron 3 Ultra, DeepSeek-V4-Flash, Kimi-K2.6, Qwen3.6, and Gemma 4, with pre-quantized checkpoints available on Hugging Face Hub.
The library integrates seamlessly with Hugging Face models and repositories, allowing users to load models directly and apply compression recipes. Compressed models are saved in the compressed-tensors format, ensuring compatibility with vLLM for optimized inference. For handling very large models, the library supports distributed data parallel (DDP) quantization and disk offloading strategies to manage memory constraints during compression.
The library offers extensive documentation and examples covering weight-only quantization, weight and activation quantization, KV cache and attention quantization, architecture-specific approaches for MoE and vision-language models, non-uniform quantization, and big model support through sequential onloading and disk offloading. Users can apply quantization through a straightforward oneshot API and immediately use the resulting checkpoints with vLLM for inference.
