Optimum Quanto is a PyTorch quantization backend designed as part of the Hugging Face Optimum ecosystem. The project provides tools for converting full-precision neural network models into quantized versions that consume less memory and can run faster on compatible hardware, particularly CUDA devices. The repository is currently in maintenance mode, accepting only minor bug fixes and documentation improvements rather than major new features.
The core functionality centers on quantizing model weights and activations to lower bitwidths, supporting int2, int4, int8, and float8 weight representations along with int8 and float8 activation quantization. A distinctive feature is that all quantization operations work in eager mode, meaning they function with non-traceable models and can place quantized models on any device including CUDA and MPS. The library automatically inserts quantization and dequantization operations, replaces standard PyTorch modules with quantized equivalents, and provides a workflow progression from float models through dynamic quantization to static quantized versions.
The quantization workflow accommodates both high-level APIs for Hugging Face models and low-level APIs for vanilla PyTorch models. For Hugging Face transformers and diffusers, the library offers helper classes that simplify quantization, saving, and reloading. The low-level workflow involves quantizing a model, optionally calibrating it on representative data to determine activation ranges, performing quantization-aware training if needed, freezing integer weights, and serializing the result. Serialization supports both PyTorch pickle format and safetensors, with compatibility for PyTorch's weight-only quantization standard.
At the technical foundation, Quanto uses a custom Tensor subclass that projects source tensors into optimal ranges for destination types and maps values accordingly. For integer types, this involves rounding operations, while floating-point types use native PyTorch casting. The projection mechanism minimizes saturated and zeroed values through symmetric per-tensor or per-channel approaches for int8 and float8, and group-wise affine quantization for lower bitwidths. The library provides quantized module implementations for Linear, Conv2d, and LayerNorm layers, with weights quantized per-channel along the output dimension while biases remain unquantized to preserve arithmetic accuracy.
Performance characteristics show that models quantized with int8 or float8 weights and float8 activations maintain accuracy close to full-precision versions. When optimized kernels are available, inference latency remains comparable to full-precision models when quantizing only weights. Device memory usage is approximately divided by the ratio of float bits to integer bits. The repository includes accelerated matrix multiplication kernels for CUDA devices supporting int8-int8, fp16-int4, bf16-int8, and bf16-int4 operations.
According to GitGenius activity tracking, the repository shows median issue and pull request response latency of 10.8 hours across 125 tracked items, with dacorvo being the most active contributor at 334 events. The most common issue labels are Stale, question, and enhancement. The project maintains connections with related Hugging Face repositories including diffusers and transformers, as well as PyTorch's main repository, reflecting its integration within the broader machine learning ecosystem. Notable unimplemented features include dynamic activation smoothing, kernels for all mixed matrix multiplications across all devices, and compatibility with PyTorch's torch.compiler.
