TurboQuant+ is a KV cache compression tool for large language models that reduces memory footprint during inference.
The tool addresses the memory bottleneck in transformer inference by compressing the key-value cache using PolarQuant quantization combined with Walsh-Hadamard rotation. This approach achieves 3.8-6.4x compression while maintaining near-original prefill speed and approximately 0.9x decode throughput at long context lengths. The compression works across model scales from 1.5B to 104B parameters and has been validated at 128K context windows, even running on resource-constrained hardware like MacBooks.
Developers working on inference optimization for large language models should consider this tool if memory efficiency is a priority and they can tolerate modest decode throughput reduction. The project is particularly suited for long-context inference scenarios where KV cache memory dominates total memory usage. The tool has proven stable enough that its core techniques have been adopted upstream: vLLM has integrated TurboQuant KV cache compression with fused Triton kernels, and llama.cpp has merged the Hadamard rotation approach with optimized implementations for CPU, CUDA, and Vulkan. For production inference, users can run TurboQuant through these established engines rather than directly through this repository.
The project serves as the research home for the TurboQuant approach, housing the Python reference implementation, validation papers, and benchmark data. Development activity shows a focus on upstreaming stable, well-tested components as small, reviewable patches to major inference frameworks rather than maintaining a monolithic implementation. The README documents getting started guidance, configuration recommendations, and benchmark results, indicating sustained attention to practical usability. The project's approach of contributing validated findings incrementally to upstream projects suggests a collaborative development model oriented toward broad adoption of the compression techniques.
