bitsandbytes is a PyTorch library that enables k-bit quantization for large language models.
The library addresses the memory constraints of training and deploying large language models by providing three quantization approaches. 8-bit optimizers use block-wise quantization to maintain 32-bit optimizer performance while drastically reducing memory overhead. LLM.int8() performs 8-bit quantization through vector-wise techniques, quantizing most weights to 8-bits while handling outliers separately with 16-bit operations, cutting inference memory requirements in half without performance loss. QLoRA extends this with 4-bit quantization combined with low-rank adaptation weights, enabling full model training on consumer hardware by quantizing model weights to 4-bits while keeping a small set of trainable LoRA parameters at higher precision. The library exposes these capabilities through dedicated modules: Linear8bitLt and Linear4bit layers for quantized operations, and an optim module for quantized optimizers.
Developers should adopt this tool if they need to run or fine-tune large language models on hardware with limited memory. It suits projects where reducing memory footprint is critical without sacrificing model quality—particularly fine-tuning scenarios where QLoRA's approach of training only adapter weights proves effective. The tool requires Python 3.10 or later and PyTorch 2.4 or later, with support varying across different accelerator types.
The project maintains nightly unit tests to catch regressions. Development targets broad hardware compatibility, with the current branch reflecting planned and partially supported accelerator backends beyond those in the latest stable release.
