AirLLM is a Python library that enables inference of extremely large language models on consumer-grade GPUs with minimal memory requirements. The core innovation is its layer-by-layer streaming approach, which keeps only one model layer in GPU memory at a time rather than loading entire models. This architectural choice allows a 70B parameter model to run on a single 4GB GPU without quantization, distillation, or pruning. The library extends to even larger models, supporting 405B Llama 3.1 on 8GB, 671B DeepSeek-V3 on approximately 12GB, and the 2.8 trillion parameter Kimi K3 on under 4GB of VRAM.

The repository is written primarily in Jupyter Notebook format and covers topics including Chinese LLMs, Chinese NLP, fine-tuning, generative AI, instruction-following models, LLAMA architectures, and techniques like LoRA and QLoRA. The project has demonstrated significant growth and adoption, with a substantial GitHub star history showing increasing community interest over time.

AirLLM provides an AutoModel interface that automatically detects model types, eliminating the need to manually specify model classes during initialization. The library supports virtually every popular open-source LLM including Llama 2, 3, 3.1, 3.3, and 4 families, Qwen variants including MoE and FP8 models, DeepSeek V2 and V3, Mistral and Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi models. New models are typically supported the day they are released.

The library includes model compression capabilities based on block-wise quantization that can achieve up to 3x inference speed improvements with minimal accuracy loss. Users can enable 4-bit or 8-bit compression during model initialization. Unlike traditional quantization approaches that quantize both weights and activations, AirLLM's compression strategy focuses only on quantizing weights, since the primary bottleneck is disk loading rather than computation. This approach maintains accuracy more effectively across diverse input types.

Configuration options include compression settings, profiling mode for timing analysis, custom paths for saving split model layers, Hugging Face token support for gated models, prefetching to overlap model loading with computation, and an option to delete original models after transformation to save disk space. The library supports inference on macOS with Apple Silicon, Linux, and includes CPU inference capabilities.

The project provides example Jupyter notebooks demonstrating usage across different model families and includes a form for requesting support for additional models. The codebase acknowledges significant contributions from SimJeg's work in the Kaggle LLM competition. The FAQ section addresses common issues including disk space requirements during model splitting, model class mismatches, gated model access, and tokenizer padding configuration.