vllm-project/vllm

The core purpose of vLLM is to provide fast and easy-to-use LLM inference and serving capabilities. Performance optimization is central to the project's design, achieved through several key technical innovations. PagedAttention enables efficient management of attention key and value memory, while continuous batching of incoming requests and chunked prefill strategies maximize throughput. The engine supports fast and flexible model execution via piecewise and full CUDA and HIP graphs. vLLM implements extensive quantization support including FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, and compressed-tensors formats. Optimized attention kernels such as FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton implementations further enhance performance, alongside optimized GEMM and Mixture-of-Expert kernels using CUTLASS and CuTeDSL. The system also incorporates speculative decoding techniques including n-gram, suffix, EAGLE, and DFlash approaches, plus automatic kernel generation through torch.compile.

vLLM supports over 200 model architectures from Hugging Face, spanning decoder-only LLMs like Llama and Qwen, Mixture-of-Expert models such as Mixtral and DeepSeek-V3, hybrid attention and state-space models like Mamba, multi-modal models including LLaVA and Qwen-VL, and embedding and retrieval models. The platform provides flexible deployment options through tensor, pipeline, data, expert, and context parallelism for distributed inference, alongside support for multiple hardware backends including NVIDIA GPUs, AMD GPUs, x86/ARM/PowerPC CPUs, Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, and Apple Silicon.

The project maintains an OpenAI-compatible API server alongside Anthropic Messages API and gRPC support, enabling seamless integration into existing workflows. Additional features include structured output generation via xgrammar or guidance, tool calling and reasoning parsers, efficient multi-LoRA support for both dense and MoE layers, and streaming output capabilities.