sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models, designed to deliver low-latency and high-throughput inference across single GPUs and large distributed clusters. The framework is written in Python and maintained as an open-source project under the LMSYS organization.

The framework's core runtime features include RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding including the next-generation DFlash and Spec V2 technologies, continuous batching, paged attention, tensor and pipeline parallelism, expert parallelism, data parallelism, structured outputs, chunked prefill, and quantization support across FP4, FP8, INT4, AWQ, and GPTQ formats. SGLang also supports multi-LoRA batching for efficient serving of multiple fine-tuned variants. The framework extends beyond traditional language model inference to support diffusion models for video and image generation through SGLang Diffusion.

Model support is extensive, covering language models including Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, and Mistral, alongside embedding models like e5-mistral, gte, and mcdse, as well as reward models such as Skywork. The framework maintains compatibility with most Hugging Face models and OpenAI APIs. Hardware support spans NVIDIA GPUs including GB200, B300, H100, and A100, AMD GPUs including MI355 and MI300, Intel Xeon CPUs, Google TPUs, and Ascend NPUs, reflecting broad deployment flexibility.

Recent development highlights demonstrate rapid iteration and industry responsiveness. The repository achieved day-zero support for DeepSeek-V4 with verified reinforcement learning integration, DeepSeek-V3.2 with sparse attention optimization, and latest models including Nemotron 3 Ultra, Nemotron 3 Super, and Higgs Audio v3 TTS. Performance achievements include 25x inference acceleration on NVIDIA GB300 NVL72, 3.8x prefill and 4.8x decode throughput improvements on GB200 with prefill-decode disaggregation and large-scale expert parallelism, and native TPU support through the SGLang-Jax backend introduced in October 2025.

SGLang has established itself as a production-grade system powering over 400,000 GPUs worldwide and generating trillions of tokens daily. The framework serves as a rollout backend for training frontier models and is integrated with post-training frameworks including AReaL, Miles, slime, Tunix, and verl. Adoption spans major technology organizations including xAI, AMD, NVIDIA, Intel, LinkedIn, Cursor, Oracle Cloud, Google Cloud, Microsoft Azure, AWS, and leading academic institutions including MIT, Stanford, UC Berkeley, and Tsinghua University.