Mini-SGLang is a lightweight inference framework for large language models that implements a compact version of the full SGLang system. The codebase comprises approximately 5,000 lines of Python and is designed to serve dual purposes: functioning as a capable inference engine while simultaneously providing a transparent reference implementation for researchers and developers seeking to understand modern LLM serving systems.
The framework implements several advanced optimization techniques to achieve high performance. Radix Cache enables reuse of key-value cache for shared prefixes across multiple requests, reducing redundant computation. Chunked Prefill reduces peak memory usage during long-context serving by processing input tokens in manageable chunks. Overlap Scheduling hides CPU scheduling overhead by overlapping it with GPU computation. The system also supports Tensor Parallelism to scale inference across multiple GPUs and integrates optimized kernels including FlashAttention and FlashInfer for maximum computational efficiency.
Platform support is currently limited to Linux systems running on x86_64 and aarch64 architectures. Windows and macOS are not supported due to dependencies on Linux-specific CUDA kernels, though the documentation provides guidance for Windows users to run Mini-SGLang via WSL2 or Docker containers. Installation requires the NVIDIA CUDA Toolkit with version matching the GPU driver, and the project recommends using the uv package manager for fast and reliable setup.
The framework provides multiple interaction modes. Users can launch an OpenAI-compatible API server with a single command for online serving, enabling integration with standard tools like curl or any OpenAI-compatible client. An interactive shell mode allows direct chat with models in the terminal using the --shell flag, with support for clearing chat history via the /reset command.
Benchmark results demonstrate performance across different scenarios. Offline inference testing on a single H200 GPU with Qwen3 models shows throughput metrics across 256 sequences with variable input and output lengths. Online inference benchmarks on a 4xH200 GPU setup connected by NVLink test the Qwen3-32B model using real-world request traces from the Qwen dataset.
According to GitGenius activity tracking, the repository shows a median issue and pull request response latency of 7.8 hours across 22 tracked items, with a mean latency of 110.5 hours. DarkSharpness is the most active contributor with 38 tracked events, followed by jiahe7ay with 4 events and GiggleWang with 3 events. The repository shares overlapping contributors with sgl-project/sglang, python/cpython, and vllm-project/vllm, indicating connections to the broader LLM inference and Python ecosystem communities.
The codebase emphasizes readability and maintainability through clean modular design and full type annotations, making it accessible for modification and study. Documentation includes detailed feature specifications and system architecture explanations to help users understand the design and data flow of the inference system.
