vLLM Kunlun is a hardware plugin that enables vLLM to run on Kunlun XPU accelerators.
The plugin solves the problem of running large language models on Kunlun hardware by implementing a pluggable backend interface that decouples XPU integration from the core vLLM codebase. It works by providing hardware-specific kernels and optimizations for Kunlun XPU while maintaining compatibility with vLLM's standard inference engine. The approach allows popular model families—including Transformer-based, Mixture-of-Expert, embedding, and multimodal models—to execute on Kunlun hardware without requiring modifications to vLLM itself.
Adoption makes sense for teams with Kunlun XPU infrastructure who want to run vLLM workloads on that hardware. The tool suits projects requiring inference of diverse model architectures on Kunlun accelerators, from dense models to MoE variants. The plugin includes quantization support for compressed-tensors W4A16, AWQ, GPTQ, and W8A8 formats, along with speculative decoding capabilities. Multi-LoRA inference achieves performance within eighty percent of non-LoRA baselines. Recent additions include support for models like Qwen3.5, Gemma4, and DeepSeek-V3.2 with multi-token prediction.
Development shows active maintenance with recent infrastructure improvements including end-to-end testing, unit test CI pipelines, and code formatting checks. The project maintains a community website alongside the main repository and provides documentation covering installation, quick start, and performance benchmarks. Performance optimization work continues across multiple areas including fused MoE operations for small batches and attention metadata building. The team has expanded model support substantially, adding multimodal models and specialized quantization methods while refactoring the CI workflow for modularity.
