KTransformers is a research framework developed by MADSys Lab at Tsinghua University in collaboration with Approaching.AI and the community, focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project exposes two primary user-facing capabilities: a high-performance inference system called kt-kernel and a fine-tuning integration with LLaMA-Factory for supervised fine-tuning tasks.
The inference component, kt-kernel, provides CPU-optimized kernel operations designed for heterogeneous LLM inference. It features Intel AMX and AVX512/AVX2 optimized kernels for INT4 and INT8 quantized inference, efficient Mixture-of-Experts optimization with NUMA-aware memory management, and support for both CPU-side INT4/INT8 quantized weights and GPU-side GPTQ quantization. The system offers a clean Python API for integration with frameworks like SGLang and enables CPU-GPU hybrid inference for large MoE models, with heterogeneous expert placement allowing hot experts to run on GPU while cold experts run on CPU. Performance benchmarks show the system achieving 227.85 tokens per second total throughput and 87.58 tokens per second output throughput on DeepSeek-R1-0528 with FP8 precision using 8 L20 GPUs and an Xeon Gold 6454S processor under 8-way concurrency.
The fine-tuning component integrates KTransformers with LLaMA-Factory to enable ultra-large MoE model fine-tuning on limited GPU memory. It supports multi-backend fine-tuning with CPU and GPU hybrid execution, INT8 and INT4 quantization, and demonstrates 6 to 12 times faster training speeds compared to ZeRO-Offload in benchmarked MoE SFT workloads while using approximately half the CPU memory of previous KTransformers fine-tuning approaches. Training speed examples include 3.7 iterations per second for DeepSeek-V3 and DeepSeek-R1 on 4 RTX 4090 GPUs and 8 plus iterations per second for Qwen3-30B-A3B on a single RTX 4090.
The repository maintains active development with extensive model support including recent additions like MiniMax-M3, GLM-5.2, DeepSeek-V4-Flash, and Kimi-K2.5.
