DeepGEMM is a unified, high-performance tensor core kernel library designed specifically for the computational primitives of modern large language models. Written in CUDA, the library consolidates key operations including GEMMs in multiple precision formats (FP8, FP4, BF16), fused Mixture of Experts with overlapped communication (Mega MoE), Multi-Query Attention scoring for the lightning indexer, and HyperConnection into a single cohesive codebase. All kernels compile at runtime through a lightweight Just-In-Time module, eliminating the need for CUDA compilation during installation.
The library draws inspiration from NVIDIA's CUTLASS and CuTe projects but deliberately avoids heavy reliance on their template systems and algebraic abstractions. This design philosophy prioritizes simplicity and accessibility, with a limited number of core kernel functions that make the codebase an effective resource for learning NVIDIA GPU kernel optimization techniques. Despite this lightweight approach, DeepGEMM's performance matches or exceeds expert-tuned libraries across various matrix shapes, achieving up to 1550 TFLOPS on H800 GPUs according to recent benchmarks.
DeepGEMM supports NVIDIA SM90 and SM100 architecture GPUs and requires Python 3.8 or higher, C++20-compatible compilers, CUDA Toolkit 12.3 or higher for SM90 (with 12.9 or higher recommended), CUDA 12.9 or higher for SM100, PyTorch 2.1 or higher, CUTLASS 4.0 or higher, and the fmt library. The library provides multiple GEMM interfaces tailored to different use cases. Normal dense GEMMs support non-grouped operations through functions like fp8_gemm_nt, while grouped GEMMs optimize for MoE scenarios where experts share fixed N and K dimensions but vary in M-axis. The library distinguishes between contiguous layout grouped GEMMs for training and inference prefilling phases, and masked layout grouped GEMMs for inference decoding when CUDA graphs are enabled and token distribution across experts is unknown at CPU time.
Recent development has been active and substantial. In April 2026, the library added Mega MoE, FP8xFP4 GEMM support, FP4 Indexer capabilities, Programmatic Dependent Launch, and faster JIT compilation. In September 2025, scoring kernels for weighted ReLU MQA logits were introduced for DeepSeek v3.2. A major refactor in July 2025 brought support for both SM90 and SM100 architectures with a low-CPU-overhead JIT C++ module. Earlier updates in May 2025 added weight gradient kernels for dense and MoE backward operations, and NVRTC support with up to 10x compilation speedup.
The repository shows active community engagement with a median issue and pull request response latency of 9.8 hours across 19 tracked items. Primary contributors include zheanxu with 13 events, leedrake5 with 3 events, and yewentao256 with 3 events. The project maintains overlapping contributors with major repositories including vllm-project/vllm, sgl-project/sglang, and pytorch/pytorch, indicating integration with significant inference and deep learning frameworks.
DeepGEMM provides extensive utility functions for configuration and optimization, including SM count management, tensor core utilization settings, alignment configuration for grouped operations, and scaling factor transformation utilities. The library supports both paged and non-paged MQA kernel variants for different inference phases and includes Mega MoE functionality that fuses expert dispatch, linear layers with SwiGLU activation, and expert combine operations while overlapping NVLink communication with tensor core computation.
