DeepEP is a high-performance communication library for expert parallelism in machine learning training and inference.
The library addresses the challenge of efficient all-to-all communication in mixture-of-experts models by providing GPU kernels for MoE dispatch and combine operations with low-precision support including FP8. It achieves this through runtime JIT compilation of kernels, eliminating the need for CUDA compilation during installation. The approach focuses on minimizing streaming multiprocessor (SM) occupation while maintaining throughput and latency performance that matches or exceeds hardware bandwidth limits.
The tool suits teams training large-scale mixture-of-experts models who need efficient expert parallelism across multiple GPUs. The recent V2 refactoring significantly reduces SM resource requirements compared to V1 while supporting larger scale-up and scale-out domains up to EP2048. The library switched from NVSHMEM to an NCCL Gin backend, allowing reuse of existing NCCL communicators and offering a header-only, lightweight design. It provides experimental primitives for pipeline parallelism, context parallelism, and remote memory access that operate with zero or minimal SM occupation. The tool is suitable for environments with RDMA-capable networking and modern GPU architectures.
The project maintains active development with a complete V2 refactoring that achieved substantial performance gains and resource efficiency improvements. Ongoing work includes elastic GPU and CPU buffer support for hybrid memory management, intermediate buffer size reduction through EP replay for load imbalance handling, and collective operation implementations for data and tensor parallelism. The library explicitly marks Engram, pipeline parallelism, and context parallelism features as experimental, indicating these remain under active development rather than production-ready.
