XTuner is a training engine designed for ultra-large mixture-of-experts models.
The tool addresses the challenge of efficiently training massive MoE models at scale. XTuner V1 departs from traditional 3D parallel training architectures by optimizing specifically for mainstream MoE training scenarios. Its core approach involves dropless training that scales 200-billion-parameter MoE models without expert parallelism, requiring only intra-node expert parallelism for 600-billion-parameter models. The engine incorporates memory-efficient techniques to support long sequence training up to 64k tokens on 200-billion-parameter models without sequence parallelism, while maintaining stability under expert load imbalance. It achieves this through a smaller expert parallelism dimension compared to conventional approaches and integrates optional DeepSpeed Ulysses sequence parallelism for flexible scaling.
Developers training ultra-large MoE models should consider this tool if they need to maximize throughput and memory efficiency beyond what traditional 3D parallel schemes provide. The project suits pre-training, instruction fine-tuning, and reinforcement learning workflows for models scaling to one trillion parameters. It demonstrates particular strength on Ascend A3 Supernodes, where it reportedly exceeds NVIDIA H800 training efficiency, making it relevant for teams with access to that hardware. The tool is positioned as a training backend intended to integrate with the broader open-source ecosystem rather than as a standalone framework.
Development activity shows sustained focus on MoE-specific optimization with a stated roadmap emphasizing continuous efficiency improvements for pre-training, instruction fine-tuning, and reinforcement learning. The project maintains active engagement channels across multiple platforms and hosts trained models on several model repositories, indicating ongoing community involvement and practical deployment of the training engine.
