The deepseek-ai/deepseek-v3 repository contains the implementation and documentation for DeepSeek-V3, a large-scale Mixture-of-Experts language model with 671 billion total parameters and 37 billion activated parameters per token. The repository is written in Python and classified across multiple domains including natural language processing, transformer architecture, computer vision, model scaling, and multimodal AI capabilities. The codebase serves as a research development platform for advanced deep learning and neural network implementations.
DeepSeek-V3 introduces several architectural innovations built upon the foundation of DeepSeek-V2. The model pioneers an auxiliary-loss-free strategy for load balancing that minimizes performance degradation while maintaining efficient token distribution across experts. The architecture incorporates Multi-head Latent Attention and DeepSeekMoE designs, and introduces a Multi-Token Prediction training objective that improves model performance and enables speculative decoding for inference acceleration. The model was pre-trained on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages.
The training process demonstrates remarkable efficiency and stability. DeepSeek-V3 required only 2.788 million H800 GPU hours for complete training, with 2.664 million hours dedicated to pre-training and 0.1 million hours for post-training stages. The repository documents an FP8 mixed precision training framework that was validated at scale for the first time on an extremely large model. The training process achieved nearly full computation-communication overlap through co-design of algorithms, frameworks, and hardware, overcoming cross-node MoE training bottlenecks. Notably, the entire training process remained stable without any irrecoverable loss spikes or rollbacks.
The repository provides access to two model variants: DeepSeek-V3-Base and DeepSeek-V3, both with 128K context length, available through Hugging Face. The total model size on Hugging Face is 685 billion parameters, including 671 billion main model weights and 14 billion Multi-Token Prediction module weights. Comprehensive evaluation results show DeepSeek-V3 outperforming other open-source models and achieving performance comparable to leading closed-source models across standard benchmarks including MMLU, BBH, code tasks, and mathematical reasoning.
