DeepSeek Coder is a series of code language models developed by DeepSeek AI, ranging from 1B to 33B parameters, designed to provide state-of-the-art code generation and completion capabilities. The models were trained from scratch on 2 trillion tokens, with a composition of 87 percent code and 13 percent natural language data in both English and Chinese. This training approach enables the models to achieve superior performance on multiple programming benchmarks compared to existing open-source alternatives.
The repository demonstrates strong empirical results across several established coding benchmarks. On HumanEval Python, the 33B base model outperforms CodeLlama-34B by 7.9 percent, while the 7B variant reaches performance comparable to CodeLlama-34B. The instruction-tuned 33B model surpasses GPT-3.5-turbo on HumanEval and achieves comparable results on MBPP. The models support an extensive range of 80 programming languages, from common languages like Python, JavaScript, and Java to specialized languages such as Solidity, CUDA, and Verilog.
A distinctive feature of DeepSeek Coder is its project-level code completion capability, enabled by a 16K context window and fill-in-the-blank training tasks. This allows the models to understand and complete code within the broader context of entire projects, effectively utilizing dependencies and relationships between files. The repository includes concrete examples demonstrating how the 6.7B model can correctly call classes and functions from multiple files to complete complex tasks.
The data creation process involved multiple filtering and preprocessing steps. The team collected code from GitHub, applied filtering rules consistent with StarCoder Data, parsed file dependencies to reorder files based on their relationships, concatenated dependent files into single examples, and applied repo-level deduplication using minhash. Additional filtering removed low-quality code containing syntax errors or poor readability.
The training procedure occurred in three stages. Initial pretraining used 1.8 trillion tokens with a 4K window size on a dataset of 87 percent code, 10 percent code-related language, and 3 percent non-code Chinese language. Extended pretraining followed with a 16K window on an additional 200 billion tokens to create the base models. Finally, instruction fine-tuning on 2 billion tokens of instruction data produced the instruction-tuned variants.
The repository provides practical tools for users, including inference examples for code completion, code insertion, chat-based interaction, and repository-level completion. A fine-tuning script supports downstream task adaptation using DeepSpeed, with documentation on dataset format requirements. The repository also addresses quantization concerns, documenting support for GGUF through llama.cpp and GPTQ through exllamav2, along with workarounds for the HuggingFace tokenizer compatibility.
GitGenius activity data shows median issue and pull request response latency of 292.1 hours across 80 tracked items, with mean latency of 2128.6 hours. The most active contributors tracked include sanwalsulehri with 8 events, wentinghome with 4 events, and guoday with 3 events. The repository shares overlapping contributors with vllm-project/vllm, microsoft/vscode, and langgenius/dify, indicating integration with inference optimization frameworks and development tools. The codebase is licensed under the MIT License, with model usage subject to a separate Model License that permits commercial applications.
