DeepSpec is a full-stack codebase for training and evaluating draft models used in speculative decoding.
The tool addresses the problem of accelerating language model inference through speculative decoding, a technique where a smaller draft model generates candidate tokens that a larger target model validates. DeepSpec provides an end-to-end pipeline: it prepares training data by downloading prompts, regenerating target model answers, and building cached target outputs; trains draft models against these cached outputs using configurable algorithms; and evaluates the resulting models on benchmark tasks to measure acceptance rates during speculative decoding. The workflow is sequential, with each stage's output feeding into the next.
Developers should choose this tool if they are researching or implementing speculative decoding algorithms and need a complete, reproducible framework rather than building from scratch. It suits projects targeting language model inference optimization where reducing latency through token prediction acceleration is valuable. The codebase includes multiple algorithm configurations and supports various target models, with released checkpoints available for immediate evaluation. The tool assumes a single-node setup with eight GPUs by default, though it can be adapted for fewer GPUs by adjusting environment variables. Users should be aware that data preparation can require substantial storage—the default configuration generates approximately 38 TB of cached target outputs.
The project shows active development with recent commits addressing core functionality. Work spans the full stack from data pipeline implementation through model training orchestration to evaluation harness construction. The codebase maintains multiple algorithm variants and target model configurations, indicating ongoing experimentation and refinement of the speculative decoding approach.
