vllm-project/speculators

Speculators is a library for training and deploying speculative decoding draft models that integrate directly with vLLM inference engines.

Speculative decoding accelerates LLM inference by using a smaller draft model to propose multiple tokens ahead of time, which a larger base model then verifies in a single forward pass. This technique is lossless—every accepted token matches what the base model would have generated independently—so it reduces latency without compromising output quality. Speculators standardizes this process by providing an end-to-end framework to train draft models with reusable formats and tools, eliminating the need to build custom solutions for each deployment.

Teams should adopt this tool if they operate LLM inference servers and want to reduce latency without retraining or modifying their base models. It suits production environments where inference speed matters and where you can maintain a smaller draft model alongside your primary model. The tool is designed to work seamlessly with vLLM, so it is most valuable for teams already using or planning to use vLLM as their inference engine.

The project maintains active engagement with its community through dedicated Slack channels and public office hours presentations. Development appears focused on expanding the framework's capabilities and documentation to support broader adoption of speculative decoding techniques in production systems.