TinyZero is a reinforcement learning framework that reproduces DeepSeek R1-Zero's reasoning capabilities on small language models.
The project demonstrates how a 3 billion parameter base language model can develop self-verification and search abilities through reinforcement learning on constrained tasks like countdown and multiplication. Rather than relying on instruction tuning or external reasoning modules, the approach trains the model end-to-end using RL to discover reasoning strategies organically. The framework is built on top of veRL and uses Qwen2.5 series models as the base, making it accessible for experimentation on modest hardware budgets.
The tool suits researchers and practitioners interested in understanding how reasoning emerges in language models through RL, particularly those working with smaller models where compute constraints are a concern. The project includes implementations for both countdown and multiplication tasks, with support for both base and instruction-tuned variants. Training is possible on single GPUs for models up to 1.5 billion parameters, though the framework shows more sophisticated reasoning development with 3 billion parameter models. The README notes that smaller base models like 0.5B variants fail to learn reasoning effectively, which is important context for choosing model sizes.
The project is no longer actively maintained, with the maintainers directing users toward the veRL library for ongoing RL experimentation work. The repository includes full experiment logs and documentation of the training process, allowing others to reproduce or build upon the results. A deprecation notice appears prominently in the README, making clear that this is a snapshot of a completed research effort rather than an actively developed tool.
