EasyR1 is a reinforcement learning training framework that enables efficient and scalable RL training for language models and vision language models.
The framework addresses the challenge of training large language models with reinforcement learning algorithms by building on the veRL project and extending it to support vision language models. It achieves efficiency and scalability through the HybridEngine design and integration with vLLM's SPMD mode. The tool supports multiple RL algorithms including GRPO, DAPO, Reinforce++, ReMax, RLOO, GSPO, and CISPO, and works with models from Llama, Qwen, and DeepSeek families. It includes training optimizations such as padding-free training and LoRA fine-tuning, along with experiment tracking through Wandb, SwanLab, Mlflow, and Tensorboard.
Teams should choose this tool if they need to run RL training on vision language models or language models at scale. It suits projects requiring multiple RL algorithm implementations and supports both full fine-tuning and parameter-efficient LoRA training. The framework provides hardware requirement estimates across different model sizes and training methods, ranging from single-GPU setups for smaller models to multi-GPU distributed training for larger ones. Docker and Apptainer environments are available to simplify setup, and the tool supports downloading models from both Hugging Face and ModelScope hubs.
The project maintains active development with recent additions of LoRA training, DAPO, GSPO, and CISPO algorithms. The codebase includes comprehensive examples and tutorials for common training scenarios, such as running Qwen2.5-VL on specific datasets. Documentation covers custom dataset preparation and provides clear installation and training workflows. The project is used in production by major cloud providers and includes practical features like checkpoint resuming and multiple experiment tracking integrations.
