TorchRL is a PyTorch-native toolkit for reinforcement learning that provides modular, composable building blocks for constructing RL systems while maintaining compatibility with PyTorch's programming model.
The library addresses the fragmentation in RL development by establishing a unified data model centered on TensorDict, a dictionary-like tensor container that carries structured information through the entire training pipeline. Rather than implementing specific algorithms in isolation, TorchRL provides independent, swappable modules for environments, policies, replay buffers, objectives, and collectors. This approach allows researchers to write code that scales seamlessly from local prototypes to vectorized, multiprocess, distributed, compiled, recurrent, multi-agent, model-based, or offline workflows without changing the underlying data model. The three core design principles ensure data maintains names, structure, batch dimensions, and devices throughout; that components remain independent and interchangeable; and that research code can evolve in complexity without architectural rewrites.
Developers should adopt TorchRL if they are building custom RL systems that require flexibility and composability rather than using pre-packaged algorithm implementations. The toolkit excels for recurrent RL, MuJoCo-based control, multi-agent training, and scenarios where replay-buffer and collector infrastructure matter. It suits projects where the ability to swap components and scale from prototype to production without code changes provides significant value. The library is particularly strong for those already comfortable with PyTorch who want RL capabilities that integrate naturally with the PyTorch ecosystem rather than introducing a separate abstraction layer.
The project maintains active development across multiple capability areas simultaneously. Recent work has expanded recurrent RL performance through scan operations and Triton-based GRU/LSTM handling. Multi-agent support has been strengthened with algorithm implementations including MAPPO and IPPO alongside utility components like value normalization and mixer configurations. Collector and replay-buffer ergonomics have been improved through async prioritized writes, ordered storage access, and optional CUDA-accelerated kernels. The codebase includes state-of-the-art implementations and examples spanning satellite control and macro-control policies alongside core algorithmic components.
