UI-TARS is a Python-based research project from ByteDance that develops automated GUI interaction agents powered by vision-language models. The repository represents a pioneering effort in creating native agents capable of understanding and interacting with graphical user interfaces across diverse environments including desktop applications, web browsers, mobile devices, and games.
The project centers on UI-TARS-1.5, an open-source multimodal agent built on a powerful vision-language model that integrates advanced reasoning capabilities enabled by reinforcement learning. This architecture allows the model to reason through its thoughts before taking action, significantly enhancing performance and adaptability particularly in inference-time scaling. The repository also documents UI-TARS-2, released in September 2025, which represents a major upgrade featuring enhanced capabilities in GUI interaction, game playing, code execution, and tool use, positioning it as an all-in-one agent model for complex tasks.
The codebase provides three distinct prompt templates designed for different use cases: COMPUTER_USE for desktop environments supporting mouse clicks, drag actions, keyboard shortcuts, and text input; MOBILE_USE for mobile devices and Android emulators with mobile-specific actions like long_press and app launching; and GROUNDING for lightweight tasks focused solely on action output without reasoning components. This flexibility allows developers to deploy the agent across Windows, Linux, macOS, mobile platforms, and web browsers.
Performance benchmarks demonstrate competitive results across multiple evaluation frameworks. On OSWorld, UI-TARS-1.5 achieves 42.5 percent success rate in 100 steps, surpassing previous state-of-the-art results. The model scores 42.1 percent on Windows Agent Arena, 84.8 percent on WebVoyager for browser automation, and 64.2 percent on Android World for mobile tasks. In grounding capability evaluation, it reaches 94.2 percent on ScreenSpot-V2 and 61.6 percent on ScreenSpotPro. Notably, the model demonstrates exceptional performance on Poki games, achieving perfect 100 percent scores across 14 different game titles, substantially outperforming OpenAI CUA and Claude 3.7. In Minecraft tasks, UI-TARS-1.5 with thought reasoning achieves 0.42 average success rate on 200 mining tasks and 0.31 on 100 mob-killing tasks.
The repository includes comprehensive deployment documentation, inference scripts compatible with the OSWorld benchmark, and coordinate processing guides to assist users in implementing the model across different environments. The project maintains an active community presence with Discord support and provides access to models through Hugging Face, alongside a desktop-specific version available in a separate UI-TARS-desktop repository for local device operation.
