web-infra-dev/midscene

Midscene.js is a GUI agent for end-to-end testing that uses AI-powered vision to automate interactions across web, mobile, and desktop platforms.

The tool solves the fragility of traditional UI automation by working from screenshots rather than relying on DOM selectors or accessibility trees. It accepts natural language descriptions of test steps and uses computer vision to understand what is displayed on screen, then performs the corresponding actions. This approach handles elements that lack semantic markup, custom controls, canvas elements, native applications, and cross-origin iframes—scenarios where conventional selector-based automation fails. The tool includes built-in support for multiple platforms and comes with batteries included for common testing scenarios.

Midscene suits teams building cross-platform applications who want to reduce maintenance burden from brittle selectors and need to test visual correctness alongside functionality. It works well for testing native mobile apps, web applications with complex custom UI, and scenarios involving multiple platforms in a single workflow. The project is particularly valuable when you cannot rely on stable DOM structure or need to verify that the interface actually looks correct, not just that elements exist.

The project maintains active development with regular updates to its core vision capabilities and platform support. The tool has expanded beyond web testing to include iOS and Android automation, demonstrating ongoing investment in cross-platform functionality. Community engagement is evident through showcases spanning web registration flows, mobile app interactions, and specialized use cases like robotic arm integration with vision systems.