AssetOpsBench is a comprehensive benchmark designed to advance the development and evaluation of AI agents for industrial asset operations and maintenance within the context of Industry 4.0. Its primary purpose is to provide a unified framework for researchers and practitioners to build, orchestrate, and assess the performance of domain-specific AI agents in realistic industrial scenarios. The repository offers a valuable resource for those seeking to apply AI to improve efficiency, reliability, and predictive capabilities in industrial settings.

The core functionality of AssetOpsBench revolves around its ability to simulate and evaluate multi-step workflows within a controlled environment. It achieves this through a combination of domain-specific AI agents, multi-agent orchestration frameworks, and a rich dataset of industrial scenarios. The repository provides four key domain-specific agents, each designed to handle specific tasks relevant to asset operations: an IoT Agent for data retrieval, an FMSR (Failure Mode, Sensor, and Rule) Agent for failure analysis, a TSFM (Time Series Forecasting and Monitoring) Agent for predictive maintenance, and a WO (Work Order) Agent for automating work order generation. These agents act as the building blocks for more complex, multi-step workflows.

To facilitate the orchestration of these agents, AssetOpsBench offers two multi-agent frameworks: MetaAgent, which utilizes a reAct-based single-agent-as-tool approach, and AgentHive, which employs a plan-and-execute sequential workflow. These frameworks provide different strategies for coordinating the actions of the domain-specific agents, allowing users to experiment with various approaches to solve complex industrial problems. The repository also includes the necessary infrastructure, including MCP (Model Context Protocol) servers and a plan-execute runner, to support the execution and evaluation of these agents.

A key component of AssetOpsBench is its extensive dataset of 141 industrial scenarios. These scenarios cover a wide range of asset classes, including turbines, HVAC systems, pumps, and more. The tasks within these scenarios span various domains, such as IoT data retrieval, failure mode analysis, time series forecasting, and work order generation. Some scenarios focus on single-domain tasks, while others represent complex, multi-step end-to-end workflows, providing a comprehensive testing ground for AI agents. The dataset is available on Hugging Face, making it easily accessible for researchers and practitioners.

The repository also provides a leaderboard for evaluating the performance of AI agents. The leaderboards are generated using seven different Large Language Models (LLMs), and the trajectories of the agents are scored using an LLM Judge (Llama-4-Maverick-17B). The evaluation criteria are based on six dimensions, measuring reasoning, execution, and data handling capabilities. This allows for a standardized and objective comparison of different AI agent approaches.

AssetOpsBench is actively being expanded and improved. The repository includes a call for contributions, inviting researchers and practitioners to submit new scenarios and expand the scope of the benchmark. The project is also actively involved in the academic community, with accepted papers at major conferences such as NeurIPS, EMNLP, and AAAI. The repository provides resources such as tutorials, papers, and blog posts to help users understand and utilize the benchmark effectively. Furthermore, the repository offers pre-built Docker images and a detailed setup guide, making it easy for users to run and experiment with AssetOpsBench. Overall, AssetOpsBench is a valuable resource for anyone interested in developing and evaluating AI agents for industrial asset operations and maintenance.