Harbor is a framework for evaluating and optimizing agents and language models.
Harbor addresses the challenge of systematically testing and improving AI agents across diverse environments and tasks. It provides a unified harness for running evaluations against arbitrary agents such as Claude Code, OpenHands, and Codex CLI. The framework allows users to construct custom benchmarks and environments, then execute experiments across thousands of parallel instances by integrating with cloud providers including Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, and Runta. Beyond evaluation, Harbor can generate rollouts suitable for reinforcement learning optimization workflows.
Harbor suits teams building or benchmarking agent systems who need reproducible evaluation infrastructure. It is particularly relevant for those working with Terminal-Bench-2.0, for which Harbor serves as the official evaluation harness, and for researchers comparing multiple agent implementations. The framework supports established benchmarks like SWE-Bench and Aider Polyglot alongside custom environments, making it adaptable to domain-specific evaluation needs. Local evaluation via Docker is available for development, while cloud provider integration enables scaling to production-grade experiment volumes.
The project maintains active engagement with its user base through the Harbor Cookbook, which documents end-to-end workflows and examples. Development activity shows consistent refinement of the evaluation pipeline and expansion of supported providers and benchmarks. The framework receives updates that extend compatibility with emerging agent systems and cloud infrastructure options.
