ukgovernmentbeis/inspect_ai

Inspect is a framework for large language model evaluations that provides built-in components for prompt engineering, tool usage, multi-turn dialog, and model graded assessments.

The framework addresses the need to systematically evaluate language model behavior and capabilities. It supplies pre-built evaluation components that handle common assessment patterns, and allows developers to extend its functionality through Python packages to support custom elicitation and scoring techniques. The tool includes a collection of over 200 pre-built evaluations that can run against any model without modification.

Teams building evaluation pipelines for language models should consider Inspect if they need a structured, extensible approach to assessment rather than ad-hoc testing. The framework suits projects that require reproducible evaluation across multiple models or that need to share evaluation definitions across teams. The tool's documentation is available in multiple formats including a structured index for coding agents and concatenated markdown guides for reference.

The project maintains a reproducible development environment through a checked-in lockfile while keeping dependency declarations in standard requirements files. Development workflows are supported through a Makefile that runs linting, formatting, and tests, with optional integration for the uv package manager. The codebase includes a TypeScript and React frontend as a git submodule, allowing frontend contributors to work independently from Python-only development. Documentation is built with Quarto and can be developed iteratively with IDE support. The project recommends specific VS Code extensions for contributors and provides setup guidance for both Python and frontend development paths.