modelscope/evalscope

EvalScope is a framework for evaluating large language models, vision language models, and generative AI systems through benchmarking and performance testing.

The tool addresses the challenge of comprehensively assessing model capabilities across different domains and modalities. It provides built-in evaluation benchmarks such as MMLU, C-Eval, and GSM8K, alongside support for multiple model types including language models, vision-language models, embedding models, rerankers, and generative systems. The framework integrates multiple evaluation backends including OpenCompass and VLMEvalKit, and includes an agent evaluation mode that runs benchmarks within a controlled multi-turn loop with pluggable strategies, tools, and Docker sandbox support. It also offers inference performance testing with metrics like time-to-first-token and time-per-output-token, plus an interactive web dashboard for model comparison and an arena mode for pairwise model battles.

Developers should choose this tool if they need to evaluate multiple model types across different domains with a single framework. It suits projects requiring comprehensive benchmarking with built-in industry-recognized datasets, performance stress testing capabilities, and interactive visualization of results. The framework is designed to be highly extensible, allowing developers to add custom datasets, models, and evaluation metrics to fit specific evaluation needs.

The project maintains active development with regular updates to its evaluation benchmarks and backend integrations. Documentation is available in multiple languages with both ReadTheDocs and a dedicated product website. The codebase supports Python and is distributed through standard package management channels. The project welcomes community contributions through pull requests and actively encourages adoption through its public repository and comprehensive documentation resources.