Simple-evals is a lightweight library for evaluating language models.
The tool addresses the need for transparent, reproducible evaluation of language model performance. It provides reference implementations for specific benchmarks that OpenAI uses when publishing accuracy numbers alongside model releases. The library is designed to be straightforward enough that evaluation methodology can be clearly understood and verified by external parties reviewing published results.
The project suits teams that need to evaluate language models using established benchmarks and want access to the exact implementations behind published accuracy claims. It is most relevant for those conducting model comparisons or validation work where understanding the evaluation methodology is critical. The tool is not intended as a general-purpose evaluation framework with ongoing updates for new models or benchmarks.
The project is in a maintenance-only state, with the README explicitly noting that it will no longer receive updates for new models or benchmark results. The repository will continue to host reference implementations for HealthBench, BrowseComp, and SimpleQA as stable, archived resources.
