safety-research/bloom

Bloom is a Python tool for generating and running automated behavioral evaluations of large language models.

The tool addresses the challenge of systematically probing LLMs for specific behaviors like sycophancy, self-preservation, or political bias. Rather than relying on fixed benchmarks, Bloom takes a seed configuration describing a target behavior and generates diverse test scenarios tailored to that behavior. It then executes conversations with a target model and scores the results to measure behavior presence. The approach treats evaluation suites as reproducible artifacts tied to their seed configuration, allowing evaluations to be cited with full transparency about how they were constructed.

Bloom suits teams building safety evaluations or conducting interpretability research on language models. It works well for organizations that need to probe custom behaviors beyond standard benchmarks, or that want to test how stable a behavior is across variations—such as different noise levels or emotional pressure. The tool supports multiple model providers through LiteLLM integration and offers both conversation and simulated environment modalities. For large-scale experiments, it integrates with Weights and Biases for sweep-based evaluation across multiple models and configurations.

Development on this repository has been frozen at its last standalone release, with the project now maintained elsewhere. The tool remains usable as documented, but new projects should adopt the actively maintained version. The codebase provides pipeline stages that can be run individually—understanding, ideation, rollout, and judgment—allowing users to customize their evaluation workflow. An interactive viewer for results and an interactive chat mode for manual testing are included for exploration and debugging.