promptfoo/promptfoo

Promptfoo is a comprehensive testing and evaluation framework for large language model applications, written in TypeScript and available as both a CLI tool and library. The project addresses the critical need for systematic evaluation of LLM-based systems by providing automated testing, red teaming capabilities, and vulnerability scanning rather than relying on trial-and-error development approaches. The tool enables developers to test prompts and models through automated evaluations, secure their LLM applications through red teaming and vulnerability scanning, compare performance across multiple model providers including OpenAI, Anthropic, Azure, Bedrock, and Ollama, and integrate security checks directly into CI/CD pipelines.

The repository emphasizes developer experience with features like live reload and caching for fast iteration cycles. A key differentiator is its privacy-first approach where LLM evaluations run entirely locally, ensuring that prompts and sensitive data never leave the developer's machine. The framework supports flexible integration with any LLM API or programming language, making it adaptable to diverse development environments. Promptfoo is battle-tested in production, powering LLM applications that serve over 10 million users, which validates its reliability and effectiveness at scale.

The project has achieved significant adoption and recognition, being used by both OpenAI and Anthropic according to its description. The codebase maintains an MIT license and remains open source despite being acquired by OpenAI. Installation is straightforward through multiple channels including npm, Homebrew, and pip, with Node.js version requirements of 20.20.0 or higher for 22.22.0 and above.

The repository connects to related projects including github/gh-aw, berriai/litellm, and anthropics/claude-code through overlapping contributor networks, suggesting integration within a broader ecosystem of LLM development tools.

The framework supports code scanning for reviewing pull requests related to LLM security and compliance, and provides security vulnerability reporting capabilities. Results can be shared with teams through various output formats. The project maintains comprehensive documentation covering getting started guides, red teaming methodologies, CLI usage, Node.js package integration, and supported model providers. An active Discord community provides peer support and discussion channels for users.