Giskard is an open-source Python library designed for testing and evaluating agentic systems, with a v3 architecture that represents a complete rewrite focused on dynamic, multi-turn testing of AI agents. The library is built as a modular set of focused packages, each carrying only the dependencies it needs, enabling it to wrap anything from a single LLM to a black-box agent or multi-step pipeline. The project requires Python 3.12 and is written primarily in Python.
The v3 release drops heavy dependencies for better efficiency while introducing a more powerful AI vulnerability scanner and enhanced RAG evaluation capabilities. The architecture consists of three main components at different maturity levels: giskard-checks is in beta status and provides testing and evaluation through a scenario API with built-in checks and LLM-as-judge functionality; giskard-scan is currently in progress and serves as an agent vulnerability scanner for red teaming, prompt injection detection, and data leakage identification; and giskard-rag is planned for future release to handle RAG evaluation and synthetic data generation. These components succeed corresponding features from the v2 release, which remains available but is no longer actively maintained.
Giskard Checks is a lightweight library for creating evaluations that test LLM-based systems, ranging from simple assertions to LLM-as-judge assessments. It is specifically designed for non-deterministic outputs where the same input can produce different valid responses. Users can employ Giskard Checks to catch regressions, validate RAG quality by checking if answers are grounded in retrieved context, enforce safety rules to ensure outputs conform to content policies, and evaluate multi-turn agents by testing full conversations rather than single exchanges. The library includes built-in evaluations for string matching, comparisons, regex patterns, semantic similarity, and LLM-as-judge checks such as Groundedness, Conformity, and LLMJudge.
Giskard Scan functions as the red-teaming and vulnerability scanning layer for agentic systems. It automatically generates adversarial test suites from plain-language descriptions of agents, covering threat categories including prompt injection, harmful content, stereotypes, and misinformation. The scanner enables users to red-team agents automatically across OWASP LLM Top-10 threat categories, run prompt-injection probes using a built-in dataset of injection payloads, and extend functionality with custom generators through the ScenarioGenerator interface or the vulnerability_suite_generator_registry.
The library includes optional, aggregated usage analytics to help improve the product, with no prompts, model outputs, or scenario text included in telemetry collection, and users can opt out of this tracking.
