Purple Llama is Meta's umbrella project designed to provide tools and evaluations for assessing and improving the security of large language models. Written primarily in Python, the repository brings together safeguarding mechanisms and cybersecurity benchmarks to help developers build responsibly with open generative AI models. The project draws its name from the cybersecurity concept of purple teaming, which combines both red team attack and blue team defensive postures into a collaborative approach for evaluating and mitigating risks in generative AI systems.
The repository contains three primary safeguard components. Llama Guard is a series of input and output moderation models built by fine-tuning Meta-Llama models to detect violating content according to the MLCommons standard hazards taxonomy. Llama Guard 3 specifically supports detection of cyberattack responses and malicious code output, with models available in multiple sizes including 8B and 1B parameter versions, plus a 11B vision-capable variant. Prompt Guard protects LLM applications from malicious prompts by detecting prompt injection attacks and jailbreaks that attempt to override model safety features. Code Shield provides inference-time filtering of insecure code generated by LLMs, offering mitigation against insecure code suggestions and code interpreter abuse.
The evaluation and benchmarking component centers on the CyberSec Eval suite, which represents the first industry-wide cybersecurity safety evaluations for LLMs. CyberSec Eval v1 was built in collaboration with security subject matter experts using industry guidance from CWE and MITRE ATT&CK standards. It provides metrics for quantifying cybersecurity risks, tools for evaluating insecure code suggestion frequency, and assessments of LLM compliance with malicious requests. CyberSec Eval 2 expanded the scope to measure code interpreter abuse propensity, offensive cybersecurity capabilities, and prompt injection susceptibility. The most recent CyberSec Eval 3 added test suites for visual prompt injection, spear phishing capabilities, and autonomous offensive cyber operations, with results available on a Hugging Face leaderboard.
Licensing across Purple Llama components is permissive to enable both research and commercial usage. Evaluations and benchmarks use the MIT license, while safeguard models use the corresponding Llama Community licenses depending on their base model version. The project emphasizes community collaboration and standardization of trust and safety tools for generative AI development, with contribution guidelines available in the repository's CONTRIBUTING file and integration planned within the broader Llama reference system and llama-recipes repository.
