garak: A Framework for Security Probing Large Language Models

2024/06/16 by Leon Derczynski, Derczynski, Leon, Erick Galinkin +7 · 14 citations
Computer Science · #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2406.11036

openalex publication_date 2024/06/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As Large Language Models (LLMs) are deployed and integrated into thousands of applications, the need for scalable evaluation of how models respond to adversarial attacks grows rapidly. However, LLM security is a moving target: models produce unpredictable output, are constantly updated, and the potential adversary is highly diverse: anyone with access to the internet and a decent command of natural language. Further, what constitutes a security weak in one context may not be an issue in a different context; one-fits-all guardrails remain theoretical. In this paper, we argue that it is time to rethink what constitutes ``LLM security'', and pursue a holistic approach to LLM security evaluation, where exploration and discovery of issues are central. To this end, this paper introduces garak (Generative AI Red-teaming and Assessment Kit), a framework which can be used to discover and identify vulnerabilities in a target LLM or dialog system. garak probes an LLM in a structured fashion to discover potential vulnerabilities. The outputs of the framework describe a target model's weaknesses, contribute to an informed discussion of what composes vulnerabilities in unique contexts, and can inform alignment and policy discussions for LLM deployment.

Cited by

• Quantifying Return on Security Controls in LLM Systems
• Chasing Shadows: Pitfalls in LLM Security Research
• Uncovering Competency Gaps in Large Language Models and Their Benchmarks
• LLM Harms: A Taxonomy and Discussion
• A Safety and Security Framework for Real-World Agentic Systems
• Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detection
• IRSDA: An Agent-Orchestrated Framework for Enterprise Intrusion Response
• ForgeDAN: An Evolutionary Framework for Jailbreaking Aligned Large Language Models
• AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
• SoK: Taxonomy and Evaluation of Prompt Security in Large Language Models
• BlackIce: A Containerized Red Teaming Toolkit for AI Security Testing
• Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering
• Risk Assessment and Security Analysis of Large Language Models
• LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems

Related

• Emerging Security Challenges of Large Language Models
• Security Concerns for Large Language Models: A Survey
• Exploring Vulnerabilities and Protections in Large Language Models: A Survey
• Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
• Attack and defense techniques in large language models: A survey and new perspectives