Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models

2025/06/08 by Ren-Jian Wang, Ke Xue, Wang, Ren-Jian +17 · 4 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Topic Modeling #cs.AI #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2506.07121

openalex publication_date 2025/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28 · arxiv created 2026/08/01 · arxiv updated 2026/08/04

Abstract

Ensuring the safety and robustness of large language models (LLMs) is a fundamental challenge and a critical prerequisite for the responsible deployment of artificial intelligence. Red-teaming, a systematic framework to identify adversarial prompts that elicit harmful responses from target LLMs, has emerged as a crucial safety evaluation paradigm. Within this framework, the diversity of adversarial prompts is critical for a comprehensive safety assessment. However, previous red-teaming approaches often pursue diversity through simplistic metrics such as word frequency or sentence embedding similarity, which may not capture meaningful variation in attack strategies. In addition, the common practice of training a single attacker model restricts coverage across all potential attack styles and risk categories. This paper introduces Quality-Diversity Red-Teaming (QDRT), a new framework designed to address these limitations. QDRT achieves goal-driven diversity through behavior-conditioned training and implements a behavioral replay buffer in an open-ended manner. In addition, it trains multiple specialized attackers capable of generating high-quality attacks across diverse styles and risk categories. Our empirical evaluation demonstrates that QDRT generates attacks that are both more diverse and more effective against a wide range of target LLMs, including open-source models GPT-2, Llama-3, Gemma-2, Qwen2.5, and commercial models GPT-4.1 and GPT-5-Chat. This work advances the field of LLM safety by providing a systematic and effective approach to automated red-teaming, ultimately supporting the responsible deployment of LLMs. Our code is available at https://github.com/lamda-bbo/QDRT.

Citations

• A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
• Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models
• Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning
• Qwen2.5 Technical Report
• AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs
• Automated Red Teaming with GOAT: the Generative Offensive Agent Tester
• Preserving Diversity in Supervised Fine-Tuning of Large Language Models
• Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique
• The Llama 3 Herd of Models
• Jailbreak Attacks and Defenses Against Large Language Models: A Survey
• Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teaming
• Open-Endedness is Essential for Artificial Superhuman Intelligence
• Learning diverse attacks on large language models for robust red-teaming and safety tuning
• Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
• Curiosity-driven Red-teaming for Large Language Models
• Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
• A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
• Jailbreaking Black Box Large Language Models in Twenty Queries
• Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Jailbroken: How Does LLM Safety Training Fail?
• Red Teaming Language Model Detectors with Language Models
• Query-Efficient Black-Box Red Teaming via Bayesian Optimization
• GPT-4 Technical Report
• Evolving Populations of Diverse RL Agents with MAP-Elites
• pyribs: A Bare-Bones Python Library for Quality Diversity Optimization
• Constitutional AI: Harmlessness from AI Feedback
• Benchmarking Quality-Diversity Algorithms on Neuroevolution for Reinforcement Learning
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
• Deep Surrogate Assisted Generation of Environments
• Training language models to follow instructions with human feedback
• Red Teaming Language Models with Language Models
• Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
• Quality-Diversity Optimization: a novel branch of stochastic optimization
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Network
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
• Proximal Policy Optimization Algorithms
• Quality and Diversity Optimization: A Unifying Modular Framework
• Illuminating search spaces by mapping elites

Cited by

• Soft Quality-Diversity Optimization
• TEAR: Temporal-aware Automated Red-teaming for Text-to-Video Models
• Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts
• Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming

Related

• Learning diverse attacks on large language models for robust red-teaming and safety tuning
• MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
• Automated Progressive Red Teaming
• Attack Prompt Generation for Red Teaming and Defending Large Language Models
• Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming