AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming
2025/10/09 by Yutao Mou, Diao, Muxi, Mou, Yutao +13
Computer Science · #Adversarial Robustness in Machine Learning #Advanced Malware Detection Techniques #Anomaly Detection Techniques and Applications
paper · pdf · doi:10.48550/arxiv.2510.08329
Abstract
The safety of Large Language Models (LLMs) is crucial for the development of trustworthy AI applications. Existing red teaming methods often rely on seed instructions, which limits the semantic diversity of the synthesized adversarial prompts. We propose AutoRed, a free-form adversarial prompt generation framework that removes the need for seed instructions. AutoRed operates in two stages: (1) persona-guided adversarial instruction generation, and (2) a reflection loop to iteratively refine low-quality prompts. To improve efficiency, we introduce a verifier to assess prompt harmfulness without querying the target models. Using AutoRed, we build two red teaming datasets -- AutoRed-Medium and AutoRed-Hard -- and evaluate eight state-of-the-art LLMs. AutoRed achieves higher attack success rates and better generalization than existing baselines. Our results highlight the limitations of seed-based approaches and demonstrate the potential of free-form red teaming for LLM safety evaluation. We will open source our datasets in the near future.
Citations
- Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective
- SaRO: Enhancing LLM Safety through Reasoning-based Alignment
- SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
- The Llama 3 Herd of Models
- Qwen2 Technical Report
- Scaling Synthetic Data Creation with 1,000,000,000 Personas
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Benchmarking Benchmark Leakage in Large Language Models
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
- ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
- EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models
- Curiosity-driven Red-teaming for Large Language Models
- Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue
- CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models
- Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
- A StrongREJECT for Empty Jailbreaks
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
- How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities
- A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily
- MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
- Don't Make Your LLM an Evaluation Benchmark Cheater
- Jailbreaking Black Box Large Language Models in Twenty Queries
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts
- Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
- Universal and Transferable Adversarial Attacks on Aligned Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
- Jailbroken: How Does LLM Safety Training Fail?
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models
- Training language models to follow instructions with human feedback
- BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
- Language Models are Few-Shot Learners
Related