Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers

2025/10/14 by Bertollo, Giacomo, Bodemir, Naz, Burgess, Jonah
#Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.16005

Abstract

Analyzing 500 CTF participants, this paper shows that while participants readily bypassed simple AI guardrails using common techniques, layered multi-step defenses still posed significant challenges, offering concrete insights for building safer AI systems.

Citations

• AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defenses
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
• WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
• InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
• Red Teaming Language Models with Language Models

Related

• Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs
• Cognitive Cybersecurity for Artificial Intelligence: Guardrail Engineering with CCS-7
• Peering Behind the Shield: Guardrail Identification in Large Language Models
• No Free Lunch with Guardrails
• Exploiting AI for Attacks: On the Interplay between Adversarial AI and Offensive AI