The Jailbreak Tax: How Useful are Your Jailbreak Outputs?

2025/04/14 by Kristina Nikolić, Nikolić, Kristina, Jie Zhang +4 · 7 citations
Computer Science · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2504.10694

openalex publication_date 2025/04/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Jailbreak attacks bypass the guardrails of large language models to produce harmful outputs. In this paper, we ask whether the model outputs produced by existing jailbreaks are actually useful. For example, when jailbreaking a model to give instructions for building a bomb, does the jailbreak yield good instructions? Since the utility of most unsafe answers (e.g., bomb instructions) is hard to evaluate rigorously, we build new jailbreak evaluation sets with known ground truth answers, by aligning models to refuse questions related to benign and easy-to-evaluate topics (e.g., biology or math). Our evaluation of eight representative jailbreaks across five utility benchmarks reveals a consistent drop in model utility in jailbroken responses, which we term the jailbreak tax. For example, while all jailbreaks we tested bypass guardrails in models aligned to refuse to answer math, this comes at the expense of a drop of up to 92% in accuracy. Overall, our work proposes the jailbreak tax as a new important metric in AI safety, and introduces benchmarks to evaluate existing and future jailbreaks. We make the benchmark available at https://github.com/ethz-spylab/jailbreak-tax

Cited by

• All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language
• Bypassing Prompt Guards in Production with Controlled-Release Prompting
• ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search
• Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models
• bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs
• Jailbreaking Large Language Models Through Content Concretization
• Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs

Related

• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
• Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
• JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models
• JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models
• A StrongREJECT for Empty Jailbreaks