Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

2025/01/31 by Mrinank Sharma, Meg Tong, Sharma, Mrinank +89 · 8 voices · 60 citations
Social Sciences · #Criminal Law and Evidence #Law, Rights, and Freedoms #Legal Systems and Judicial Processes

paper · pdf · doi:10.48550/arxiv.2501.18837

Abstract

Large language models (LLMs) are vulnerable to universal jailbreaks-prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Constitutional Classifiers: safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content. In over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak that could extract information from an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries. On automated evaluations, enhanced classifiers demonstrated robust defense against held-out domain-specific jailbreaks. These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead. Our work demonstrates that defending against universal jailbreaks while maintaining practical deployment viability is tractable.

Cited by

• The Trojan Example: Jailbreaking LLMs through Template Filling and Unsafety Reasoning
• Boundary Point Jailbreaking of Black-Box LLMs
• Practical challenges of control monitoring in frontier AI deployments
• What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents
• SAGE: Safety-First Defense-in-Depth Guardrails for Verified Lifecycle Control of High-Impact Generative AI
• Latent Space Probing for Adult Content Detection in Video Generative Models
• Scaling Reinforcement Learning for Content Moderation with Large Language Models
• Invasive Context Engineering to Control Large Language Models
• Crystalyse: a multi-tool agent for materials design
• Password-Activated Shutdown Protocols for Misaligned Frontier Agents
• The Impact of Off-Policy Training Data on Probe Generalisation
• Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges
• Adaptively Robust LLM Monitoring via Activation Watermarking
• Adversarial Déjà Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attacks
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
• CourtGuard: A Local, Multiagent Prompt Injection Classifier
• Qwen3Guard Technical Report
• Don't Walk the Line: Boundary Guidance for Filtered Generation
• All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language
• The Alignment Waltz: Jointly Training Agents to Collaborate for Safety
• Stress-Testing Model Specs Reveals Character Differences among Language Models
• Bypassing Prompt Guards in Production with Controlled-Release Prompting
• Agentic Misalignment: How LLMs Could Be Insider Threats
• Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed
• Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?
• D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models
• Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain
• PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming
• CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention
• Evaluating Language Model Reasoning about Confidential Information
• Real-Time Detection of Hallucinated Entities in Long-Form Generation
• Involuntary Jailbreak: On Self-Prompting Attacks
• Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development
• Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity
• Multi-Turn Jailbreaks Are Simpler Than They Seem
• A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection
• LoRA is All You Need for Safety Alignment of Reasoning LLMs
• PurpCode: Reasoning for Safer Code Generation
• Shaping capabilities with token-level data filtering
• Combining Cost-Constrained Runtime Monitors for AI Safety
• Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing
• Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreak
• The bitter lesson of misuse detection
• The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models
• STACK: Adversarial Attacks on LLM Safeguard Pipelines
• A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreaks
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards
• When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment
• Model Editing as a Double-Edged Sword: Steering Agent Ethical Behavior Toward Beneficence or Harm
• LLM Jailbreak Oracle
• RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?
• FORTRESS: Frontier Risk Evaluation for National Security and Public Safety
• Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Values
• Jailbreak Transferability Emerges from Shared Representations
• SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
• Detecting High-Stakes Interactions with Activation Probes
• SoK: Evaluating Jailbreak Guardrails for Large Language Models
• Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors
• From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring
• Benchmarking Misuse Mitigation Against Covert Adversaries

Discussions

• How to defend LLM-based systems from jailbreaks? Add more LLMs? Experienced infosec folks know this approach well. Now there are two models one needs to jailbreak :) arxiv.org/pdf/2501.18837 [bsky, 8 points, 3 comments]
• Anthropic's Constitutional Classifiers needs an open alternative the promise is sweet — come up with a "constitution", a list of plain-text rules, and the rest can be automated. synthetic data, LLM f [bsky, 6 points, 0 comments]
• arxiv.org/abs/2501.188... [bsky, 3 points, 0 comments]
• Constitutional Classifiers from Anthropic can be used to defend large language models against universal jailbreaks - use natural language rules to generate synthetic training data for safeguards - fo [bsky, 2 points, 0 comments]
• arxiv.org/pdf/2501.18837 #hardening llm (: [bsky, 2 points, 1 comments]
• have you read anthropic's recent paper on constitutional classifiers? i feel there's a ton of low hanging alpha at the nexus of both approaches (i.e. SFT output classifier on SPRI convo examples inste [bsky, 1 points, 1 comments]
• This is interesting! Confront universal jailbreaks with a practical approach using "Constitutional Classifiers." Define a clear content constitution and generate synthetic training data to build robus [bsky, 0 points, 0 comments]
• Thoroughly enjoyed reading the paper on constitutional classifiers for defending against universal jailbreaks in #AI #AISecurity #AISafety #TechSky arxiv.org/pdf/2501.18837 [bsky, 0 points, 0 comments]

Related

• Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approach
• Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning
• Boundary Point Jailbreaking of Black-Box LLMs
• BreakFun: Jailbreaking LLMs via Schema Exploitation
• Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations