Mitigating Many-Shot Jailbreaking

2025/04/13 by Christopher Ackerman, Nina Panickssery, Ackerman, Christopher M. +1 · 5 citations
Computer Science · #Advanced Malware Detection Techniques #Adversarial Robustness in Machine Learning #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.09604

openalex publication_date 2025/04/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many-shot jailbreaking (MSJ) is an adversarial technique that exploits the long context windows of modern LLMs to circumvent model safety training by including in the prompt many examples of a "fake" assistant responding inappropriately before the final request. With enough examples, the model's in-context learning abilities override its safety training, and it responds as if it were the "fake" assistant. In this work, we probe the effectiveness of different fine-tuning and input sanitization approaches on mitigating MSJ attacks, alone and in combination. We find incremental mitigation effectiveness for each, and show that the combined techniques significantly reduce the effectiveness of MSJ attacks, while retaining model performance in benign in-context learning and conversational tasks. We suggest that our approach could meaningfully ameliorate this vulnerability if incorporated into model safety post-training.

Citations

• Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instruct
• Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
• OR-Bench: An Over-Refusal Benchmark for Large Language Models
• Steering Llama 2 via Contrastive Activation Addition
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
• Extracting Latent Steering Vectors from Pretrained Language Models
• An Explanation of In-context Learning as Implicit Bayesian Inference
• Language Models are Few-Shot Learners

Cited by

• Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards

Related

• Multi-round jailbreak attack on large language models
• Fight Back Against Jailbreaking via Prompt Adversarial Tuning
• MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue
• Distract Large Language Models for Automatic Jailbreak Attack
• Break the Breakout: Reinventing LM Defense Against Jailbreak Attacks with Self-Refinement