ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks

2025/07/02 by Ziming Ren, Siyuan Liang, Ren, Zhiyao +5 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.01321

openalex publication_date 2025/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In-context learning (ICL) has demonstrated remarkable success in large language models (LLMs) due to its adaptability and parameter-free nature. However, it also introduces a critical vulnerability to backdoor attacks, where adversaries can manipulate LLM behaviors by simply poisoning a few ICL demonstrations. In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demonstrations, jointly influencing the probability of model outputs. Through theoretical analysis, we derive an upper bound for ICL backdoor effects, revealing that the vulnerability is dominated by the concept preference ratio between the task and the backdoor. Motivated by these findings, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio. Our method encourages LLMs to select clean demonstrations during the ICL phase by leveraging confidence and similarity scores, effectively mitigating susceptibility to backdoor attacks. Extensive experiments across multiple LLMs and tasks demonstrate that our method achieves state-of-the-art defense effectiveness, significantly outperforming existing approaches (+26.02% on average). Furthermore, our method exhibits exceptional adaptability and defensive performance even for closed-source models (e.g., GPT-4).

Citations

• What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis
• Adversarial Backdoor Defense in CLIP
• Compromising Embodied Agents with Contextual Backdoor Attacks
• The Llama 3 Herd of Models
• Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
• BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models
• Fine-grained Controllable Text Generation through In-context Learning with Feedback
• Towards Robust Physical-world Backdoor Attacks on Lane Detection
• Language Models for Text Classification: Is In-Context Learning Enough?
• Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning
• BadEdit: Backdooring large language models by model editing
• Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultation
• Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detection
• Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning
• Wordflow: Social Prompt Engineering for Large Language Models
• Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning
• Pre-trained Trojan Attacks for Visual Recognition
• Universal Jailbreak Backdoors from Poisoned Human Feedback
• Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations
• In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering
• Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
• In-Context Learning for Text Classification with Many Labels
• RobustMQ: Benchmarking Robustness of Quantized Models
• Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Backdoor Attacks for In-Context Learning with Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models
• The Closeness of In-Context Learning and Weight Shifting for Softmax Regression
• GPT-4 Technical Report
• Compositional Exemplars for In-context Learning
• Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
• Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
• Transformers learn in-context by gradient descent
• Structured Prompting: Scaling In-Context Learning to 1,000 Examples
• Universal Backdoor Attacks Detection via Adaptive Adversarial Probe
• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
• OPT: Open Pre-trained Transformer Language Models
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• An Explanation of In-context Learning as Implicit Bayesian Inference
• Training Verifiers to Solve Math Word Problems
• Anti-Backdoor Learning: Training Clean Models on Poisoned Data
• Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
• What Makes Good In-Context Examples for GPT-3?
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
• ONION: A Simple and Effective Defense Against Textual Backdoor Attacks
• Language Models are Few-Shot Learners
• Character-level Convolutional Networks for Text Classification
• Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Cited by

• SAID: Safety-Aware Intent Defense via Prefix Probing for Large Language Models
• Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs
• Towards Size-invariant Salient Object Detection: A Generic Evaluation and Optimization Approach

Related

• Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning
• Backdoor Attacks for In-Context Learning with Language Models
• Data Poisoning for In-context Learning
• Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
• Learning to Poison Large Language Models for Downstream Manipulation