Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models

2023/05/24 by Jiashu Xu, Mingyu Derek Ma, Xu, Jiashu +7 · 1 voice · 22 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Security and Verification in Computing #Software Testing and Debugging Techniques #cs.AI #cs.CL #cs.CR #cs.LG

paper · pdf · doi:10.48550/arxiv.2305.14710

openalex publication_date 2023/05/24 · arxiv published 2023/05/24 · arxiv updated 2024/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We investigate security concerns of the emergent instruction tuning paradigm, that models are trained on crowdsourced datasets with task instructions to achieve superior performance. Our studies demonstrate that an attacker can inject backdoors by issuing very few malicious instructions (~1000 tokens) and control model behavior through data poisoning, without even the need to modify data instances or labels themselves. Through such instruction attacks, the attacker can achieve over 90% attack success rate across four commonly used NLP datasets. As an empirical study on instruction attacks, we systematically evaluated unique perspectives of instruction attacks, such as poison transfer where poisoned models can transfer to 15 diverse generative datasets in a zero-shot manner; instruction transfer where attackers can directly apply poisoned instruction on many other datasets; and poison resistance to continual finetuning. Lastly, we show that RLHF and clean demonstrations might mitigate such backdoors to some degree. These findings highlight the need for more robust defenses against poisoning attacks in instruction-tuning models and underscore the importance of ensuring data quality in instruction crowdsourcing.

Cited by

• Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Models
• On The Dangers of Poisoned LLMs In Security Automation
• Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption
• Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning
• Microsaccade-Inspired Probing: Positional Encoding Perturbations Reveal LLM Misbehaviours
• Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models
• bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs
• Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm
• A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
• Multi-Target Backdoor Attacks Against Speaker Recognition
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• The Dark Side of LLMs: Agent-based Attacks for Complete Computer Takeover
• TextCloak: Thwarting Unauthorized LLM Exploitation via RL-Driven Unlearnable Text
• ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks
• DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation
• Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation
• Daunce: Data Attribution through Uncertainty Estimation
• A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluation Methods
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
• Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!
• Sequential Data Poisoning in LLM Post-Training
• Exploring Backdoor Attack and Defense for LLM-empowered Recommendations

Discussions

• Small instruction sets to influence LLMs: arxiv.org/abs/2305.14710 [bsky, 0 points, 0 comments]

Related

• On the Exploitability of Instruction Tuning
• Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods
• A Study of Backdoors in Instruction Fine-tuned Language Models
• Watch Out for Your Guidance on Generation! Exploring Conditional Backdoor Attacks against Large Language Models
• Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges