On the Exploitability of Instruction Tuning

2023/06/28 by Manli Shu, Shu, Manli, Jiongxiao Wang +9 · 25 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Adversary #Benchmark (surveying) #Code (set theory) #Computation and Language (cs.CL) #Computer science #Computer security #Cryptography and Security (cs.CR) #Engineering #Exploit #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Oracle #Pipeline (software) #Programming language #Quality (philosophy) #Scheme (mathematics) #Software engineering #Topic Modeling #Work (physics)

paper · pdf · doi:10.48550/arxiv.2306.17194

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/06/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Instruction tuning is an effective technique to align large language models (LLMs) with human intents. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose AutoPoison, an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs. Code is available at \urlhttps://github.com/azshue/AutoPoison.

Cited by

• ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior
• AutoBackdoor: Automating Backdoor Attacks via LLM Agents
• Importance-Aware Data Selection for Efficient LLM Instruction Tuning
• LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data
• Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation
• Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
• Fewer Weights, More Problems: A Practical Attack on LLM Pruning
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain
• FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systems
• Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
• Lethe: Purifying Backdoored Large Language Models with Knowledge Dilution
• AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection
• SDD: Self-Degraded Defense against Malicious Fine-tuning
• BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profit
• Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
• Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMs
• Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training
• A Systematic Review of Poisoning Attacks Against Large Language Models
• A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluation Methods
• Security Concerns for Large Language Models: A Survey
• Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning
• Security and Privacy Challenges of Large Language Models: A Survey
• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction
• Sequential Data Poisoning in LLM Post-Training
• Security in the Fine‐Tuning Lifecycle of Large Language Models: Threats, Defenses, Evaluation, and Future Directions

Related

• Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models
• Poisoning Language Models During Instruction Tuning
• Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
• DRIP: Defending Prompt Injection via Token-wise Representation Editing and Residual Instruction Fusion
• Defense Against Prompt Injection Attack by Leveraging Attack Techniques