Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment

2025/10/06 by Nevan Wichers, Wichers, Nevan, Aram Ebtekar +19 · 3 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Software Engineering Research #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.05024

openalex publication_date 2025/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models are sometimes trained with imperfect oversight signals, leading to undesired behaviors such as reward hacking and sycophancy. Improving oversight quality can be expensive or infeasible, motivating methods that improve learned behavior despite an imperfect training signal. We introduce Inoculation Prompting (IP), a simple but counterintuitive technique that prevents learning of an undesired behavior by modifying training prompts to explicitly request it. For example, to inoculate against reward hacking, we modify the prompts used in supervised fine-tuning to request code that only works on provided test cases but fails on other inputs. Across four settings we find that IP reduces the learning of undesired behavior without substantially reducing the learning of desired capabilities. We also show that prompts which more strongly elicit the undesired behavior prior to fine-tuning more effectively inoculate against the behavior when used during training; this serves as a heuristic to identify promising inoculation prompts. Overall, IP is a simple yet effective way to control how models generalize from fine-tuning, preventing learning of undesired behaviors without substantially disrupting desired capabilities.

Citations

• Recontextualization Mitigates Specification Gaming without Modifying the Specification
• Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
• Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training
• Persona Vectors: Monitoring and Controlling Character Traits in Language Models
• Safety Pretraining: Toward the Next Generation of Safe AI
• LookAhead Tuning: Safer Language Models via Partial Answer Previews
• Training large language models on narrow tasks can lead to broad misalignment
• Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
• Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
• A StrongREJECT for Empty Jailbreaks
• Explore Spurious Correlations at the Concept Level in Language Models for Text Classification
• Instruction-Following Evaluation for Large Language Models
• AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms
• Pretraining Language Models with Human Preferences
• Affective Coherence Monitoring for Transformer-Based Language Models
• A Holistic Approach to Undesired Content Detection in the Real World
• CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
• Training language models to follow instructions with human feedback
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
• Recursively Summarizing Books with Human Feedback
• Program Synthesis with Large Language Models
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Deep reinforcement learning from human preferences

Cited by

• Recontextualization Mitigates Specification Gaming without Modifying the Specification
• Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives
• Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
• Reinforcement Learning Towards Broadly and Persistently Beneficial Models
• In-Training Defenses against Emergent Misalignment in Language Models

Related

• Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
• Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors
• Natural Emergent Misalignment from Reward Hacking in Production RL
• Language Models in the Loop: Incorporating Prompting into Weak Supervision
• Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMs