From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning

2023/09/30 by Xuansheng Wu, Wenlin Yao, Wu, Xuansheng +11 · 11 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.00492

openalex publication_date 2023/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have achieved remarkable success, where instruction tuning is the critical step in aligning LLMs with user intentions. In this work, we investigate how the instruction tuning adjusts pre-trained models with a focus on intrinsic changes. Specifically, we first develop several local and global explanation methods, including a gradient-based method for input-output attribution, and techniques for interpreting patterns and concepts in self-attention and feed-forward layers. The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models. This approach provides an internal perspective of the model shifts on a human-comprehensible level. Our findings reveal three significant impacts of instruction tuning: 1) It empowers LLMs to recognize the instruction parts of user prompts, and promotes the response generation constantly conditioned on the instructions. 2) It encourages the self-attention heads to capture more word-word relationships about instruction verbs. 3) It encourages the feed-forward networks to rotate their pre-trained knowledge toward user-oriented tasks. These insights contribute to a more comprehensive understanding of instruction tuning and lay the groundwork for future work that aims at explaining and optimizing LLMs for various applications. Our code and data are publicly available at https://github.com/JacksonWuxs/Interpret_Instruction_Tuning_LLMs.

Cited by

• What do vision-language models see in the context? Investigating multimodal in-context learning
• Flight Delay Prediction via Cross-Modality Adaptation of Large Language Models and Aircraft Trajectory Representation
• Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences
• Understanding the Effects of Domain Finetuning on LLMs
• Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling
• ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models
• Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
• How Instruction-Tuning Imparts Length Control: A Cross-Lingual Mechanistic Analysis
• Can Large Models Teach Student Models to Solve Mathematical Problems Like Human Beings? A Reasoning Distillation Method via Multi-LoRA Interaction
• Thinking Machines: Mathematical Reasoning in the Age of LLMs
• Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformation

Related

• Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering
• Instruction Tuning for Large Language Models: A Survey
• A Closer Look at the Limitations of Instruction Tuning
• Instruction Following by Boosting Attention of Large Language Models
• Revealing the Inherent Instructability of Pre-Trained Language Models