The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives

2025/10/20 by Lim, Henry, Lim, Kwan Hui
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.17388

Abstract

Instruction-tuned large language models (IT-LLMs) exhibit strong zero-shot reasoning, yet their ability to execute simple, self-contained instructions remains underexplored, despite this being foundational to complex instruction-following. We evaluate 20 IT-LLMs on modified MMLU and MMLU-Pro benchmarks, by systematically varying the format of option labels (alphabetic, numeric, Roman) while keeping their meaning identical under four paradigms, namely: (1) With explicit instructions, label changes cause large performance shifts (e.g., -30.45% for Roman vs. numeric), revealing instruction-format bias. (2) Without instructions, performance drops further (up to -10.84%) and label sensitivity intensifies, underscoring the role of explicit guidance. (3) When option contents are removed, models fail random-choice baselines except with numeric labels, suggesting weak adherence to atomic directives. (4) Three-shot exemplars yield no significant gains in robustness or fidelity, and generation analyses show persistent label errors, especially for non-numeric formats. Across model sizes, larger LLMs achieve higher accuracy but remain inconsistent in instruction adherence. These results expose the insufficiencies of current instruction-tuning paradigms and highlight the need for evaluation methods and training strategies that explicitly target atomic instruction-following.

Citations

• StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following
• The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• A Thorough Examination of Decoding Methods in the Era of LLMs
• Large Language Models Are Zero-Shot Text Classifiers
• FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
• Instruction-following Evaluation through Verbalizer Manipulation
• Inverse Scaling: When Bigger Isn't Better
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• Large Language Models Are Human-Level Prompt Engineers
• Leveraging Large Language Models for Multiple Choice Question Answering
• Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
• Training language models to follow instructions with human feedback
• Reframing Instructional Prompts to GPTk's Language
• Finetuned Language Models Are Zero-Shot Learners
• Measuring Massive Multitask Language Understanding
• Language Models are Few-Shot Learners

Related

• Does Instruction Tuning Make LLMs More Consistent?
• Evaluating the Zero-shot Robustness of Instruction-tuned Language Models
• Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance
• Smaller Language Models Are Better Instruction Evolvers
• Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation