Revisiting LLM Reasoning via Information Bottleneck

2025/07/24 by Lei, Shiye, Cheng, Zhihao, Jia, Kai +1 · 9 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.18391

Abstract

Large language models (LLMs) have recently demonstrated remarkable progress in reasoning capabilities through reinforcement learning with verifiable rewards (RLVR). By leveraging simple rule-based rewards, RL effectively incentivizes LLMs to produce extended chain-of-thought (CoT) reasoning trajectories, progressively guiding them toward correct answers. However, existing approaches remain largely heuristic and intuition-driven, limiting the development of principled methodologies. In this paper, we present a theoretical characterization of LLM reasoning grounded in information bottleneck (IB) principle, introducing IB-aware reasoning optimization (IBRO), a framework that encourages reasoning trajectories to be both informative about the final correct answer and generalizable across diverse prompts. We derive a practical token-level surrogate objective and propose an efficient approximation, resulting in the lightweight IB regularization method. This technique integrates seamlessly into existing RL-based post-training frameworks without additional computational overhead, requiring only a one-line code modification. Empirically, we validate IB regularization across multiple mathematical reasoning benchmarks and RL algorithms, demonstrating consistent improvements in LLM reasoning performance.

Citations

• Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
• Reasoning with Exploration: An Entropy Perspective
• Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
• Diversity-Aware Policy Optimization for Large Language Model Reasoning
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• Skywork Open Reasoner 1 Technical Report
• One-shot Entropy Minimization
• The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
• Memorization-Compression Cycles Improve Generalization
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Qwen2.5 Technical Report
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• How Does Information Bottleneck Help Deep Learning?
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Proximal Policy Optimization Algorithms
• Deep Learning and the Information Bottleneck Principle
• Measures of Entropy from Data Using Infinitely Divisible Kernels
• The information bottleneck method
• Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
• OpenAI o1 System Card

Cited by

• The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models
• ReLaX: Reasoning with Latent Exploration for Large Reasoning Models
• From generative AI to the brain: five takeaways
• Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization
• PEAR: Phase Entropy Aware Reward for Efficient Reasoning
• No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
• Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning
• Self-Aligned Reward: Towards Effective and Efficient Reasoners
• Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck

Related

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
• Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
• Rectifying LLM Thought from Lens of Optimization
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks