One-shot Entropy Minimization

2025/05/26 by Gao, Zitian, Chen, Lynx, Luo, Haoming +2 · 1 voice · 15 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.20282

Abstract

We trained 13,440 large language models and found that entropy minimization requires only a single unlabeled data and 10 steps optimization to achieve performance improvements comparable to or even greater than those obtained using thousands of data and carefully designed rewards in rule-based reinforcement learning. This striking result may prompt a rethinking of post-training paradigms for large language models. Our code is avaliable at https://github.com/zitian-gao/one-shot-em.

Cited by

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
• Differential Set Selection via Confidence-Guided Entropy Minimization
• Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulation
• Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
• Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
• RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
• Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
• Spectral Logit Sculpting: Adaptive Low-Rank Logit Transformation for Controlled Text Generation
• Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
• Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
• Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
• Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
• Revisiting LLM Reasoning via Information Bottleneck

Discussions

• I've been thinking a lot about how statistical mechanics applies to Large Language Models. My aim is to minimise "hallucinations" (wrong guesses, basically) in generated code, and I've reframed this a [bsky, 7 points, 1 comments]

Related

• The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
• Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data
• EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
• Entropy Controllable Direct Preference Optimization
• Revisiting Entropy in Reinforcement Learning for Large Reasoning Models