Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

2025/12/17 by Mathieu Blondel, Blondel, Mathieu, Michael E. Sander +7 · 1 voice
Computer Science · Mathematics · Medicine · #Artificial Intelligence in Healthcare and Education #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Topic Modeling #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2512.15605

openalex publication_date 2025/12/17 · arxiv published 2025/12/17 · openalex created_date 2025/12/19 · arxiv updated 2026/05/25 · openalex updated_date 2026/07/28

Abstract

Autoregressive models (ARMs) currently constitute the dominant paradigm for large language models (LLMs). Energy-based models (EBMs) represent another class of models, which have historically been less prevalent in LLM development, yet naturally characterize the optimal policy in post-training alignment. In this paper, we provide a unified view of these two model classes. Taking the chain rule of probability as a starting point, we establish an explicit bijection between ARMs and EBMs in function space, which we show to correspond to a special case of the soft Bellman equation in maximum entropy reinforcement learning. Building upon this bijection, we derive the equivalence between supervised learning of ARMs and EBMs. Furthermore, we analyze the distillation of EBMs into ARMs by providing theoretical error bounds. Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.

Citations

• ShiQ: Bringing back Bellman to LLMs
• Loss Functions and Operators Generated by f-Divergences
• Joint Learning of Energy-based Models and their Partition Function
• The Mystery of the Pathological Path-star Task for Language Models
• Transformers are Universal In-context Learners
• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
• Offline Regularised Reinforcement Learning for Large Language Models Alignment
• Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
• The Elements of Differentiable Programming
• The pitfalls of next-token prediction
• Formal Aspects of Language Modeling
• Generative Flow Networks as Entropy-Regularized RL
• Amortizing intractable inference in large language models
• Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
• Aligning Language Models with Preferences through f-divergence Minimization
• RL with KL penalties is better viewed as Bayesian inference
• Training language models to follow instructions with human feedback
• Trajectory balance: Improved credit assignment in GFlowNets
• GFlowNet Foundations
• Training Verifiers to Solve Math Word Problems
• Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
• How to Train Your Energy-Based Models
• Learning to summarize from human feedback
• Fine-Tuning Language Models from Human Preferences
• A Theory of Regularized Markov Decision Processes
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Differentiable Dynamic Programming for Structured Prediction and Attention
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A short variational proof of equivalence between policy gradients and soft Q learning
• Attention Is All You Need
• A unified view of entropy-regularized Markov decision processes
• Equivalence Between Policy Gradients and Soft Q-Learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Generative Adversarial Imitation Learning
• Dynamic Policy Programming

Discussions

• ..continuous time dynamics, every advance points toward the same insight: the most powerful architectures aren't the most complex ones. They're the ones that mirror how natural intelligence actually w [bsky, 0 points, 0 comments]

Related

• Your Autoregressive Generative Model Can be Better If You Treat It as an Energy-Based One
• Limitations of Autoregressive Models and Their Alternatives
• A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
• Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data
• A Markov Categorical Framework for Language Modeling