Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models

2026/03/12 by Samy Jelassi, Mujin Kwun, Rosie Zhao +6 · 1 voice
Computer Science · #Feature (linguistics) #Language model #Matching (statistics) #Natural Language Processing Techniques #Perspective (graphical) #Prefix #Sampling (signal processing) #Scalability #Semantics (computer science) #Speech Recognition and Synthesis #Topic Modeling #cs.LG

paper · pdf · open access · doi:10.48550/arxiv.2603.12248

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/03/12 · arxiv published 2026/03/12 · openalex created_date 2026/03/14 · arxiv updated 2026/03/16 · openalex updated_date 2026/07/28

Abstract

Cross-entropy (CE) training provides dense and scalable supervision for language models, but it optimizes next-token prediction under teacher forcing rather than sequence-level behavior under model rollouts. We introduce a feature-matching objective for language-model fine-tuning that targets sequence-level statistics of the completion distribution, providing dense semantic feedback without requiring a task-specific verifier or preference model. To optimize this objective efficiently, we propose energy-based fine-tuning (EBFT), which uses strided block-parallel sampling to generate multiple rollouts from nested prefixes concurrently, batches feature extraction over these rollouts, and uses the resulting embeddings to perform an on-policy policy-gradient update. We present a theoretical perspective connecting EBFT to KL-regularized feature-matching and energy-based modeling. Empirically, across Q&A coding, unstructured coding, and translation, EBFT matches RLVR and outperforms SFT on downstream accuracy while achieving a lower validation cross-entropy than both methods.

Citations

• Escaping the Verifier: Learning to Reason via Demonstrations
• Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
• RLP: Reinforcement as a Pretraining Objective
• Reinforcement Learning on Pre-Training Data
• FlowRL: Matching Reward Distributions for LLM Reasoning
• Reinforcement Pre-Training
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
• OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
• The Llama 3 Herd of Models
• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Flow Matching for Generative Modeling
• Training language models to follow instructions with human feedback
• Program Synthesis with Large Language Models
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Calibration, Entropy Rates, and Memory in Language Models
• BERTScore: Evaluating Text Generation with BERT
• MTNT: A Testbed for Machine Translation of Noisy Text
• Classical Structured Prediction Losses for Sequence to Sequence Learning
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• Self-critical Sequence Training for Image Captioning
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Sequence Level Training with Recurrent Neural Networks
• Strictly Proper Scoring Rules, Prediction, and Estimation
• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Discussions

• New paper proposes Energy-Based Fine-Tuning: match sequence-level features, not tokens. It avoids verifiers, matches RLVR performance, beats SFT, and even lowers validation cross-entropy. A compelling [bsky, 1 points, 0 comments]

Related

• Preserving Diversity in Supervised Fine-Tuning of Large Language Models
• LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure
• The Coverage Principle: How Pre-Training Enables Post-Training
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models
• Fine-tuning of Language Models with Discriminator