Decoding-based Regression

2025/01/31 by Xingyou Song, Dara Bahri, Song, Xingyou +1 · 2 voices · 8 citations
Computer Science · Mathematics · #Computer science #Decoding methods #Mathematics #Neural Networks and Applications #Regression #Regression analysis #Statistics

paper · pdf · doi:10.48550/arxiv.2501.19383

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/01/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Language models have recently been shown capable of performing regression wherein numeric predictions are represented as decoded strings. In this work, we provide theoretical grounds for this capability and furthermore investigate the utility of causal sequence decoding models as numeric regression heads given any feature representation. We find that, despite being trained in the usual way - for next-token prediction via cross-entropy loss - decoder-based heads are as performant as standard pointwise heads when benchmarked over standard regression tasks, while being flexible enough to capture smooth numeric distributions, such as in the task of density estimation.

Citations

• From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples
• Constitutional AI: Harmlessness from AI Feedback
• Attention Is All You Need
• Connectionist learning of belief networks
• On Estimation of a Probability Density Function and Mode
• Remarks on Some Nonparametric Estimates of a Density Function
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Cited by

• LASER: Language Model Regression for Semi-Structured Workflow Resource and Runtime Estimation
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• Regression Language Models for Code
• RELISH: LLM REgression with a Latent Iterative State Head
• Reasoning-Intensive Regression
• Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
• Always Tell Me The Odds: Fine-grained Conditional Probability Estimation
• The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?

Discussions

• Decoding-based Regression arxiv.org/abs/2501.19383 #statistics #regression #llms [bsky, 2 points, 0 comments]
• Paper can be read here: arxiv.org/pdf/2501.19383 [bsky, 0 points, 0 comments]

Related

• Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models
• Know Your Limits: Entropy Estimation Modeling for Compression and Generalization
• Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models
• Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference
• On Next-Token Prediction in LLMs: How End Goals Determine the Consistency of Decoding Algorithms