Adaptive Decoding via Latent Preference Optimization

2024/11/14 by Shehzaad Dhuliawala, Ilia Kulikov, Dhuliawala, Shehzaad +12 · 3 voices · 2 citations
Computer Science · #Neural Networks and Applications #cs.CL

paper · pdf · doi:10.48550/arxiv.2411.09661

openalex publication_date 2024/11/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

During language model decoding, it is known that using higher temperature sampling gives more creative responses, while lower temperatures are more factually accurate. However, such models are commonly applied to general instruction following, which involves both creative and fact seeking tasks, using a single fixed temperature across all examples and tokens. In this work, we introduce Adaptive Decoding, a layer added to the model to select the sampling temperature dynamically at inference time, at either the token or example level, in order to optimize performance. To learn its parameters we introduce Latent Preference Optimization (LPO) a general approach to train discrete latent variables such as choices of temperature. Our method outperforms all fixed decoding temperatures across a range of tasks that require different temperatures, including UltraFeedback, Creative Story Writing, and GSM8K.

Cited by

• Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVR
• On the Role of Temperature Sampling in Test-Time Scaling

Discussions

• 🚨 Adaptive Decoding via Latent Preference Optimization 🚨 - New layer for Transformer, selects decoding params automatically *per token* - Learnt via new method Latent Preference Optimization - Outpe [bsky, 43 points, 2 comments]
• Adaptive Decoding via Latent Preference Optimization: arxiv.org/abs/2411.09661 * Add a small MLP + classifier which predict a temperature per token * They train the MLP with a variant of DPO (arxiv. [bsky, 11 points, 0 comments]
• arxiv.org/abs/2411.09661 [bsky, 1 points, 0 comments]

Related

• Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models
• Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs
• Improving Open-Ended Text Generation via Adaptive Decoding
• Improving Diversity in Language Models: When Temperature Fails, Change the Loss
• KL-Divergence Guided Temperature Sampling