Distributional Alignment Games for Answer-Level Fine-Tuning

2026/04/29 by Mehryar Mohri, Jon Schneider, Yifan Wu · 1 voice
Computer Science · #cs.GT #cs.LG

paper · pdf · doi:10.48550/arxiv.2604.27166

Abstract

We focus on the problem of Answer-Level Fine-Tuning (ALFT), where the goal is to optimize a language model based on the correctness or properties of its final answers, rather than the specific reasoning traces used to produce them. Directly optimizing answer-level objectives is computationally intractable due to the need to marginalize over the vast space of latent reasoning paths. To overcome this, we propose a general game-theoretical framework that lifts the problem to a Distributional Alignment Game. We formulate ALFT as a two-player game between a Policy (the generator) and a Target (an auxiliary distribution). We prove that the Nash Equilibrium of this game corresponds exactly to the solution of the original answer-level optimization problem. This variational perspective transforms the intractable marginalization problem into a tractable projection problem. We demonstrate that this framework unifies recent approaches to diversity and self-improvement (coherence) and provide efficient algorithms compatible with Group Relative Policy Optimization (GRPO), such as Coherence-GRPO, yielding significant complexity gains in mathematical reasoning tasks.

Citations

• Coherence Mechanisms for Provable Self-Improvement
• Jointly Reinforcing Diversity and Quality in Language Model Generations
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Solving math word problems with process- and outcome-based feedback
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization

Discussions

• ### Morning Intelligence: The Architectures of Reasoning The current research frontier has pivoted from mere "next-token prediction" to the intentional engineering of reasoning. We are seeing a struct [bsky, 1 points, 1 comments]

Related

• The Consensus Game: Language Model Generation via Equilibrium Search
• Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
• COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
• GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
• Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees