Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

2025/05/29 by Gölz, Paul, Haghtalab, Nika, Yang, Kunhe · 6 citations
#Computer Science and Game Theory (cs.GT) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2505.23749

Abstract

After pre-training, large language models are aligned with human preferences based on pairwise comparisons. State-of-the-art alignment methods (such as PPO-based RLHF and DPO) are built on the assumption of aligning with a single preference model, despite being deployed in settings where users have diverse preferences. As a result, it is not even clear that these alignment methods produce models that satisfy users on average -- a minimal requirement for pluralistic alignment. Drawing on social choice theory and modeling users' comparisons through individual Bradley-Terry (BT) models, we introduce an alignment method's distortion: the worst-case ratio between the optimal achievable average utility, and the average utility of the learned policy. The notion of distortion helps draw sharp distinctions between alignment methods: Nash Learning from Human Feedback achieves the minimax optimal distortion of ((1)/(2) + o(1)) ⋅ β (for the BT temperature β), robustly across utility distributions, distributions of comparison pairs, and permissible KL divergences from the reference policy. RLHF and DPO, by contrast, suffer ≥ (1 - o(1)) ⋅ β distortion already without a KL constraint, and e^Ω(β) or even unbounded distortion in the full setting, depending on how comparison pairs are sampled.

Cited by

• Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior
• Human-AI Collaboration with Misaligned Preferences
• The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity
• Optimized Distortion in Linear Social Choice
• Eliciting Truthful Feedback for Preference-Based Learning via the VCG Mechanism
• Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Related

• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• Beyond RLHF: A Unified Theoretical Framework of Alignment
• Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
• Human Alignment of Large Language Models through Online Preference Optimisation
• HPS: Hard Preference Sampling for Human Preference Alignment