Preference Models assume Proportional Hazards of Utilities

2025/08/15 by Nagpal, Chirag
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2508.13189

Abstract

Approaches for estimating preferences from human annotated data typically involves inducing a distribution over a ranked list of choices such as the Plackett-Luce model. Indeed, modern AI alignment tools such as Reward Modelling and Direct Preference Optimization are based on the statistical assumptions posed by the Plackett-Luce model. In this paper, I will connect the Plackett-Luce model to another classical and well known statistical model, the Cox Proportional Hazards model and attempt to shed some light on the implications of the connection therein.

Citations

• Preference Learning Algorithms Do Not Learn Preference Rankings
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training language models to follow instructions with human feedback
• Deep Survival Analysis

Related

• Strong Preferences Affect the Robustness of Preference Models and Value Alignment
• Learning Correlated Reward Models: Statistical Barriers and Opportunities
• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
• Preference Learning for AI Alignment: a Causal Perspective
• HPS: Hard Preference Sampling for Human Preference Alignment