Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models

2025/05/12 by Taiji Suzuki, Higuchi, Rei, Suzuki, Taiji · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.07558

openalex publication_date 2025/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Aligning large language models (LLMs) with human preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model. This assumption leads to statistical inconsistency, where more data doesn't guarantee convergence to true human preferences. To address this critical gap, we introduce a novel alignment method Direct Density Ratio Optimization (DDRO). DDRO directly estimates the density ratio between preferred and unpreferred output distributions, circumventing the need for explicit human preference modeling. We theoretically prove that DDRO is statistically consistent, ensuring convergence to the true preferred distribution as the data size grows, regardless of the underlying preference structure. Experiments demonstrate that DDRO achieves superior performance compared to existing methods on many major benchmarks. DDRO unlocks the potential for truly data-driven alignment, paving the way for more reliable and human-aligned LLMs.

Citations

• Self-Play Preference Optimization for Language Model Alignment
• Binary Classifier Optimization for Large Language Model Alignment
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Improving alignment of dialogue agents via targeted human judgements
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• PaLM: Scaling Language Modeling with Pathways
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Datasets: A Community Library for Natural Language Processing
• On the Dangers of Stochastic Parrots
• Measuring Massive Multitask Language Understanding
• Learning to summarize from human feedback
• Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation
• Artificial Intelligence, Values and Alignment
• Fine-Tuning Language Models from Human Preferences
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• Advances in prospect theory: Cumulative representation of uncertainty
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• UltraFeedback: Boosting Language Models with Scaled AI Feedback

Cited by

• Distillation of Large Language Models via Concrete Score Matching
• Preference Optimization by Estimating the Ratio of the Data Distribution

Related

• Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
• Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
• Less is More: Improving LLM Alignment via Preference Data Selection
• Preference Optimization by Estimating the Ratio of the Data Distribution
• Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization