Probability-Consistent Preference Optimization for Enhanced LLM Reasoning

2025/05/29 by Yunqiao Yang, Houxing Ren, Yang, Yunqiao +15
Computer Science · #Computation and Language (cs.CL) #Consistency (knowledge bases) #Constraint Satisfaction and Optimization #Correctness #FOS: Computer and information sciences #Leverage (statistics) #Natural Language Processing Techniques #Pairwise comparison #Preference #Preference learning #Range (aeronautics) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.23540

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Recent advances in preference optimization have demonstrated significant potential for improving mathematical reasoning capabilities in large language models (LLMs). While current approaches leverage high-quality pairwise preference data through outcome-based criteria like answer correctness or consistency, they fundamentally neglect the internal logical coherence of responses. To overcome this, we propose Probability-Consistent Preference Optimization (PCPO), a novel framework that establishes dual quantitative metrics for preference selection: (1) surface-level answer correctness and (2) intrinsic token-level probability consistency across responses. Extensive experiments show that our PCPO consistently outperforms existing outcome-only criterion approaches across a diverse range of LLMs and benchmarks. Our code is publicly available at https://github.com/YunqiaoYang/PCPO.

Citations

• SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks
• LRHP: Learning Representations for Human Preferences via Preference Pairs
• Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• The Llama 3 Herd of Models
• Rho-1: Not All Tokens Are What You Need
• ORPO: Monolithic Preference Optimization without Reference Model
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Inverse Scaling: When Bigger Isn't Better
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• LLaMA: Open and Efficient Foundation Language Models
• Constitutional AI: Harmlessness from AI Feedback
• Large Language Models are Zero-Shot Reasoners
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• Measuring Mathematical Problem Solving With the MATH Dataset
• Learning to summarize from human feedback
• Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
• The Curious Case of Neural Text Degeneration
• Deep reinforcement learning from human preferences
• Attention Is All You Need
• RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment

Related

• Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
• Plug-and-Play Training Framework for Preference Optimization
• PORT: Preference Optimization on Reasoning Traces
• MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples
• MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning