Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

2024/06/13 by Xuan Zhang, Chao Du, Zhang, Xuan +9 · 26 citations
Business, Management and Accounting · Computer Science · #Business Process Modeling and Analysis #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies #Service-Oriented Architecture and Web Services

paper · pdf · doi:10.48550/arxiv.2406.09136

openalex publication_date 2024/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. Our code is available at https://github.com/sail-sg/CPO.

Cited by

• What Does Chain-of-Thought Contribute at Probe Time? Evidence for Local Co-Occurrence Activation
• Visual-Aware CoT: Achieving High-Fidelity Visual Consistency in Unified Models
• Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization
• Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python
• Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thought
• SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces
• Textual Self-attention Network: Test-Time Preference Optimization through Textual Gradient-based Attention
• LaRe: Latent Refocusing for Multimodal Reasoning
• Personalized Decision Modeling: Utility Optimization or Textualized-Symbolic Reasoning
• Inference-Time Chain-of-Thought Pruning with Latent Informativeness Signals
• Learning "Partner-Aware" Collaborators in Multi-Party Collaboration
• 3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models
• Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigm
• DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Models
• Reallocating Attention Across Layers to Reduce Multimodal Hallucination
• LOGicalThought: Logic-Based Ontological Grounding of LLMs for High-Assurance Reasoning
• DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision
• Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
• A-VERT: Agnostic Verification with Embedding Ranking Targets
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
• Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generation
• GPO: Learning from Critical Steps to Improve LLM Reasoning
• Let's Roleplay: Examining LLM Alignment in Collaborative Dialogues
• Toward Edge General Intelligence with Agentic AI and Agentification: Concepts, Technologies, and Future Directions
• Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS
• MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement

Related

• Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning
• On the Empirical Complexity of Reasoning and Planning in LLMs
• Chain-in-Tree: Back to Sequential Reasoning in LLM Tree Search
• ReCUT: Balancing Reasoning Length and Accuracy in LLMs via Stepwise Trails and Preference Optimization
• Chain or tree? Re-evaluating complex reasoning from the perspective of a matrix of thought