SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression

2025/08/18 by Yuyang Xu, Xu, Yuyang, Cheng Yi +13
Business, Management and Accounting · Computer Science · #Artificial Intelligence (cs.AI) #Business Process Modeling and Analysis #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies #Service-Oriented Architecture and Web Services

paper · pdf · doi:10.48550/arxiv.2508.12604

openalex publication_date 2025/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Test-time scaling has proven effective in further enhancing the performance of pretrained Large Language Models (LLMs). However, mainstream post-training methods (i.e., reinforcement learning (RL) with chain-of-thought (CoT) reasoning) often incur substantial computational overhead due to auxiliary models and overthinking. In this paper, we empirically reveal that the incorrect answers partially stem from verbose reasoning processes lacking correct self-fix, where errors accumulate across multiple reasoning steps. To this end, we propose Self-traced Step-wise Preference Optimization (SSPO), a pluggable RL process supervision framework that enables fine-grained optimization of each reasoning step. Specifically, SSPO requires neither auxiliary models nor stepwise manual annotations. Instead, it leverages step-wise preference signals generated by the model itself to guide the optimization process for reasoning compression. Experiments demonstrate that the generated reasoning sequences from SSPO are both accurate and succinct, effectively mitigating overthinking behaviors without compromising model performance across diverse domains and languages.

Citations

• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
• Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards
• Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
• Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs
• Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning
• A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• VisualPRM: An Effective Process Reward Model for Multimodal Reasoning
• Process-based Self-Rewarding Language Models
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• PRAG: Paninian Retrieval-Augmented Generation for Safety-Critical Medical Question Answering
• Proximal Policy Optimization Algorithms
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation

Related

• Rectifying LLM Thought from Lens of Optimization
• PSPO*: An Effective Process-supervised Policy Optimization for Reasoning Alignment
• Patience Is The Key to Large Language Model Reasoning
• Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
• Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregation