Post-Completion Learning for Language Models

2025/07/27 by Fei, Xiang, Wang, Siqi, Wei, Shu +5
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.20252

Abstract

Current language model training paradigms typically terminate learning upon reaching the end-of-sequence () token, overlooking the potential learning opportunities in the post-completion space. We propose Post-Completion Learning (PCL), a novel training framework that systematically utilizes the sequence space after model output completion, to enhance both the reasoning and self-evaluation abilities. PCL enables models to continue generating self-assessments and reward predictions during training, while maintaining efficient inference by stopping at the completion point. To fully utilize this post-completion space, we design a white-box reinforcement learning method: let the model evaluate the output content according to the reward rules, then calculate and align the score with the reward functions for supervision. We implement dual-track SFT to optimize both reasoning and evaluation capabilities, and mixed it with RL training to achieve multi-objective hybrid optimization. Experimental results on different datasets and models demonstrate consistent improvements over traditional SFT and RL methods. Our method provides a new technical path for language model training that enhances output quality while preserving deployment efficiency.

Citations

• SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
• Spurious Rewards: Rethinking Training Signals in RLVR
• Qwen3 Technical Report
• The Lessons of Developing Process Reward Models in Mathematical Reasoning
• Self-Improvement in Language Models: The Sharpening Mechanism
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
• The Llama 3 Herd of Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Self-Refine: Iterative Refinement with Self-Feedback
• Solving math word problems with process- and outcome-based feedback
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• Finetuned Language Models Are Zero-Shot Learners
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
• Language Models are Few-Shot Learners
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• Prototypical Networks for Few-shot Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Matching Networks for One Shot Learning
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Towards a Unified View of Large Language Model Post-Training
• Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
• Reinforcement Mid-Training
• Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training