Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning

2025/10/29 by Deng, Yihe, Hsu, I-Hung, Yan, Jun +7 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.25992

Abstract

Large Language Models (LLMs) often struggle with problems that require multi-step reasoning. For small-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to overfit long demonstrations through rigid token-by-token imitation. To address this gap, we propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical "actions". SRL trains the model to generate an internal reasoning monologue before committing to each action. It provides smoother rewards based on the similarity between the model's actions and expert actions extracted from the SFT dataset in a step-wise manner. This supervision offers richer learning signals even when all rollouts are incorrect, while encouraging flexible reasoning guided by expert demonstrations. As a result, SRL enables small models to learn challenging problems previously unlearnable by SFT or RLVR. Moreover, initializing training with SRL before refining with RLVR yields the strongest overall performance. Beyond reasoning benchmarks, SRL generalizes effectively to agentic software engineering tasks, establishing it as a robust and versatile training framework for reasoning-oriented LLMs.

Citations

• Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning
• Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning
• A Survey on Large Language Models for Mathematical Reasoning
• Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
• SWE-smith: Scaling Data for Software Engineering Agents
• SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• ToolRL: Reward is All Tool Learning Needs
• A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
• GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
• GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
• Understanding R1-Zero-Like Training: A Critical Perspective
• Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
• Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
• Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
• Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
• Small Models Struggle to Learn from Strong Reasoners
• LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
• LIMO: Less is More for Reasoning
• Demystifying Long Chain-of-Thought Reasoning in LLMs
• On Teacher Hacking in Language Model Distillation
• s1: Simple test-time scaling
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Training Software Engineering Agents and Verifiers with SWE-Gym
• Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
• O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Qwen2.5-Coder Technical Report
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning
• CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation
• A Survey on Large Language Models for Code Generation
• Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents
• Persistent memory for AI coding agents: a pre-registered SWE-bench Verified benchmark
• Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
• Solving Quantitative Reasoning Problems with Language Models
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles

Cited by

• Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewards

Related

• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
• Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards
• Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning