RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs

2025/05/19 by Soumya Rani Samineni, Samineni, Soumya Rani, Durgesh Kalwar +7 · 3 voices · 5 citations
Health Professions · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Law #Concatenation (mathematics) #Context (archaeology) #FOS: Computer and information sciences #Higher Education Learning Practices #Interpreting and Communication in Healthcare #Iterative and incremental development #Iterative learning control #Machine Learning (cs.LG) #Narrative #Reinforcement learning #Set (abstract data type)

paper · pdf · doi:10.48550/arxiv.2505.13697

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Reinforcement learning based post-training of large language models (LLMs) has recently gained attention, particularly following the release of DeepSeek R1, which applied GRPO for fine-tuning. Amid the growing claims around improved reasoning abilities attributed to RL post-training, we critically examine the formulation and assumptions underlying these methods. We start by highlighting popular structural assumptions made in modeling LLM training as an MDP, and show how they lead to a degenerate MDP, that characterizes the problem as a contextual bandit, where RL updates naturally collapse into a form of on-policy variant of outcome-driven supervised learning. The two critical structural assumptions include (1) making the MDP states be just a concatenation of the actions with states becoming the context window and the actions becoming the tokens in LLMs and (2) splitting the reward of a state-action trajectory uniformly across the trajectory. Our comprehensive analysis demonstrates that, due to these simplifying assumptions, GRPO objective reduces to filtered Iterative SFT, an on-policy variant of supervised fine-tuning. Our experiments on benchmarks including GSM8K and Countdown, across a diverse set of model families show that Filtered Iterative SFT, incorporating both positive and negative samples, achieves performance comparable to GRPO-based training. We also show that these structural assumptions indirectly incentivize RL to generate longer sequences of intermediate tokens which in turn feeds into the narrative of "RL incentivizing thinking because it generates longer thinking traces."

Citations

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
• Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
• Geometric-Mean Policy Optimization
• Thought Anchors: Which LLM Reasoning Steps Matter?
• Spurious Rewards: Rethinking Training Signals in RLVR
• Thinkless: LLM Learns When to Think
• AdaptThink: Reasoning Models Can Learn When to Think
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
• GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• Concise Reasoning via Reinforcement Learning
• Understanding R1-Zero-Like Training: A Critical Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
• L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
• Training Language Models to Reason Efficiently
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
• Reinforced Self-Training (ReST) for Language Modeling
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• Chain of Thought Imitation with Procedure Cloning
• STaR: Bootstrapping Reasoning With Reasoning
• Training Verifiers to Solve Math Word Problems
• Proximal Policy Optimization Algorithms
• Trust Region Policy Optimization

Cited by

• Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
• Can GRPO Help LLMs Transcend Their Pretraining Origin?
• Tracing Uncertainty in Language Model "Reasoning"
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video

Discussions

• This RLiNo? paper (arxiv.org/abs/2505.13697) lead by Soumya Samineni and Durgesh_kalwar dives into the MDP model used in the RL post-training methods inspired by DeepSeek R1, and asks if some of the i [bsky, 4 points, 1 comments]
• RL in Name Only? Analyzing the Structural Assumptions in RL Post-Training [hn, 2 points, 0 comments]
• RL in Name Only? [hn, 1 points, 0 comments]

Related

• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• On Predictability of Reinforcement Learning Dynamics for Large Language Models
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
• A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce