RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning

2025/10/03 by Arzhantsev, Aleksei, Sakhi, Otmane, Vasile, Flavian
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.02892

Abstract

Reinforcement learning (RL) is central to improving reasoning in large language models (LLMs) but typically requires ground-truth rewards. Test-Time Reinforcement Learning (TTRL) removes this need by using majority-vote rewards, but relies on heavy online RL and incurs substantial computational cost. We propose RoiRL: Reasoning with offline iterative Reinforcement Learning, a family of lightweight offline learning alternatives that can target the same regularized optimal policies. Unlike TTRL, RoiRL eliminates the need to maintain a reference model and instead optimizes weighted log-likelihood objectives, enabling stable training with significantly lower memory and compute requirements. Experimental results show that RoiRL trains to 2.5x faster and consistently outperforms TTRL on reasoning benchmarks, establishing a scalable path to self-improving LLMs without labels.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Solving Quantitative Reasoning Problems with Language Models
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Measuring Mathematical Problem Solving With the MATH Dataset
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• Reinforcement Learning: An Introduction

Related

• TTRL: Test-Time Reinforcement Learning
• SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
• You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models
• SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
• Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions