LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts

2025/10/22 by Siyuan Wang, Wang, Siyuan, Li Lyna Zhang +10
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.19363

openalex publication_date 2025/10/22 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

Reasoning over long contexts is essential for large language models. While reinforcement learning (RL) enhances short-context reasoning by inducing "Aha" moments in chain-of-thought, the advanced thinking patterns required for long-context reasoning remain largely unexplored, and high-difficulty RL data are scarce. In this paper, we introduce LoongRL, a data-driven RL method for advanced long-context reasoning. Central to LoongRL is KeyChain, a synthesis approach that transforms short multi-hop QA into high-difficulty long-context tasks by inserting UUID chains that hide the true question among large collections of distracting documents. Solving these tasks requires the model to trace the correct chain step-by-step, identify the true question, retrieve relevant facts and reason over them to answer correctly. RL training on KeyChain data induces an emergent plan-retrieve-reason-recheck reasoning pattern that generalizes far beyond training length. Models trained at 16K effectively solve 128K tasks without prohibitive full-length RL rollout costs. On Qwen2.5-7B and 14B, LoongRL substantially improves long-context multi-hop QA accuracy by +23.5% and +21.1% absolute gains. The resulting LoongRL-14B reaches a score of 74.2, rivaling much larger frontier models such as o3-mini (74.5) and DeepSeek-R1 (74.9). It also improves long-context retrieval, passes all 128K needle-in-a-haystack stress tests, and preserves short-context reasoning capabilities.

Citations

• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
• QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
• Qwen3 Technical Report
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example
• OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• LongRoPE2: Near-Lossless LLM Context Window Scaling
• SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models
• LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• MiniMax-01: Scaling Foundation Models with Lightning Attention
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Large Language Models Can Self-Improve in Long-context Reasoning
• ALR²: A Retrieve-then-Reason Framework for Long-context Question Answering
• HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
• Making Long-Context Language Models Better Multi-Hop Reasoners
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• Instruction-Following Evaluation for Large Language Models
• YaRN: Efficient Context Window Extension of Large Language Models
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Let's Verify Step by Step
• GPT-4 Technical Report
• MuSiQue: Multihop Questions via Single-hop Question Composition
• A Dataset of Information-Seeking Questions and Answers Anchored in\n Research Papers
• Measuring Mathematical Problem Solving With the MATH Dataset
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• Measuring Massive Multitask Language Understanding
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• The NarrativeQA Reading Comprehension Challenge
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning

Related

• QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
• QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management
• SPELL: Self-Play Reinforcement Learning for evolving Long-Context Language Models
• Adaptive Deep Reasoning: Triggering Deep Thinking When Needed
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models