Structured In-context Environment Scaling for Large Language Model Reasoning

2025/09/27 by Yu Peng, Zeyuan Zhao, Yu, Peng +9
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2509.23330

openalex publication_date 2025/09/27 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have achieved significant advancements in reasoning capabilities through reinforcement learning (RL) via environmental exploration. As the intrinsic properties of the environment determine the abilities that LLMs can learn, the environment plays a important role in the RL finetuning process. An ideal LLM reasoning environment should possess three core characteristics: scalability, generalizable reasoning, and verifiability. However, existing mathematical and coding environments are difficult to scale due to heavy reliance on expert annotation, while the skills learned in game-based environments are too specialized to generalize. To bridge this gap, we introduce the Structured In-context Environment (SIE) framework. SIE achieves scalability by automatically constructing reasoning environments from large-scale structured data, where the rich compositional patterns naturally support generalizable reasoning. Moreover, the explicit schemas and reasoning chains in structured data provide a foundation for rule-based verifiability. Experimental results show that SIE framework not only achieves substantial improvements in in-domain structured reasoning, but also enables the learned compositional reasoning skills to generalize effectively to out-of-domain mathematical and logical reasoning tasks. We further explored learning in information-limited partial SIEs and found that LLMs can infer the missing information through exploring the environment, leading to robust reasoning improvements and generalization performance.

Citations

• Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoning
• Towards General Agentic Intelligence via Environment Scaling
• Group Sequence Policy Optimization
• Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Qwen3 Technical Report
• WebThinker: Empowering Large Reasoning Models with Deep Research Capability
• DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments
• MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
• Leveraging Dual Process Theory in Language Agent Framework for Real-time Simultaneous Human-AI Collaboration
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs
• On Memorization of Large Language Models in Logical Reasoning
• Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning
• The Llama 3 Herd of Models
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning
• Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
• Let's Verify Step by Step
• Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
• Training Verifiers to Solve Math Word Problems
• The Web as a Knowledge-base for Answering Complex Questions
• Proximal Policy Optimization Algorithms
• R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning
• OpenAI o1 System Card
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph

Related

• You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
• A Survey of Reinforcement Learning for Large Reasoning Models