ToRL: Scaling Tool-Integrated RL

2025/03/30 by Xuefeng Li, Haoyang Zou, Li, Xuefeng +3 · 81 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Reinforcement Learning in Robotics #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2503.23383

openalex publication_date 2025/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce ToRL (Tool-Integrated Reinforcement Learning), a framework for training large language models (LLMs) to autonomously use computational tools via reinforcement learning. Unlike supervised fine-tuning, ToRL allows models to explore and discover optimal strategies for tool use. Experiments with Qwen2.5-Math models show significant improvements: ToRL-7B reaches 43.3% accuracy on AIME~24, surpassing reinforcement learning without tool integration by 14% and the best existing Tool-Integrated Reasoning (TIR) model by 17%. Further analysis reveals emergent behaviors such as strategic tool invocation, self-regulation of ineffective code, and dynamic adaptation between computational and analytical reasoning, all arising purely through reward-driven learning.

Cited by

• Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents
• Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents
• AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
• Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision
• Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solving
• Achieving Olympia-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learning
• Agentic Policy Optimization via Instruction-Policy Co-Evolution
• Rectifying LLM Thought from Lens of Optimization
• CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions
• ML-Tool-Bench: Tool-Augmented Planning for ML Tasks
• ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
• Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
• SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent
• Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Reinforcement Learning
• PORTool: Tool-Use LLM Training with Rewarded Tree
• On the Faithfulness of Visual Thinking: Measurement and Enhancement
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
• Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
• Demystifying Reinforcement Learning in Agentic Reasoning
• A²FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning
• A Survey on Agentic Multimodal Large Language Models
• Don't Just Fine-tune the Agent, Tune the Environment
• Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
• TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics
• Multi-Agent Tool-Integrated Policy Optimization
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
• TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture
• Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs
• Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• Tree Search for LLM Agent Reinforcement Learning
• Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions
• ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learning
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
• SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
• rStar2-Agent: Agentic Reasoning Technical Report
• Understanding Tool-Integrated Reasoning
• MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use
• Harnessing Rule-Based Reinforcement Learning for Enhanced Grammatical Error Correction
• G²RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
• Simple o3: Towards Interleaved Vision-Language Reasoning
• Careful Queries, Credible Results: Teaching RAG Models Advanced Web Search Tools with Reinforcement Learning
• Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
• PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
• RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning
• StepFun-Prover Preview: Let's Think and Verify Step by Step
• Agentic Reinforced Policy Optimization
• A Survey on Large Language Models for Mathematical Reasoning
• PrefixAgent: An LLM-Powered Design Framework for Efficient Prefix Adder Optimization
• ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL
• SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam?
• APPO: Agentic Procedural Policy Optimization
• Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning
• SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
• Distilling Tool Knowledge into Language Models via Back-Translated Traces
• Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
• Computational Thinking Reasoning in Large Language Models
• Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents
• Towards Effective Code-Integrated Reasoning
• R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning
• RM-R1: Reward Modeling as Reasoning
• Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers
• Progressive Agent Skill Generation via Reinforcement Learning
• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
• TCPO: Turn-Level Credit Policy Optimization
• Visual Agentic Reinforcement Fine-Tuning
• Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design
• Time-R1: Towards Comprehensive Temporal Reasoning in LLMs
• Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving
• SOD: Step-wise On-policy Distillation for Small Language Model Agents
• WebThinker: Empowering Large Reasoning Models with Deep Research Capability
• Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence
• Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning
• AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset
• ToolRL: Reward is All Tool Learning Needs
• Contextual Information Policy Optimization for Search Agents
• To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling

Related

• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
• AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
• Can Tool-Integrated Reinforcement Learning Generalize Across Diverse Domains?
• Acting Less is Reasoning More! Teaching Model to Act Efficiently