ToRL: Scaling Tool-Integrated RL

2025/03/30 by Li, Xuefeng, Zou, Haoyang, Liu, Pengfei · 51 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2503.23383

Abstract

We introduce ToRL (Tool-Integrated Reinforcement Learning), a framework for training large language models (LLMs) to autonomously use computational tools via reinforcement learning. Unlike supervised fine-tuning, ToRL allows models to explore and discover optimal strategies for tool use. Experiments with Qwen2.5-Math models show significant improvements: ToRL-7B reaches 43.3% accuracy on AIME~24, surpassing reinforcement learning without tool integration by 14% and the best existing Tool-Integrated Reasoning (TIR) model by 17%. Further analysis reveals emergent behaviors such as strategic tool invocation, self-regulation of ineffective code, and dynamic adaptation between computational and analytical reasoning, all arising purely through reward-driven learning.

Cited by

• Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents
• Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents
• AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
• Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision
• Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving
• Achieving Olympia-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learning
• Agentic Policy Optimization via Instruction-Policy Co-Evolution
• Rectifying LLM Thought from Lens of Optimization
• CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions
• ML-Tool-Bench: Tool-Augmented Planning for ML Tasks
• ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
• Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
• SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent
• Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Reinforcement Learning
• PORTool: Tool-Use LLM Training with Rewarded Tree
• On the Faithfulness of Visual Thinking: Measurement and Enhancement
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
• Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents
• Demystifying Reinforcement Learning in Agentic Reasoning
• A²FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning
• A Survey on Agentic Multimodal Large Language Models
• Don't Just Fine-tune the Agent, Tune the Environment
• Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
• TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics
• Multi-Agent Tool-Integrated Policy Optimization
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
• TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture
• Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs
• Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• Tree Search for LLM Agent Reinforcement Learning
• Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions
• ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learning
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
• SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
• rStar2-Agent: Agentic Reasoning Technical Report
• Understanding Tool-Integrated Reasoning
• MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use
• Harnessing Rule-Based Reinforcement Learning for Enhanced Grammatical Error Correction
• G²RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
• Simple o3: Towards Interleaved Vision-Language Reasoning
• Careful Queries, Credible Results: Teaching RAG Models Advanced Web Search Tools with Reinforcement Learning
• Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
• PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
• RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning
• StepFun-Prover Preview: Let's Think and Verify Step by Step
• Agentic Reinforced Policy Optimization

Related

• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
• AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
• Can Tool-Integrated Reinforcement Learning Generalize Across Diverse Domains?
• Acting Less is Reasoning More! Teaching Model to Act Efficiently