Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning

2025/04/25 by Shaokun Zhang, Zhang, Shaokun, Yi Dong +15 · 18 citations
Computer Science · #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.00024

openalex publication_date 2025/04/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.

Cited by

• SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL
• CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions
• ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
• CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic
• LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls
• One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning
• PORTool: Tool-Use LLM Training with Rewarded Tree
• Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents
• Polar: Agentic RL on Any Harness at Scale
• ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMs
• Automating Android Build Repair: Bridging the Reasoning-Execution Gap in LLM Agents with Domain-Specific Tools
• PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness
• Towards General Agentic Intelligence via Environment Scaling
• Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
• LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
• How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on τ-bench
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling

Related

• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
• Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch
• Can Tool-Integrated Reinforcement Learning Generalize Across Diverse Domains?
• ToolRL: Reward is All Tool Learning Needs
• AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning