StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

2024/03/12 by Zhicheng Guo, Guo, Zhicheng, Sijie Cheng +15 · 28 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2403.07714

openalex publication_date 2024/03/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.

Cited by

• MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles
• From Word to World: Can Large Language Models be Implicit Text-based World Models?
• AppSelectBench: Application-Level Tool Selection Benchmark
• Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning
• ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset
• How Brittle is Agent Safety? Rethinking Agent Risk under Intent Concealment and Task Complexity
• MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
• PORTool: Tool-Use LLM Training with Rewarded Tree
• OrchDAG: Complex Tool Orchestration in Multi-Turn Interactions with Plan DAGs
• Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
• GOAT: A Training Framework for Goal-Oriented Agent with Tools
• ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memory
• COMPASS: Benchmarking Constrained Optimization in LLM Agents
• BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics
• PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Cases
• Ticket-Bench: A Kickoff for Multilingual and Regionalized Agent Evaluation
• From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
• Towards General Agentic Intelligence via Environment Scaling
• Mind Your Server: A Systematic Study of Parasitic Toolchain Attacks on the MCP Ecosystem
• MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
• ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution
• Kimi K2: Open Agentic Intelligence
• Benchmarking LLM Tool-Use in the Wild
• Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use
• NLI4VolVis: Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splatting
• SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

Related

• StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
• ToLeaP: Rethinking Development of Tool Learning with Large Language Models
• RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning
• MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models