Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Bases

2024/10/18 by Elias Lumer, Lumer, Elias, V. Subbiah +6 · 12 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2410.14594

openalex publication_date 2024/10/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in tool-equipped Agents (LLMs) have enabled complex tasks like secure database interactions and multi-agent code development. However, scaling tool capacity beyond agent reasoning or model limits remains a challenge. In this paper, we address these challenges by introducing Toolshed Knowledge Bases, a tool knowledge base (vector database) designed to store enhanced tool representations and optimize tool selection for large-scale tool-equipped Agents. Additionally, we propose Advanced RAG-Tool Fusion, a novel ensemble of tool-applied advanced retrieval-augmented generation (RAG) techniques across the pre-retrieval, intra-retrieval, and post-retrieval phases, without requiring model fine-tuning. During pre-retrieval, tool documents are enhanced with key information and stored in the Toolshed Knowledge Base. Intra-retrieval focuses on query planning and transformation to increase retrieval accuracy. Post-retrieval refines the retrieved tool documents and enables self-reflection. Furthermore, by varying both the total number of tools (tool-M) an Agent has access to and the tool selection threshold (top-k), we address trade-offs between retrieval accuracy, agent performance, and token cost. Our approach achieves 46%, 56%, and 47% absolute improvements on the ToolE single-tool, ToolE multi-tool and Seal-Tools benchmark datasets, respectively (Recall@5).

Cited by

• Solving Context Window Overflow in AI Agents
• Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems
• Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models
• Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems
• Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems
• ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering
• ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem
• Repairing Tool Calls Using Post-tool Execution Reflection and RAG
• Online-Optimized RAG for Tool Use and Function Calling
• AI Reasoning for Wireless Communications and Networking: A Survey and Perspectives
• A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges
• MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations

Related

• Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems
• Graph RAG-Tool Fusion
• Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval
• Tools are under-documented: Simple Document Expansion Boosts Tool Retrieval
• Tools Are Not Islands: Set-Level Tool Retrieval for LLM Agents via Query-Conditioned Hyperedge Prediction