Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems

2025/11/03 by Elias Lumer, Faheem Nizar, Lumer, Elias +6 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2511.01854

openalex publication_date 2025/11/03 · openalex created_date 2025/11/06 · openalex updated_date 2026/07/28

Abstract

Recent advances in LLM Multi-Agent Systems enable scalable orchestration of sub-agents, each coordinating hundreds or thousands of tools or Model Context Protocol (MCP) servers. However, existing retrieval methods typically match queries against coarse agent-level descriptions before routing, which obscures fine-grained tool functionality and often results in suboptimal agent selection. We introduce Tool-to-Agent Retrieval, a unified framework that embeds both tools and their parent agents in a shared vector space and connects them through metadata relationships. By explicitly representing tool capabilities and traversing metadata to the agent level, Tool-to-Agent Retrieval enables granular tool-level or agent-level retrieval, ensuring that agents and their underlying tools or MCP servers are equally represented without the context dilution that arises from chunking many tools together. Evaluating Tool-to-Agent Retrieval across eight embedding models, our approach achieves consistent improvements of 19.4% in Recall@5 and 17.7% in nDCG@5 over previous state-of-the-art agent retrievers on the LiveMCPBench benchmark.

Citations

• LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?
• MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations
• MCP-Zero: Active Tool Discovery for Autonomous LLM Agents
• ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents
• ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario
• Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Bases
• Graph Retrieval-Augmented Generation: A Survey
• HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction
• ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
• Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval
• REAPER: Reasoning based Retrieval Planning for Complex RAG Systems
• Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark
• Planning and Editing What You Retrieve for Enhanced Tool Learning
• ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph
• AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls
• ProTIP: Progressive Tool Retrieval Improves Planning
• ControlLLM: Augment Language Models with Tools by Searching on Graphs
• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
• Counterfactually Auditable Lifecycle Certification for Autonomous Agents
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Query Rewriting for Retrieval-Augmented Large Language Models
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
• Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach

Cited by

• Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems
• Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models
• Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems

Related

• Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems
• Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Bases
• ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents
• Tools Are Not Islands: Set-Level Tool Retrieval for LLM Agents via Query-Conditioned Hyperedge Prediction
• MCP-Zero: Active Tool Discovery for Autonomous LLM Agents