Optimizing Agentic Language Model Inference via Speculative Tool Calls

2025/12/17 by Nichols, Daniel, Singhania, Prajwal, Jekel, Charles +2
#Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Parallel #Performance (cs.PF) #Programming Languages (cs.PL) #Software Engineering (cs.SE) #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2512.15834

Abstract

Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files, run code, call APIs, etc. Further, modern reasoning-based LMs use tools such as web search and Python code execution to enhance their reasoning capabilities. While tools greatly improve the capabilities of LMs, they also introduce performance bottlenecks during the inference process. In this paper, we introduce novel systems optimizations to address such performance bottlenecks by speculating tool calls and forcing sequences to remain resident in the inference engine to minimize overheads. Our optimizations lead to throughput improvements of several hundred tokens per second when hosting inference for LM agents. We provide a theoretical analysis of our algorithms to provide insights into speculation configurations that will yield the best performance. Further, we recommend a new "tool cache" API endpoint to enable LM providers to easily adopt these optimizations.

Citations

• LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication
• Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning
• The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems
• xLAM: A Family of Large Action Models to Empower AI Agent Systems
• Large Language Model-Based Agents for Software Engineering: A Survey
• An LLM-Tool Compiler for Fused Parallel Function Calling
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
• SGLang: Efficient Execution of Structured Language Model Programs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Gorilla: Large Language Model Connected with Massive APIs
• Tool Learning with Foundation Models
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• ReAct: Synergizing Reasoning and Acting in Language Models

Related

• A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency
• Towards Efficient Agents: A Co-Design of Inference Architecture and System
• InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents
• ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Calling
• Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution