Stateful Inference for Low-Latency Multi-Agent Tool Calling

2026/05/25 by Victor Norgren · 1 voice · 1 citation
Computer Science · #cs.LG

paper · pdf

Abstract

Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-processing the entire conversation from scratch even though 85-95% of the prompt is unchanged from the previous turn. We present a stateful inference architecture that converts the O(n_t) per-turn cost of conventional serving into an O(Δ_t) delta-only cost: a persistent KV cache lives across turns and advances by ingesting only the new tokens, while a radix prefix cache extends this across interleaved multi-agent traffic and a prompt-lookup speculative decoder accelerates structured output. Against vLLM and SGLang on novel, fully-generated workloads, the reference implementation is 2.1× faster per turn on a 6-turn agentic workflow and 4.2× on the median turn of a 35-turn one, halving end-to-end wall time. The advantage comes from stateful reuse and speculation, not caching.

Citations

• Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• SGLang: Efficient Execution of Structured Language Model Programs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Fast Inference from Transformers via Speculative Decoding

Cited by

• Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path

Discussions

• Stateful Inference for Low-Latency Multi-Agent Tool Calling [hn, 2 points, 1 comments]

Related

• Optimizing Agentic Language Model Inference via Speculative Tool Calls
• Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
• ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
• Towards Efficient Agents: A Co-Design of Inference Architecture and System
• Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput