2024/05/29 by Yechen Xu, Xinhao Kong, Xu, Yechen +5 · 3 citations
Computer Science · Engineering · #Computation and Language (cs.CL) #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Industrial Automation and Control Systems #Machine Learning (cs.LG) #Modular Robots and Swarm Intelligence #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2406.00059
openalex publication_date 2024/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The complexity of large language model (LLM) serving workloads has substantially increased due to the integration with external tool invocations, such as ChatGPT plugins. In this paper, we identify a new opportunity for efficient LLM serving for requests that trigger tools: tool partial execution alongside LLM decoding. To this end, we design Conveyor, an efficient LLM serving system optimized for handling requests involving external tools. We introduce a novel interface for tool developers to expose partial execution opportunities to the LLM serving system and a request scheduler that facilitates partial tool execution. Our results demonstrate that tool partial execution can improve request completion latency by up to 38.8%.