2025/06/12 by Ziyi Zhang, Zhang, Ziyi, Ziheng Jiang +13 · 2 citations
Computer Science · #Algorithms and Data Compression #Coding theory and cryptography #Distributed #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Machine Learning (cs.LG) #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2506.11309
openalex publication_date 2025/06/12 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28
Low-latency decoding for large language models (LLMs) is crucial for applications like chatbots and code assistants, yet generating long outputs remains slow in single-query settings. Prior work on speculative decoding (which combines a small draft model with a larger target model) and tensor parallelism has each accelerated decoding. However, conventional approaches fail to apply both simultaneously due to imbalanced compute requirements (between draft and target models), KV-cache inconsistencies, and communication overheads under small-batch tensor-parallelism. This paper introduces SwiftSpec, a system that targets ultra-low latency for LLM decoding. SwiftSpec redesigns the speculative decoding pipeline in an asynchronous and disaggregated manner, so that each component can be scaled flexibly and remove draft overhead from the critical path. To realize this design, SwiftSpec proposes parallel tree generation, tree-aware KV cache management, and fused, latency-optimized kernels to overcome the challenges listed above. Across 5 model families and 6 datasets, SwiftSpec achieves an average of 1.75x speedup over state-of-the-art speculative decoding systems and, as a highlight, serves Llama3-70B at 348 tokens/s on 8 Nvidia Hopper GPUs, making it the fastest known system for low-latency LLM serving at this scale.