2024/01/12 by M. Liu, Liu, Mingdao, Aohan Zeng +9 · 11 citations
Computer Science · #Algorithm #Cache #Computation and Language (cs.CL) #Computer science #Computer security #Decoding methods #FOS: Computer and information sciences #Key (lock) #Latency (audio) #Natural Language Processing Techniques #Parallel computing #Real-time computing #Software deployment #Speech Recognition and Synthesis #Speedup #Telecommunications #Throughput #Topic Modeling #Wireless
paper · pdf · doi:10.48550/arxiv.2401.06761
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/01/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The massive adoption of large language models (LLMs) demands efficient deployment strategies. However, the auto-regressive decoding process, which is fundamental to how most LLMs generate text, poses challenges to achieve efficient serving. In this work, we introduce a parallel auto-regressive generation method. By instruct-tuning on general domain data that contains hierarchical structures, we enable LLMs to independently plan their generation process and perform auto-parallel auto-regressive (APAR) generation, significantly reducing the number of generation steps. APAR alone can achieve up to 2x speed-up, and when combined with speculative decoding, the speed-up can reach up to 4x. In addition, APAR reduces the key-value cache consumption and attention computation during generation. This leads to a throughput increase of 20-70% and a latency reduce of 20-35% in high-throughput scenarios, compared to state-of-the-art serving frameworks.