2023/10/28 by Qidong Su, Su, Qidong, Christina Giannoula +3 · 5 citations
Computer Science · #Algorithm #Artificial intelligence #Computer science #Computer security #Decoding methods #Finance #Inference #Key (lock) #Natural Language Processing Techniques #Parallel computing #Programming language #Security token #Speculation #Speech Recognition and Synthesis #State (computer science) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2310.18813
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/10/28 · openalex created_date 2023/11/01 · openalex updated_date 2026/07/28
Large Language Models (LLMs) like GPT are state-of-the-art text generation models that provide significant assistance in daily routines. However, LLM execution is inherently sequential, since they only produce one token at a time, thus incurring low hardware utilization on modern GPUs. Batching and speculative decoding are two techniques to improve GPU hardware utilization in LLM inference. To study their synergy, we implement a prototype implementation and perform an extensive characterization analysis on various LLM models and GPU architectures. We observe that the optimal speculation length depends on the batch size used. We analyze the key observation and build a quantitative model to explain it. Based on our analysis, we propose a new adaptive speculative decoding strategy that chooses the optimal speculation length for different batch sizes. Our evaluations show that our proposed method can achieve equal or better performance than the state-of-the-art speculation decoding schemes with fixed speculation length.