2025/03/28 by A. K. Gautam, Gautam, Aayush, Susav Shrestha +3 · 2 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2504.00030
openalex publication_date 2025/03/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Speculative decoding accelerates large language model (LLM) inference by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, selecting an optimal speculation length is critical for maximizing speedup while minimizing wasted computation. We introduce GammaTune and GammaTune+, training-free adaptive algorithms that dynamically adjust speculation length based on token acceptance rates using a heuristic-based switching mechanism. Evaluated on SpecBench across multiple tasks and model pairs, our method outperforms other heuristic-based approaches and fixed-length speculative decoding, achieving an average speedup of 15% (±5%) with GammaTune and 16% (±3%) with GammaTune+, while reducing performance variance. This makes GammaTune a robust and efficient solution for real-world deployment.