Dynamic Depth Decoding: Faster Speculative Decoding for LLMs

2024/08/30 by O.R. Brown, Brown, Oscar, Zhengjie Wang +7 · 3 citations
Computer Science · #Advanced Data Storage Technologies #Algorithms and Data Compression #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2409.00142

openalex publication_date 2024/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by 44%, giving DDD an average speedup of 3.16x.

Cited by

• Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
• MC-SJD : Maximal Coupling Speculative Jacobi Decoding for Autoregressive Visual Generation Acceleration
• Grouped Speculative Decoding for Autoregressive Image Generation

Related

• GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
• Accelerating LLM Inference with Staged Speculative Decoding
• From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
• SSSD: Simply-Scalable Speculative Decoding
• Recurrent Drafter for Fast Speculative Decoding in Large Language Models