Speculative Decoding and Beyond: An In-Depth Survey of Techniques

2025/02/27 by Yi Hu, Zining Liu, Hu, Yunhai +8 · 9 citations
Computer Science · #Cellular Automata and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2502.19732

openalex publication_date 2025/02/27 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models, particularly for real-time applications. While traditional optimization approaches like pruning and quantization often compromise model quality, recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated. This survey presents a comprehensive taxonomy of generation-refinement frameworks, analyzing methods across autoregressive sequence tasks. We categorize methods based on their generation strategies (from simple n-gram prediction to sophisticated draft models) and refinement mechanisms (including single-pass verification and iterative approaches). Through systematic analysis of both algorithmic innovations and system-level implementations, we examine deployment strategies across computing environments and explore applications spanning text, images, and speech generation. This systematic examination of both theoretical frameworks and practical implementations provides a foundation for future research in efficient autoregressive decoding.

Cited by

• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
• Steering Pretrained Drafters during Speculative Decoding
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• Democratizing Agentic AI with Fast Test-Time Scaling on the Edge
• P/D-Device: Disaggregated Large Language Model between Cloud and Devices
• Draft-based Approximate Inference for LLMs
• Accelerated Test-Time Scaling with Model-Free Speculative Sampling
• DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
• KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization

Related

• Continuous Speculative Decoding for Autoregressive Image Generation
• A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
• A Theoretical Perspective for Speculative Decoding Algorithm
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Grouped Speculative Decoding for Autoregressive Image Generation