Accelerating LLM Inference with Staged Speculative Decoding

2023/08/08 by Benjamin Spector, Chris Ré, Spector, Benjamin +1 · 38 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning and Algorithms #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2308.04623

openalex publication_date 2023/08/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances with large language models (LLM) illustrate their diverse capabilities. We propose a novel algorithm, staged speculative decoding, to accelerate LLM inference in small-batch, on-device scenarios. We address the low arithmetic intensity of small-batch inference by improving upon previous work in speculative decoding. First, we restructure the speculative batch as a tree, which reduces generation costs and increases the expected tokens per batch. Second, we add a second stage of speculative decoding. Taken together, we reduce single-batch decoding latency by 3.16x with a 762M parameter GPT-2-L model while perfectly preserving output quality.

Cited by

• LoPA: Scaling dLLM Inference via Lookahead Parallel Decoding
• Speculative Decoding Speed-of-Light: Optimal Lower Bounds via Branching Random Walks
• Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimization
• FlashMesh: Faster and Better Autoregressive Mesh Synthesis via Structured Speculation
• Steering Pretrained Drafters during Speculative Decoding
• Batch Speculative Decoding Done Right
• Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review
• Efficiently Executing High-throughput Lightweight LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management
• Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
• A Survey on Parallel Reasoning
• A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness
• Task-Aware Reduction for Scalable LLM-Database Systems
• Efficient LLM Inference over Heterogeneous Edge Networks with Speculative Decoding
• Staircase Streaming for Low-Latency Multi-Agent Inference
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
• Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction
• Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management
• MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness
• Uncovering Scaling Laws for Large Language Models via Inverse Problems
• Enabling MoE on the Edge via Importance-Driven Expert Scheduling
• Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
• Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
• ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling
• Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation
• Utility-Driven Speculative Decoding for Mixture-of-Experts
• S⁴C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models
• DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
• Reinforcement Speculative Decoding for Fast Ranking
• Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
• Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps
• FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge
• Automatic Task Detection and Heterogeneous LLM Speculative Decoding
• PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding
• RTP-LLM: High-Performance Alibaba LLM Inference Engine
• AutoJudge: Judge Decoding Without Manual Annotation
• Efficient Reasoning for LLMs through Speculative Chain-of-Thought
• SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding

Related

• The Synergy of Speculative Decoding and Batching in Serving Large Language Models
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
• From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
• Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
• Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models