SPIRe: Boosting LLM Inference Throughput with Speculative Decoding

2025/04/08 by Daniel Heinlein, Neelam, Sanjit, Heinlein, Daniel +6
Computer Science · #Advanced Data Storage Technologies #Boosting (machine learning) #Cache #Caching and Content Delivery #Context (archaeology) #Decoding methods #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques #Security token #Sequential decoding #Throughput

paper · pdf · doi:10.48550/arxiv.2504.06419

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing the cost per token requires decoding with large batch sizes. Recent work shows that SD can accelerate decoding with large batch sizes too if the context is sufficiently long and the draft model's KV cache is sparse. We introduce SPIRe, a draft model that combines static sparse attention, pruned initialization, and feedback memory to increase the modeled throughput of speculative decoding by over 100% compared to speculation with a much smaller draft model and by over 35% compared to the strong baseline of sparse self-speculation. Our approach is particularly effective when context lengths vary significantly across requests.

Citations

• MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
• GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• Efficient Streaming Language Models with Attention Sinks
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• Efficiently Scaling Transformer Inference
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Distilling the Knowledge in a Neural Network

Related

• Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
• Cascade Speculative Drafting for Even Faster LLM Inference
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
• Accelerating LLM Inference with Staged Speculative Decoding
• SSSD: Simply-Scalable Speculative Decoding