Speculative Speculative Decoding

2026/03/03 by Tanishq Kumar, Tri Dao, Avner May · 12 voices · 2 citations
#cs.LG

paper · pdf

Abstract

Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.

Citations

• Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
• Large Language Diffusion Models
• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Cited by

• Adversarial Prompts for Acceptance Collapse in Speculative Decoding
• Speculative Pipeline Decoding: Higher-Accuracy Drafting with Hidden Latency via Pipeline Parallelism

Discussions

• Speculative Speculative Decoding (SSD) [hn, 61 points, 9 comments]
• The authors reveal 'speculative speculative decoding' (SSD), an approach that enhances inference efficiency by parallelizing speculation and verification, yielding speed gains of up to 2x over existin [bsky, 1 points, 1 comments]
• Speculative Speculative Decoding (SSD) https://arxiv.org/abs/2603.03251 (https://news.ycombinator.com/item?id=47242637) [bsky, 0 points, 0 comments]
• 0:12 - Intro from YC Visiting Partner @FrancoisChauba1 3:49 - Tanishq Kumar (@tanishqkumar07) — Speculative Speculative Decoding (https://arxiv.org/abs/2603.03251) 18:33 - Guangyao (Stannis) Zhou (@zh [bsky, 0 points, 1 comments]
• Speculative Speculative Decoding (SSD) [bsky, 0 points, 0 comments]
• Speculative Speculative Decoding (SSD) #HackerNews https://arxiv.org/abs/2603.03251 [bsky, 0 points, 0 comments]
• Speculative Speculative Decoding (SSD) https://arxiv.org/abs/2603.03251 [bsky, 0 points, 0 comments]
• Speculative Speculative Decoding (SSD) https://arxiv.org/abs/2603.03251 [comments] [51 points] [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3mg7p3rt4sch2 [bsky, 0 points, 0 comments]
• Speculative Speculative Decoding (SSD) https:// arxiv.org/abs/2603.03251 # arxiv [mastodon, 0 points, 0 comments]
• Speculative Speculative Decoding (SSD) https:// arxiv.org/abs/2603.03251 # HackerNews # Speculative # Speculative # Decoding # SSD # AI # Research # MachineLearning # Arxiv [mastodon, 0 points, 0 comments]
• Speculative Speculative Decoding (SSD) https://arxiv.org/abs/2603.03251 (https://news.ycombinator.com/item?id=47242637) [bsky, 0 points, 0 comments]

Related

• Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
• Speeding up Speculative Decoding via Sequential Approximate Verification
• Decoding Speculative Decoding
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• Confidence-Modulated Speculative Decoding for Large Language Models