Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling

2025/08/30 by Shengyin Sun, Dafa Li, Sun, Shengyin +19 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Network Packet Processing and Optimization #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2509.04474

openalex publication_date 2025/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Test-time scaling has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs) by allocating additional computational resources during inference. However, this paradigm is inherently inefficient due to the generation of redundant and repetitive reasoning traces, leading to significant computational overhead. Speculative decoding offers a promising avenue for mitigating this inefficiency, yet its efficacy in the structured, repetition-rich context of test-time scaling remains largely unexplored. To bridge this gap, we introduce the first comprehensive benchmark designed to evaluate speculative decoding methods for accelerating LLM test-time scaling. Our benchmark provides consistent experimental protocols across representative test-time scaling paradigms (e.g., Best-of-N sampling and multi-round thinking), enabling a fair comparison of three major categories of speculative decoding: model-based, training-based, and n-gram-based methods. Extensive experiments reveal that simple n-gram-based methods effectively capture repetitive patterns, demonstrating unique potential in accelerating test-time scaling. This phenomenon demonstrates the value of integrating n-gram-based methods with model-based or training-based approaches to balance acceleration for both repetitive and diverse reasoning in test-time scaling. We hope this benchmark spurs further research on speculative decoding for test-time scaling, enabling faster and more practical reasoning in LLMs through better handling of repetitive and diverse reasoning paths.

Citations

• R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search
• TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
• ThinkSwitcher: When to Think Hard, When to Think Fast
• AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning
• Qwen3 Technical Report
• Reasoning Models Can Be Effective Without Thinking
• SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning
• A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
• Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging
• Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
• CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Model
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Reasoning Beyond Words ? Exploring framework for hidden state reasoning
• SAM Decoding: Speculative Decoding via Suffix Automaton
• SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
• Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• REST: Retrieval-Based Speculative Decoding
• Graph of Thoughts: Solving Elaborate Problems with Large Language Models
• Let's Verify Step by Step
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Self-Refine: Iterative Refinement with Self-Feedback
• Accelerating Large Language Model Decoding with Speculative Sampling
• Fast Inference from Transformers via Speculative Decoding
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training Verifiers to Solve Math Word Problems
• Measuring Mathematical Problem Solving With the MATH Dataset
• OpenAI o1 System Card

Cited by

• Towards Efficient Agents: A Co-Design of Inference Architecture and System
• Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding

Related

• Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
• Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspective
• Scaling Laws for Speculative Decoding
• The Art of Scaling Test-Time Compute for Large Language Models
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning