SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models

2025/05/12 by Hang Wu, Wu, Hang, Jian Zhu +9 · 1 citation
Computer Science · #Big Data and Digital Economy #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Parallel #Software System Performance and Reliability #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2505.07680

openalex publication_date 2025/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) present a critical trade-off between inference quality and computational cost: larger models offer superior capabilities but incur significant latency, while smaller models are faster but less powerful. Existing serving strategies often employ fixed model scales or static two-stage speculative decoding, failing to dynamically adapt to the varying complexities of user requests or fluctuations in system performance. This paper introduces \systemname, a novel framework that reimagines LLM inference as an adaptive routing problem solved through multi-level speculative decoding. \systemname dynamically constructs and optimizes inference "paths" (chains of models) based on real-time feedback, addressing the limitations of static approaches. Our contributions are threefold: (1) An adaptive model chain scheduling mechanism that leverages performance profiling (execution times) and predictive similarity metrics (derived from token distribution divergence) to continuously select the optimal sequence of draft and verifier models, minimizing predicted latency per generated token. (2) A multi-level collaborative verification framework where intermediate models within the selected chain can validate speculative tokens, reducing the verification burden on the final, most powerful target model. (3) A synchronized state management system providing efficient, consistent KV cache handling across heterogeneous models in the chain, including precise, low-overhead rollbacks tailored for asynchronous batch processing inherent in multi-level speculation. Preliminary experiments demonstrate the validity of our method.

Citations

• FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
• SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
• Leveraging Uncertainty Estimation for Efficient LLM Routing
• Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference
• BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Caching
• A Theoretical Perspective for Speculative Decoding Algorithm
• GraphRouter: A Graph-based Router for LLM Selections
• Increasing transformer token length with a Maximum Entropy Principle Method
• PEARL: Parallel Speculative Decoding with Adaptive Draft Length
• The Llama 3 Herd of Models
• InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
• RouteLLM: Learning to Route LLMs with Preference Data
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
• Vidur: A Large-Scale Simulation Framework For LLM Inference
• A Survey on Efficient Inference for Large Language Models
• LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
• Characterization of Large Language Model Development in the Datacenter
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
• Cascade Speculative Drafting for Even Faster LLM Inference
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
• SGLang: Efficient Execution of Structured Language Model Programs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• LLaMA: Open and Efficient Foundation Language Models
• Scaling Laws for Generative Mixed-Modal Language Models
• Fast Inference from Transformers via Speculative Decoding
• Language Models are Multilingual Chain-of-Thought Reasoners
• PaLM: Scaling Language Modeling with Pathways
• Unified Scaling Laws for Routed Language Models
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• Scaling Laws for Neural Language Models
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Attention Is All You Need
• Clipper: A Low-Latency Online Prediction Serving System

Cited by

• SpecMemo: Speculative Decoding is in Your Pocket

Related

• SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding
• GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference
• FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
• Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
• Accelerating LLM Inference with Staged Speculative Decoding