SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

2024/10/09 by Heming Xia, Xia, Heming, Yongqi Li +7 · 17 citations
Computer Science · Physics and Astronomy · #Advanced Data Storage Technologies #Computation and Language (cs.CL) #FOS: Computer and information sciences #Magnetic confinement fusion research #Network Packet Processing and Optimization

paper · pdf · doi:10.48550/arxiv.2410.06916

openalex publication_date 2024/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Speculative decoding (SD) has emerged as a widely used paradigm to accelerate LLM inference without compromising quality. It works by first employing a compact model to draft multiple tokens efficiently and then using the target LLM to verify them in parallel. While this technique has achieved notable speedups, most existing approaches necessitate either additional parameters or extensive training to construct effective draft models, thereby restricting their applicability across different LLMs and tasks. To address this limitation, we explore a novel plug-and-play SD solution with layer-skipping, which skips intermediate layers of the target LLM as the compact draft model. Our analysis reveals that LLMs exhibit great potential for self-acceleration through layer sparsity and the task-specific nature of this sparsity. Building on these insights, we introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. SWIFT does not require auxiliary models or additional training, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Our extensive experiments across a wide range of models and downstream tasks demonstrate that SWIFT can achieve over a 1.3x-1.6x speedup while preserving the original distribution of the generated text. We release our code in https://github.com/hemingkx/SWIFT.

Cited by

• AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
• GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference
• SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
• Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding
• Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
• Beat the long tail: Distribution-Aware Speculative Decoding for RL Training
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs
• From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
• A Survey on Parallel Reasoning
• MixReasoning: Switching Modes to Think
• HiSpec: Hierarchical Speculative Decoding for LLMs
• Self-Speculative Biased Decoding for Faster Live Translation
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding
• ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
• SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
• Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

Related

• Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
• SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
• PLD+: Accelerating LLM inference by leveraging Language Model Artifacts
• Faster LLM Inference via Sequential Monte Carlo
• Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism