Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity

2024/10/01 by Metel, Michael R., Lu, Peng, Chen, Boxing +2 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2410.01028

Abstract

We present a simple on the fly method for faster inference of large language models. Unlike other (self-)speculative decoding techniques, our method does not require fine-tuning or black-box optimization to generate a fixed draft model, relying instead on simple rules to generate varying draft models adapted to the input context. We show empirically that our light-weight algorithm is competitive with the current SOTA for self-speculative decoding, while being a truly plug-and-play method.

Cited by

• Polybasic Speculative Decoding Through a Theoretical Perspective
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding

Related

• Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
• AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs
• KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization