On Speculative Decoding for Multimodal Large Language Models

2024/04/13 by Mukul Gagrani, Raghavv Goel, Gagrani, Mukul +9 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2404.08856

openalex publication_date 2024/04/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37× using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.

Cited by

• Fast Inference of Visual Autoregressive Model with Adjacency-Adaptive Dynamical Draft Trees
• Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding
• HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
• ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
• SpecVLM: Fast Speculative Decoding in Vision-Language Models
• Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

Related

• Speculative Decoding Reimagined for Multimodal Large Language Models
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
• Decoding Speculative Decoding
• Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
• Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding