iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models

2024/12/09 by Lianyu Hu, Gao, Liqing, Hu, Lianyu +6 · 6 citations
Computer Science · Mathematics · #Anomaly Detection Techniques and Applications #Arithmetic #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #FOS: Computer and information sciences #Image (mathematics) #Mathematics #Multimodal Machine Learning Applications #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2412.06263

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/12/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent methods have made notable progress in accelerating Large Vision-Language Models (LVLMs) by exploiting the inherent redundancy in visual inputs. Most existing approaches, however, focus narrowly on reducing image tokens before or within the Large Language Model (LLM) stage to lower computational cost. This overlooks other major bottlenecks, particularly the image encoder, which itself requires substantial computation. As a result, these methods fall short of achieving true end-to-end acceleration. Importantly, the image encoder is the primary contributor of input tokens to the LLM. Thus, reducing visual redundancy at the encoder stage not only speeds up the encoder itself but also significantly lightens the workload for the subsequent LLM. Motivated by this, we investigate how to jointly optimize the image encoder and the LLM along with other LVLM components for comprehensive acceleration. To mitigate the risk of performance degradation from token reduction, we propose a novel token merging strategy that recycles useful information from otherwise discarded tokens. Our approach, iLLaVA, delivers consistent improvements across both image and video understanding tasks, achieving up to a 2 times throughput boost and a 4 times reduction in prefilling time. Notably, iLLaVA enables a larger model (e.g., InternVL-2.5 26B) to surpass a smaller counterpart (e.g., InternVL-2.5 8B) in both accuracy and efficiency. Extensive comparisons with state-of-the-art token pruning and merging techniques demonstrate the clear superiority of our method. Finally, we provide detailed visualizations for the merging steps of iLLaVA , offering deeper insights into how different LVLM components contribute to efficient computation.

Cited by

• PoRe: Position-Reweighted Visual Token Pruning for Vision Language Models
• VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
• METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
• A Survey of Token Compression for Efficient Multimodal Large Language Models
• Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality
• CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts
• Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

Related

• LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
• LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
• Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models