Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features

2025/04/01 by Jewon Lee, Ki-Ung Song, Lee, Jewon +14 · 3 citations
Computer Science · #Advanced Neural Network Applications #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2504.00557

Abstract

Visual token reduction lowers inference costs caused by extensive image features in large vision-language models (LVLMs). Unlike relevant studies that prune tokens in self-attention-only LVLMs, our work uniquely addresses cross-attention-based models, which achieve superior performance. We identify that the key-value (KV) cache size for image tokens in cross-attention layers significantly exceeds that of text tokens in self-attention layers, posing a major compute bottleneck. To mitigate this issue, we exploit the sparse nature in cross-attention maps to selectively prune redundant visual features. Our Trimmed Llama effectively reduces KV cache demands without requiring additional training. By benefiting from 50%-reduced visual features, our model can reduce inference latency and memory usage while achieving benchmark parity.

Cited by

• IF-VidCap: Can Video Caption Models Follow Instructions?
• ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
• Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spaces

Related

• STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference
• ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
• ST³: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
• CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference