Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention

2025/10/03 by Xin Zou, Di Lu, Zou, Xin +13 · 6 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2510.02912

openalex publication_date 2025/10/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite their powerful capabilities, Multimodal Large Language Models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens. Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention or [CLS] attention to assess and discard redundant visual tokens. In this work, we identify a critical limitation of such attention-first pruning approaches, i.e., they tend to preserve semantically similar tokens, resulting in pronounced performance drops under high pruning ratios. To this end, we propose HoloV, a simple yet effective, plug-and-play visual token pruning framework for efficient inference. Distinct from previous attention-first schemes, HoloV rethinks token retention from a holistic perspective. By adaptively distributing the pruning budget across different spatial crops, HoloV ensures that the retained tokens capture the global visual context rather than isolated salient features. This strategy minimizes representational collapse and maintains task-relevant information even under aggressive pruning. Experimental results demonstrate that our HoloV achieves superior performance across various tasks, MLLM architectures, and pruning ratios compared to SOTA methods. For instance, LLaVA1.5 equipped with HoloV preserves 95.8% of the original performance after pruning 88.9% of visual tokens, achieving superior efficiency-accuracy trade-offs.

Citations

• DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
• Efficient Token Compression for Vision Transformer with Spatial Information Preserved
• Qwen2.5-VL Technical Report
• Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
• Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
• Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
• [CLS] Token Tells Everything Needed for Training-free Efficient MLLMs
• VisionZip: Longer is Better but Not Necessary in Vision Language Models
• Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
• freePruner: A Training-free Approach for Large Multimodal Model Acceleration
• Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy
• FoPru: Focal Pruning for Efficient Large Vision-Language Models
• FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
• Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model
• VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
• PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction
• ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
• Towards Interpreting Visual Information Processing in Vision-Language Models
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
• Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models
• HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
• Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models
• LLaVA-OneVision: Easy Visual Task Transfer
• The Llama 3 Herd of Models
• Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
• Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models
• DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
• The Evolution of Multimodal Model Architectures
• LOVA3: Learning to Visual Question Answering, Asking and Assessment
• Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning
• LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Model
• MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning
• Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models
• LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
• LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
• Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
• The Revolution of Multimodal Large Language Models: A Survey
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
• TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models
• Multimodal Large Language Models: A Survey
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
• Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Improved Baselines with Visual Instruction Tuning
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
• Improving Multimodal Datasets with Image Captioning
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• MMBench: Is Your Multi-modal Model an All-around Player?
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
• Generating Images with Multimodal Language Models
• Evaluating Object Hallucination in Large Vision-Language Models
• Visual Instruction Tuning
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models
• Token Merging: Your ViT But Faster
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• OPT: Open Pre-trained Transformer Language Models
• Flamingo: a Visual Language Model for Few-Shot Learning
• Training language models to follow instructions with human feedback
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks
• Transformer Feed-Forward Layers Are Key-Value Memories
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Evolving Normalization-Activation Layers
• Towards VQA Models That Can Read
• Attention Is All You Need
• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models
• Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs
• From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks
• Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

Cited by

• ShaRP: SHAllow-LayeR Pruning for Video Large Language Models Acceleration
• Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inference
• TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
• MoEGCL: Mixture of Ego-Graphs Contrastive Representation Learning for Multi-View Clustering
• Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
• Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models

Related

• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
• CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference
• Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See
• Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs
• VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization