Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective

2025/06/01 by Lei, Lei, Jie Gu, Xiaolei Ma +8 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.01097

openalex publication_date 2025/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Existing Multimodal Large Language Models (MLLMs) process a large number of visual tokens, leading to significant computational costs and inefficiency. Instruction-related visual token compression demonstrates strong task relevance, which aligns well with MLLMs ultimate goal of instruction following. Previous works generally assume that visual tokens achieve better vision-language alignment in the shallow layers of LLMs, which have led to task-related token compression being primarily applied in intermediate LLM layers. In contrast, our study reveals that with proper selection, task-related token compression is feasible at the input stage of LLM with negligible performance loss. This new paradigm significantly reduces task-irrelevant visual tokens and its model-agnostic design enables application without modifying the LLM architecture. Specifically, we suggest that explainability methods for transformer-based architechtures can evaluate the global importance of each visual token with respect to the given instruction, which can effectively guide the task-related token compression for MLLMs. Furthermore, we propose to learn a mapping from the attention map of the first LLM layer to the explanation results, thereby avoiding the need for a full inference pass. Interestingly, this mapping can be learned using a simple and lightweight convolutional network, whose training is efficient and independent of MLLMs. Extensive experiments on 13 image and video benchmarks across three leading MLLMs (Qwen2-VL, LLaVA-OneVision, and VILA1.5) demonstrate the remarkable effectiveness and strong generalization of our approach. Additionally, our new compression paradigm achieves faster inference with reductions in both prefilling time and KV cache memory.

Citations

• TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models
• DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
• Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
• FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance
• PruneVid: Visual Token Pruning for Efficient Video Large Language Models
• Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM
• VisionZip: Longer is Better but Not Necessary in Vision Language Models
• NVILA: Efficient Frontier Visual Language Models
• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
• Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
• PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models
• TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval
• LLaVA-OneVision: Easy Visual Task Transfer
• VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
• InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
• Efficient Large Multi-modal Models via Visual Context Compression
• MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
• DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• Are We on the Right Way for Evaluating Large Vision-Language Models?
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
• MMBench: Is Your Multi-modal Model an All-around Player?
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• Evaluating Object Hallucination in Large Vision-Language Models
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Visual Instruction Tuning
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Token Merging: Your ViT But Faster
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions
• Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers
• Transformer Interpretability Beyond Attention Visualization
• Exploiting Behavioral Consistence for Universal User Representation
• DocVQA: A Dataset for VQA on Document Images
• Language Models are Few-Shot Learners
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Towards VQA Models That Can Read
• VizWiz Grand Challenge: Answering Visual Questions from Blind People
• Xception: Deep Learning with Depthwise Separable Convolutions
• LLaVA-Video: Video Instruction Tuning With Synthetic Data
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

Cited by

• CLIP-MG: Guiding Semantic Attention with Skeletal Pose Features and RGB Data for Micro-Gesture Recognition on the iMiGUE Dataset

Related

• TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models
• LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models
• AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
• [CLS] Token Tells Everything Needed for Training-free Efficient MLLMs
• Efficient Large Multi-modal Models via Visual Context Compression