EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models

2025/05/31 by Zekun Wang, Wang, Zekun, Minghua Ma +10 · 2 citations
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2506.00479

openalex publication_date 2025/05/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success, yet their significant computational demands hinder practical deployment. While efforts to improve LVLM efficiency are growing, existing methods lack comprehensive evaluation across diverse backbones, benchmarks, and metrics. In this work, we systematically evaluate mainstream acceleration techniques for LVLMs, categorized into token and parameter compression. We introduce EffiVLM-Bench, a unified framework for assessing not only absolute performance but also generalization and loyalty, while exploring Pareto-optimal trade-offs. Our extensive experiments and in-depth analyses offer insights into optimal strategies for accelerating LVLMs. We open-source code and recipes for EffiVLM-Bench to foster future research.

Citations

• MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
• VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
• CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
• LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference
• SnapKV: LLM Knows What You are Looking for Before Generation
• DeepSeek-VL: Towards Real-World Vision-Language Understanding
• Honeybee: Locality-enhanced Projector for Multimodal LLM
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• Improved Baselines with Visual Instruction Tuning
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Efficient Streaming Language Models with Attention Sinks
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• MMBench: Is Your Multi-modal Model an All-around Player?
• Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• Hierarchical multimodal transformers for Multi-Page DocVQA
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Token Merging: Your ViT But Faster
• EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• Distilled Dual-Encoder Model for Vision-Language Understanding
• Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
• DocVQA: A Dataset for VQA on Document Images
• Towards VQA Models That Can Read
• A Diagram Is Worth A Dozen Images
• Microsoft COCO Captions: Data Collection and Evaluation Server

Cited by

• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
• A Survey of Token Compression for Efficient Multimodal Large Language Models

Related

• LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
• ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
• EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
• EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
• FastVLM: Efficient Vision Encoding for Vision Language Models