Efficient Large Multi-modal Models via Visual Context Compression

2024/06/28 by Jieneng Chen, Chen, Jieneng, Luoxin Ye +9 · 14 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Multimodal Machine Learning Applications #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.2406.20092

Abstract

While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments show that eliminating up to 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accuracy on the GQA benchmark, indicating significant redundancy in visual context. Addressing this, we introduce Visual Context Compressor, which reduces the number of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression on visual tokens while maintaining training efficiency, we develop LLaVolta as a light and staged training scheme that incorporates stage-wise visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding no loss of information when testing. Extensive experiments demonstrate that our approach enhances the performance of MLLMs in both image-language and video-language understanding, while also significantly cutting training costs and improving inference efficiency.

Cited by

• Visual Token Compression Enhances Robustness of MLLMs
• Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models
• EEG-VLM: A Hierarchical Vision-Language Model with Multi-Level Feature Alignment and Visually Enhanced Language-Guided Reasoning for EEG Image-Based Sleep Stage Prediction
• MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding
• Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
• Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
• Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models
• PoRe: Position-Reweighted Visual Token Pruning for Vision Language Models
• Sparse Training Scheme for Multimodal LLM
• CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
• VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understanding
• VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
• A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
• Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models

Related

• Generic Token Compression in Multimodal Large Language Models from an Explainability Perspective
• Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information
• FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
• VoCo-LLaMA: Towards Vision Compression with Large Language Models
• AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning