Token Merging: Your ViT But Faster

2022/10/17 by Daniel Bolya, Bolya, Daniel, Cheng-Yang Fu +9 · 170 citations
Computer Science · #Generative Adversarial Networks and Image Synthesis #Advanced Neural Network Applications #Image Processing and 3D Reconstruction

paper · pdf · doi:10.48550/arxiv.2210.09461

Abstract

We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tuning on video. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop. Qualitatively, we find that ToMe merges object parts into one token, even over multiple frames of video. Overall, ToMe's accuracy and speed are competitive with state-of-the-art on images, video, and audio.

Cited by

• Revisiting [CLS] and Patch Token Interaction in Vision Transformers
• Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformers
• PathSelect: Sequential Token Selection for Whole Slide Pathology
• OmniCache: Multidimensional Hierarchical Feature Caching For Diffusion Models
• MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning
• Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs
• WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation
• Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
• Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
• PCA: Persistence-Aware Compression and Aggregation for Fast Video Large Language Models
• RoboMME-Interference: Benchmarking Robot Memory Under Interference
• Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspective
• FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
• HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformer
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
• IPCV: Information-Preserving Compression for MLLM Visual Encoders
• LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation
• Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models
• Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models
• Efficient Vision-Language Reasoning via Adaptive Token Pruning
• StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
• Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Models
• Rethinking Chain-of-Thought Reasoning for Videos
• Towards Lossless Ultimate Vision Token Compression for VLMs
• Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
• HybridToken-VLM: Hybrid Token Compression for Vision-Language Models
• All You Need Are Random Visual Tokens? Demystifying Token Pruning in VLLMs
• DFIR-DETR: Frequency Domain Enhancement and Dynamic Feature Aggregation for Cross-Scene Small Object Detection
• Multi-view Pyramid Transformer: Look Coarser to See Broader
• MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP
• Hierarchical geometric deep learning enables scalable analysis of molecular dynamics
• LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models
• ShaRP: SHAllow-LayeR Pruning for Video Large Language Models Acceleration
• LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
• UniComp: Rethinking Video Compression Through Informational Uniqueness
• AVGGT: Rethinking Global Attention for Accelerating VGGT
• VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm
• Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models
• FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention
• Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
• MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba
• InvarDiff: Cross-Scale Invariance Caching for Accelerated Diffusion Models
• Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering
• Energy-Efficient Vision Transformer Inference for Edge-AI Deployment
• Fast3Dcache: Training-free 3D Geometry Synthesis Acceleration
• Frequency-Aware Token Reduction for Efficient Vision Transformer
• HTTM: Head-wise Temporal Token Merging for Faster VGGT
• Towards an Effective Action-Region Tracking Framework for Fine-grained Video Action Recognition
• Object-Centric Vision Token Pruning for Vision Language Models
• Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation
• Foundry: Distilling 3D Foundation Models for the Edge
• Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulation
• Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
• EVCC: Enhanced Vision Transformer-ConvNeXt-CoAtNet Fusion for Classification
• AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
• TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
• A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
• ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generation
• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging
• TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing
• D³ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs
• Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning
• TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks
• How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?
• Let Me Show You: Learning by Retrieving from Egocentric Video for Robotic Manipulation
• Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
• Differentiable Hierarchical Visual Tokenization
• Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications
• Mitigating Semantic Collapse in Partially Relevant Video Retrieval
• OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
• OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs
• MedARC: Training-Free Adaptive Redundancy Compression of Visual Tokens for 3D Medical Vision-Language Models
• Dynamic Parameterization Is Not Dynamic Inference
• BATS: Resource-Efficient Volumetric Segmentation with Boundary-Aware Mixed-Resolution Tokens
• Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
• Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models
• Human Machine Social Hybrid Intelligence:A Collaborative Decision Making Framework for Large Model Agent Groups and Human Experts
• VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
• MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding
• Positional Preservation Embedding for Multimodal Large Language Models
• Transformers from Compressed Representations
• CARES: Context-Aware Resolution Selector for VLMs
• Socialized Learning and Emergent Behaviors in Multi-Agent Systems based on Multimodal Large Language Models
• StreamingTOM: Streaming Token Compression for Efficient Video Understanding
• Accelerating Vision Transformers with Adaptive Patch Sizes
• SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
• ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
• What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
• A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning
• ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution
• FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging with Diffusion Decoding
• ClustViT: Clustering-based Token Merging for Semantic Segmentation
• Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
• FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
• The silence of the weights: an investigation of structural pruning strategies for attention-based audio signal architectures
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory
• ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
• DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
• SVAC: Scaling Is All You Need For Referring Video Object Segmentation
• AutoPrune: Each Complexity Deserves a Pruning Policy
• Token Merging via Spatiotemporal Information Mining for Surgical Video Understanding
• GaussianVision: Vision-Language Alignment from Compressed Image Representations using 2D Gaussian Splatting
• CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones
• Revisiting Data Challenges of Computational Pathology: A Pack-based Multiple Instance Learning Training Framework
• Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
• Recursive transformers for semiconductor thermo-mechanical reliability
• Driving on Registers
• Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models
• Transformers and genome language models
• VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference
• PoRe: Position-Reweighted Visual Token Pruning for Vision Language Models
• Language-Guided Temporal Token Pruning for Efficient VideoLLM Processing
• Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models
• MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification
• Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformers
• Sequential Token Merging: Revisiting Hidden States
• Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perception
• Walk and Read Less: Improving the Efficiency of Vision-and-Language Navigation via Tuning-Free Multimodal Token Pruning
• Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
• ResidualViT for Efficient Temporally Dense Video Encoding
• CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
• The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
• ToMA: Token Merge with Attention for Diffusion Models
• Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge
• DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibration
• Adaptive Pareto-Optimal Token Merging for Edge Transformer Models in Semantic Communication
• SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models
• VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI
• Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models
• Detecting Regional Spurious Correlations in Vision Transformers via Token Discarding
• TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
• FastVGGT: Training-Free Acceleration of Visual Geometry Transformer
• Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers
• Efficient Large Language Models with Zero-Shot Adjustable Acceleration
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
• Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
• TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
• Representation Learning with Adaptive Superpixel Coding
• MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
• Towards Efficient Vision State Space Models via Token Merging
• Revisiting MLLM Token Technology through the Lens of Classical Visual Coding
• EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Models
• TTF-VLA: Temporal Token Fusion via Pixel-Attention Integration for Vision-Language-Action Models
• LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
• LET-US: Long Event-Text Understanding of Scenes
• Elastic Diffusion Transformer
• AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance
• Temporal Cluster Assignment for Efficient Real-Time Video Segmentation
• TCSAFormer: Efficient Vision Transformer with Token Compression and Sparse Attention for Medical Image Segmentation
• Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition
• Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation
• Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
• HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models
• Representation Shift: Unifying Token Compression with FlashAttention
• Frequency-Aware Autoregressive Modeling for Efficient High-Resolution Image Synthesis
• A Survey of Token Compression for Efficient Multimodal Large Language Models
• Modality Agnostic Efficient Long Range Encoder
• Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformers
• Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows
• A Hidden Stumbling Block in Generalized Category Discovery: Distracted Attention
• Training-free Token Reduction for Vision Mamba
• Variance-Based Pruning for Accelerating and Compressing Trained Networks
• Local Representative Token Guided Merging for Text-to-Image Generation
• Compact Vision Transformer by Reduction of Kernel Complexity
• Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
• ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inference
• FTCFormer: Fuzzy Token Clustering Transformer for Image Classification

Related

• Video, How Do Your Tokens Merge?
• Thicker and Quicker: A Jumbo Token for Fast Plain Vision Transformers
• Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging
• Multi-Scale And Token Mergence: Make Your ViT More Efficient