Token Merging: Your ViT But Faster

2022/10/17 by Daniel Bolya, Bolya, Daniel, Cheng-Yang Fu +9 · 1 voice · 279 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Artificial intelligence #Computer network #Computer science #Computer vision #Engineering #Generative Adversarial Networks and Image Synthesis #Image Processing and 3D Reconstruction #Operating system #Real-time computing #Security token #Throughput #Transformer #Wireless #cs.CV

paper · pdf · doi:10.48550/arxiv.2210.09461

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2022/10/17 · arxiv published 2022/10/17 · arxiv updated 2023/03/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tuning on video. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop. Qualitatively, we find that ToMe merges object parts into one token, even over multiple frames of video. Overall, ToMe's accuracy and speed are competitive with state-of-the-art on images, video, and audio.

Cited by

• Revisiting [CLS] and Patch Token Interaction in Vision Transformers
• Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformers
• PathSelect: Sequential Token Selection for Whole Slide Pathology
• OmniCache: Multidimensional Hierarchical Feature Caching For Diffusion Models
• MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning
• Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs
• WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation
• Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
• Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
• PCA: Persistence-Aware Compression and Aggregation for Fast Video Large Language Models
• RoboMME-Interference: Benchmarking Robot Memory Under Interference
• Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspective
• FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
• HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformer
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
• IPCV: Information-Preserving Compression for MLLM Visual Encoders
• LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation
• Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models
• Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models
• Efficient Vision-Language Reasoning via Adaptive Token Pruning
• StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
• Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Models
• Rethinking Chain-of-Thought Reasoning for Videos
• Towards Lossless Ultimate Vision Token Compression for VLMs
• Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
• HybridToken-VLM: Hybrid Token Compression for Vision-Language Models
• All You Need Are Random Visual Tokens? Demystifying Token Pruning in VLLMs
• DFIR-DETR: Frequency Domain Enhancement and Dynamic Feature Aggregation for Cross-Scene Small Object Detection
• Multi-view Pyramid Transformer: Look Coarser to See Broader
• MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP
• Hierarchical geometric deep learning enables scalable analysis of molecular dynamics
• LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models
• ShaRP: SHAllow-LayeR Pruning for Video Large Language Models Acceleration
• LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
• UniComp: Rethinking Video Compression Through Informational Uniqueness
• AVGGT: Rethinking Global Attention for Accelerating VGGT
• VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm
• Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models
• FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention
• Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
• MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba
• InvarDiff: Cross-Scale Invariance Caching for Accelerated Diffusion Models
• Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering
• Energy-Efficient Vision Transformer Inference for Edge-AI Deployment
• Fast3Dcache: Training-free 3D Geometry Synthesis Acceleration
• Frequency-Aware Token Reduction for Efficient Vision Transformer
• HTTM: Head-wise Temporal Token Merging for Faster VGGT
• Towards an Effective Action-Region Tracking Framework for Fine-grained Video Action Recognition
• Object-Centric Vision Token Pruning for Vision Language Models
• Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation
• Foundry: Distilling 3D Foundation Models for the Edge
• Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulation
• Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
• EVCC: Enhanced Vision Transformer-ConvNeXt-CoAtNet Fusion for Classification
• AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
• TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
• A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
• ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generation
• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging
• TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing
• D³ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs
• Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning
• TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks
• How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?
• Let Me Show You: Learning by Retrieving from Egocentric Video for Robotic Manipulation
• Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
• Differentiable Hierarchical Visual Tokenization
• Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications
• Mitigating Semantic Collapse in Partially Relevant Video Retrieval
• OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
• OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs
• MedARC: Training-Free Adaptive Redundancy Compression of Visual Tokens for 3D Medical Vision-Language Models
• Dynamic Parameterization Is Not Dynamic Inference
• BATS: Resource-Efficient Volumetric Segmentation with Boundary-Aware Mixed-Resolution Tokens
• Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
• Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models
• Human Machine Social Hybrid Intelligence:A Collaborative Decision Making Framework for Large Model Agent Groups and Human Experts
• VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion
• MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding
• Positional Preservation Embedding for Multimodal Large Language Models
• Transformers from Compressed Representations
• CARES: Context-Aware Resolution Selector for VLMs
• Socialized Learning and Emergent Behaviors in Multi-Agent Systems based on Multimodal Large Language Models
• StreamingTOM: Streaming Token Compression for Efficient Video Understanding
• Accelerating Vision Transformers with Adaptive Patch Sizes
• SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
• ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
• What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
• A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning
• ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution
• FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging with Diffusion Decoding
• ClustViT: Clustering-based Token Merging for Semantic Segmentation
• Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
• FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
• The silence of the weights: an investigation of structural pruning strategies for attention-based audio signal architectures
• StreamForest: Efficient Online Video Understanding with Persistent Event Memory
• ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
• DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
• SVAC: Scaling Is All You Need For Referring Video Object Segmentation
• AutoPrune: Each Complexity Deserves a Pruning Policy
• Token Merging via Spatiotemporal Information Mining for Surgical Video Understanding
• GaussianVision: Vision-Language Alignment from Compressed Image Representations using 2D Gaussian Splatting
• CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones
• Revisiting Data Challenges of Computational Pathology: A Pack-based Multiple Instance Learning Training Framework
• Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
• Recursive transformers for semiconductor thermo-mechanical reliability
• Driving on Registers
• Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models
• Transformers and genome language models
• VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference
• PoRe: Position-Reweighted Visual Token Pruning for Vision Language Models
• Language-Guided Temporal Token Pruning for Efficient VideoLLM Processing
• Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models
• MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification
• Evict3R: Training-Free Token Eviction for Memory-Bounded Streaming Visual Geometry Transformers
• Sequential Token Merging: Revisiting Hidden States
• Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perception
• Walk and Read Less: Improving the Efficiency of Vision-and-Language Navigation via Tuning-Free Multimodal Token Pruning
• Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
• ResidualViT for Efficient Temporally Dense Video Encoding
• CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
• The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
• ToMA: Token Merge with Attention for Diffusion Models
• Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge
• DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibration
• Adaptive Pareto-Optimal Token Merging for Edge Transformer Models in Semantic Communication
• SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models
• VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI
• Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models
• Detecting Regional Spurious Correlations in Vision Transformers via Token Discarding
• TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
• FastVGGT: Training-Free Acceleration of Visual Geometry Transformer
• Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers
• Efficient Large Language Models with Zero-Shot Adjustable Acceleration
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
• Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
• TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
• Representation Learning with Adaptive Superpixel Coding
• MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
• Towards Efficient Vision State Space Models via Token Merging
• Revisiting MLLM Token Technology through the Lens of Classical Visual Coding
• EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Models
• TTF-VLA: Temporal Token Fusion via Pixel-Attention Integration for Vision-Language-Action Models
• LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
• LET-US: Long Event-Text Understanding of Scenes
• Elastic Diffusion Transformer
• AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance
• Temporal Cluster Assignment for Efficient Real-Time Video Segmentation
• TCSAFormer: Efficient Vision Transformer with Token Compression and Sparse Attention for Medical Image Segmentation
• Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition
• Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation
• Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Models
• HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models
• Representation Shift: Unifying Token Compression with FlashAttention
• Frequency-Aware Autoregressive Modeling for Efficient High-Resolution Image Synthesis
• A Survey of Token Compression for Efficient Multimodal Large Language Models
• Modality Agnostic Efficient Long Range Encoder
• Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformers
• Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows
• A Hidden Stumbling Block in Generalized Category Discovery: Distracted Attention
• Training-free Token Reduction for Vision Mamba
• Variance-Based Pruning for Accelerating and Compressing Trained Networks
• Local Representative Token Guided Merging for Text-to-Image Generation
• Compact Vision Transformer by Reduction of Kernel Complexity
• Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
• Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation
• SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping
• ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inference
• FTCFormer: Fuzzy Token Clustering Transformer for Image Classification
• QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models
• Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
• Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding
• Upsample What Matters: Region-Adaptive Latent Sampling for Accelerated Diffusion Transformers
• Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs
• AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
• SPoT: Subpixel Placement of Tokens in Vision Transformers
• Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
• OptiPrune: Boosting Prompt-Image Consistency with Attention-Guided Noise and Dynamic Token Selection
• Diversity-Guided MLP Reduction for Efficient Large Vision Transformers
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment
• LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
• Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
• ToSA: Token Merging with Spatial Awareness
• Déjà Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reuse
• LHM++: An Efficient Large Human Reconstruction Model for Pose-free Images to 3D
• GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
• CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
• Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformers
• Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings
• Video, How Do Your Tokens Merge?
• DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
• METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding
• Image Recognition with Online Lightweight Vision Transformer: A Survey
• Sparse Imagination for Efficient Visual World Model Planning
• Towards Predicting Any Human Trajectory In Context
• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective
• Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs
• EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
• Towards Unified Neural Decoding with Brain Functional Network Modeling
• The Surprising Soupability of Documents in State Space Models
• One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory
• QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining
• CF-DETR: Coarse-to-Fine Transformer for Real-Time Object Detection
• VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
• RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers
• Sparsified State-Space Models are Efficient Highway Networks
• HoliTom: Holistic Token Merging for Fast Video Large Language Models
• FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
• FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation
• Shifting AI Efficiency From Model-Centric to Data-Centric Compression
• ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
• Neural Coding Is Not Always Semantic: Towards the Standardized Coding Workflow in Semantic Communications
• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
• Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality
• HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams
• CHAOS: Chart Analysis with Outlier Samples
• Native Segmentation Vision Transformers
• Ranked Entropy Minimization for Continual Test-Time Adaptation
• Exploring The Visual Feature Space for Multimodal Neural Decoding
• DiffPrune: differentiable information throttling for token pruning in vision-language models
• Dynamic Resolution Routing for Efficient Egocentric Grounding
• Allocation Before Ranking: Decoupled Token Compression for OmniLLMs
• CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
• Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition
• Lossless Token Merging Even Without Fine-Tuning in Vision Transformers
• Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere
• Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models
• DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computation
• Think in Sets for Streaming Video Token Compression
• STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference
• UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction
• FreqNav: Stage-Wise Frequency Routing for Object-Oriented Aerial Vision-Language Navigation
• Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration
• A probabilistic framework for dynamic quantization
• Why 1 + 1 < 1 in Visual Token Pruning: Beyond Naive Integration via Multi-Objective Balanced Covering
• ORCA: ORgan-Centroid Aggregation for Training-Free 3D CT Visual Token Compression
• Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference
• Beyond Consistency: Preserving Temporal Structure in Zero-Shot Video Editing
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
• AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference
• Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
• 2K Retrofit: Entropy-Guided Efficient Sparse Refinement for High-Resolution 3D Geometry Prediction
• FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers
• Video-Based Reward Modeling for Computer-Use Agents
• WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer
• Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answering
• Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction
• ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation
• VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuning
• Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning
• GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
• Semantic Event Graphs for Long-Form Video Question Answering
• Coverage Matters: MarginMerge for Compressing Multi-Vector Visual Document Retrievers
• Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
• OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models
• Token Sequence Compression for Efficient Multimodal Computing
• TimeSoccer: An End-to-End Multimodal Large Language Model for Soccer Commentary Generation
• DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
• Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles
• EvtGraph: Event-Adaptive Compression for Sparse Temporal Graph Learning in Multimodal Time Series
• CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
• DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models
• HiSC: Hierarchical Spatial Clustering Token Compression for Efficient 3D Scene Understanding
• RUTA: Principled Visual Token Allocation via Rate-Utility Optimization
• Rethinking Vision Transformer for Large-Scale Fine-Grained Image Retrieval
• Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light
• Describe Anything: Detailed Localized Image and Video Captioning
• MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention
• Window Token Concatenation for Efficient Visual Large Language Models
• VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate
• MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation
• TMCIR: Token Merge Benefits Composed Image Retrieval
• PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models
• Storybooth: Training-free Multi-Subject Consistency for Improved Visual Storytelling
• REEF: Relevance-Aware and Efficient LLM Adapter for Video Understanding

Discussions

• Token Merging: Your ViT but Faster [hn, 1 points, 0 comments]

Related

• Video, How Do Your Tokens Merge?
• Thicker and Quicker: A Jumbo Token for Fast Plain Vision Transformers
• Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging
• Multi-Scale And Token Mergence: Make Your ViT More Efficient