Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformers

2025/11/18 by Chen, Yutian, Qiu, Yuheng, Li, Ruogu +4
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2511.14751

Abstract

We propose Confidence-Guided Token Merging (Co-Me), an acceleration mechanism for visual geometric transformers without retraining or finetuning the base model. Co-Me distilled a light-weight confidence predictor to rank tokens by uncertainty and selectively merge low-confidence ones, effectively reducing computation while maintaining spatial coverage. Compared to similarity-based merging or pruning, the confidence signal in Co-Me reliably indicates regions emphasized by the transformer, enabling substantial acceleration without degrading performance. Co-Me applies seamlessly to various multi-view and streaming visual geometric transformers, achieving speedups that scale with sequence length. When applied to VGGT and MapAnything, Co-Me achieves up to 11.3× and 7.2× speedup, making visual geometric transformers practical for real-time 3D perception and reconstruction.

Citations

• Depth Anything 3: Recovering the Visual Space from Any Views
• STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
• VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences
• Streaming 4D Visual Geometry Transformer
• Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors
• MUSt3R: Multi-view Network for Stereo 3D Reconstruction
• Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass
• Continuous 3D Perception Model with Persistent State
• Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization
• Flex Attention: A Programming Model for Generating Optimized Attention Kernels
• Align3R: Aligned Monocular Depth Estimation for Dynamic Videos
• MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry
• 3D Reconstruction with Spatial Memory
• DUSt3R: Geometric 3D Vision Made Easy
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging
• Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• PuMer: Pruning and Merging Tokens for Efficient Vision Language Models
• Token Merging for Fast Stable Diffusion
• SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer
• X-Pruner: eXplainable Pruning for Vision Transformers
• AdaViT: Adaptive Tokens for Efficient Vision Transformer
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• On the uncertainty of self-supervised monocular depth estimation
• Sparsity Invariant CNNs
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

Related

• LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
• HTTM: Head-wise Temporal Token Merging for Faster VGGT
• Efficient Visual Transformer by Learnable Token Merging
• Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
• FastVGGT: Training-Free Acceleration of Visual Geometry Transformer