Vision Transformers Need Registers

2023/09/28 by Timothée Darcet, Darcet, Timothée, Maxime Oquab +5 · 12 voices · 199 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Advanced Neural Network Applications #Advanced Image and Video Retrieval Techniques

paper · pdf · doi:10.48550/arxiv.2309.16588

Abstract

Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

Citations

• ImageNet classification with deep convolutional neural networks
• DINOv2: Learning Robust Visual Features without Supervision

Cited by

• Ordered Action Tokens for Visuomotor Policy Learning
• DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation
• LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition
• IMMoE: Incomplete Multi-View Anomaly Detection via Mixture of View Experts Fusion
• Show Me Examples: Inferring Visual Concepts from Image Sets
• VG3S: Visual Geometry Grounded Gaussian Splatting for Semantic Occupancy Prediction
• Test-Time Registers as Global Priors for Tokenized Image Generation
• Beyond Post-Quantization: Native Hash Learning with a Dedicated HASH Token
• Norm or Direction? Decoding Vision Mambas for High-Resolution Vision
• KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale
• Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
• When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMs
• REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion
• The Role of Initialization in 3D Gaussian Splatting
• A Better Start for Language Models: Domain-Conditional Position Offsets
• Multi-Scale ViT Inference with Habitat-Fit Priors and kNN Retrieval for Multi-Species Plant Identification
• Parameter-efficient Prompt Tuning of Vision Foundation Model With Adaptive Focal Loss for Interpretable MCI Screening
• Enhancing next token prediction based pre-training for jet foundation models
• Stable Audio 3
• Chimère Ω — blueprint for a physico-cognitively inspired local-first LLM runtime
• Kinaema: a recurrent sequence model for memory and pose in motion
• Improving Robotic Manipulation with Efficient Geometry-Aware Vision Encoder
• Perception Encoder: The best visual embeddings are not at the output of the network
• Accurate Ab-initio Neural-network Solutions to Large-Scale Electronic Structure Problems
• VGGT: Visual Geometry Grounded Transformer
• DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers
• ILIAS: Instance-Level Image retrieval At Scale
• ELViS: Efficient Visual Similarity from Local Descriptors that Generalizes Across Domains
• VGGT-Ω
• Revisiting [CLS] and Patch Token Interaction in Vision Transformers
• DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection
• Unlocking Spatial Grounding in Large Audio-Visual Retrieval models
• When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
• Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
• AnyAD: Unified Any-Modality Anomaly Detection in Incomplete Multi-Sequence MRI
• Block-Recurrent Dynamics in Vision Transformers
• SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models
• Enhancing Medical Large Vision-Language Models via Alignment Distillation
• Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentation
• Collaborative Edge-to-Server Inference for Vision-Language Models
• Open Ad-hoc Categorization with Contextualized Feature Learning
• In Pursuit of Pixel Supervision for Visual Pre-training
• Magnification-Aware Distillation (MAD): A Self-Supervised Framework for Unified Representation Learning in Gigapixel Whole-Slide Images
• Unlocking Generalization in Polyp Segmentation with DINO Self-Attention "keys"
• Patch-wise Retrieval: A Bag of Practical Techniques for Instance-level Matching
• VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
• Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving
• Bridging Code Graphs and Large Language Models for Better Code Understanding
• A Geometric Unification of Concept Learning with Concept Cones
• Multi-view Pyramid Transformer: Look Coarser to See Broader
• Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
• On Memory: A comparison of memory mechanisms in world models
• ShaRP: SHAllow-LayeR Pruning for Video Large Language Models Acceleration
• When Do Domain-Specific Foundation Models Justify Their Cost? A Systematic Evaluation Across Retinal Imaging Tasks
• Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion
• Vision Foundry: A System for Training Foundational Vision AI Models
• Flexible Gravitational-Wave Parameter Estimation with Transformers
• CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
• AVGGT: Rethinking Global Attention for Accelerating VGGT
• Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
• FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention
• Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
• FOM-Nav: Frontier-Object Maps for Object Goal Navigation
• TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
• Rethinking Cross-Generator Image Forgery Detection through DINOv3
• RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models
• Patch-Level Glioblastoma Subregion Classification with a Contrastive Learning-Based Encoder
• ADNet: A Large-Scale and Extensible Multi-Domain Benchmark for Anomaly Detection Across 380 Real-World Categories
• DinoLizer: Separating VAE and Diffusion Artifacts in Generative Inpainting Localization
• V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs
• LookSharp: Attention Entropy Minimization for Test-Time Adaptation
• NeAR: Coupled Neural Asset-Renderer Stack
• NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering
• RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models
• DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecture
• Predicting one-year clinical instability and mortality in heart failure patients using sequence modeling
• Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective
• Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensembles
• A Dataset and Baseline for Deep Learning-Based Visual Quality Inspection in Remanufacturing
• Attention Grounded Enhancement for Visual Document Retrieval
• Training-free Detection of AI-generated images via Cropping Robustness
• SOMA: Feature Gradient Enhanced Affine-Flow Matching for SAR-Optical Registration
• Direct Visual Grounding by Directing Attention of Visual Tokens
• C3Net: Context-Contrast Network for Camouflaged Object Detection
• OPFormer: Object Pose Estimation leveraging foundation model with geometric encoding
• Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
• Φeat: Physically Grounded Material Feature Representation
• EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance
• TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models
• Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
• From Linear Probing to Joint-Weighted Token Hierarchy: A Foundation Model Bridging Global and Cellular Representations in Biomarker Detection
• ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology
• A Cognitive Process-Inspired Architecture for Subject-Agnostic Brain Visual Decoding
• HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation
• Real-IAD Variety: Pushing Industrial Anomaly Detection Dataset to a Modern Era
• Knowledge-guided Disentanglement with Atomic Actions for Action Recognition
• PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstruction
• DINOSim: Zero-Shot Object Detection and Semantic Segmentation on Microscopy Images
• SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense
• What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
• Understanding Multi-View Transformers
• Differential Privacy: Gradient Leakage Attacks in Federated Learning Environments
• Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentation
• Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
• Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
• Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers
• Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detection
• ShortcutBreaker: Low-Rank Noisy Bottleneck with Global Perturbation Attention for Multi-Class Unsupervised Anomaly Detection
• Adapting a global plant identification model to detect invasive alien plant species in high-resolution road side images
• Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs
• Adapting Self-Supervised Representations as a Latent Space for Efficient Generation
• Scaling Vision Transformers for Functional MRI with Flat Maps
• The Mechanistic Emergence of Symbol Grounding in Language Models
• Unconditional Human Motion and Shape Generation via Balanced Score-Based Diffusion
• Diffusion Transformers with Representation Autoencoders
• Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
• ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training
• Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers
• FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging with Diffusion Decoding
• ProteinAE: Protein Diffusion Autoencoders for Structure Encoding
• Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
• Enhancing Concept Localization in CLIP-based Concept Bottleneck Models
• Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imaging
• Visual Representations inside the Language Model
• Activation Quantization of Vision Encoders Needs Prefixing Registers
• Scaling Sequence-to-Sequence Generative Neural Rendering
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
• CLASP: Adaptive Spectral Clustering for Unsupervised Per-Image Segmentation
• Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
• Training Agents Inside of Scalable World Models
• LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transport
• RefAM: Attention Magnets for Zero-Shot Referral Segmentation
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation
• Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
• On the Status of Foundation Models for SAR Imagery
• RetoVLA: Reusing Register Tokens for Spatial Reasoning in Vision-Language-Action Models
• Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
• SiNGER: A Clearer Voice Distills Vision Transformers Further
• Large Pre-Trained Models for Bimanual Manipulation in 3D
• Anatomically Constrained Transformers for Cardiac Amyloidosis Classification
• Enhancing Transformer-Based Vision Models: Addressing Feature Map Anomalies Through Novel Optimization Strategies
• Interpreting ResNet-based CLIP via Neuron-Attention Decomposition
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
• Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
• Driving on Registers
• Isolating to Harness: Cross-Division Distillation for Fully Unsupervised Anomaly Detection
• Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training
• Visual Instruction Pretraining for Domain-Specific Foundation Models
• Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
• Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
• Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few
• Lynx: Towards High-Fidelity Personalized Video Generation
• Language Modeling with Learned Meta-Tokens
• Communication Efficient Split Learning of ViTs with Attention-based Double Compression
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
• HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models
• Towards Foundational Models for Single-Chip Radar
• DRAG: Data Reconstruction Attack using Guided Diffusion
• Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
• ViewSparsifier: Killing Redundancy in Multi-View Plant Phenotyping
• Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening
• Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models
• Dimensionally Reduced Open-World Clustering: DROWCULA
• Patch-Level Kernel Alignment for Dense Self-Supervised Learning
• Towards Efficient Pixel Labeling for Industrial Anomaly Detection and Localization
• COGITAO: A Visual Reasoning Framework To Study Compositionality & Generalization
• Vision encoders should be image size agnostic and task driven
• Correlates of Image Memorability in Vision Encoders: Activations, Attention Entropy, Patch Uniformity and Autoencoder Losses
• Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation
• Text-Driven 3D Hand Motion Generation from Sign Language Data
• SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass
• DNP-Guided Contrastive Reconstruction with a Reverse Distillation Transformer for Medical Anomaly Detection
• Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
• UniDCF: A Foundation Model for Comprehensive Dentocraniofacial Hard Tissue Reconstruction
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception
• StyleMM: Stylized 3D Morphable Face Model via Text-Driven Aligned Image Translation
• Multi-Label Plant Species Prediction with Metadata-Enhanced Multi-Head Vision Transformers
• Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing
• Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer
• Toward Context-Aware Exoskeleton Assistance: Integrating Computer Vision Payload Estimation with a User-Centric Optimization Space
• Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens
• Neuro-MoBRE: Exploring Multi-subject Multi-task Intracranial Decoding via Explicit Heterogeneity Resolving
• Hidden Dynamics of Massive Activations in Transformer Training
• Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings
• FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
• Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
• What are you sinking? A geometric approach on attention sink
• Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models
• Self-Navigated Residual Mamba for Universal Industrial Anomaly Detection
• CoProU-VO: Combining Projected Uncertainty for End-to-End Unsupervised Monocular Visual Odometry
• Representation Shift: Unifying Token Compression with FlashAttention
• Unveiling Super Experts in Mixture-of-Experts Large Language Models
• FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
• Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction
• Temporally Consistent Unsupervised Segmentation for Mobile Robot Perception
• Multi-View Reconstruction with Global Context for 3D Anomaly Detection
• Compositional Video Synthesis by Temporal Object-Centric Learning

Discussions

• Vision Transformers Need Registers [hn, 94 points, 9 comments]
• Vision Transformers Need Registers [hn, 22 points, 0 comments]
• Vision transformers need registers! Or at least, it seems they 𝘸𝘢𝘯𝘵 some… ViTs have artifacts in attention maps. It’s due to the model using these patches as “registers”. Just add new tokens (“[re [bsky, 11 points, 0 comments]
• Similar phenomena in vision models. arxiv.org/abs/2309.16588 [bsky, 1 points, 2 comments]
• Vision Transformers Need Registers (extra tokens discarded after last layer) #HackerNews https://arxiv.org/abs/2309.16588 [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 [comments] [70 points] [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3lnvku27r2ud2 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2309.16588 [bsky, 0 points, 1 comments]
• Vision Transformers Need Registers [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 (https://news.ycombinator.com/item?id=43823485) [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 (https://news.ycombinator.com/item?id=43823485) [bsky, 0 points, 0 comments]

Related

• What do Vision Transformers Learn? A Visual Exploration
• Where are my Neighbors? Exploiting Patches Relations in Self-Supervised Vision Transformer
• Vision Transformers with Self-Distilled Registers
• Teaching Matters: Investigating the Role of Supervision in Vision Transformers
• Do Vision Transformers See Like Convolutional Neural Networks?