Vision Transformers Need Registers

2023/09/28 by Timothée Darcet, Darcet, Timothée, Maxime Oquab +5 · 12 voices · 308 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Computation #Computer science #Domain Adaptation and Few-Shot Learning #Engineering #Feature (linguistics) #Inference #Machine learning #Pattern recognition (psychology) #Supervised learning #Transformer

paper · pdf · doi:10.48550/arxiv.2309.16588

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/09/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

Citations

• ImageNet classification with deep convolutional neural networks
• DINOv2: Learning Robust Visual Features without Supervision

Cited by

• Ordered Action Tokens for Visuomotor Policy Learning
• DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation
• LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition
• IMMoE: Incomplete Multi-View Anomaly Detection via Mixture of View Experts Fusion
• Show Me Examples: Inferring Visual Concepts from Image Sets
• VG3S: Visual Geometry Grounded Gaussian Splatting for Semantic Occupancy Prediction
• Test-Time Registers as Global Priors for Tokenized Image Generation
• Beyond Post-Quantization: Native Hash Learning with a Dedicated HASH Token
• Norm or Direction? Decoding Vision Mambas for High-Resolution Vision
• KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale
• Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
• When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMs
• REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion
• The Role of Initialization in 3D Gaussian Splatting
• A Better Start for Language Models: Domain-Conditional Position Offsets
• Multi-Scale ViT Inference with Habitat-Fit Priors and kNN Retrieval for Multi-Species Plant Identification
• Parameter-efficient Prompt Tuning of Vision Foundation Model With Adaptive Focal Loss for Interpretable MCI Screening
• Enhancing next token prediction based pre-training for jet foundation models
• Stable Audio 3
• Chimère Ω — blueprint for a physico-cognitively inspired local-first LLM runtime
• Kinaema: a recurrent sequence model for memory and pose in motion
• Improving Robotic Manipulation with Efficient Geometry-Aware Vision Encoder
• Perception Encoder: The best visual embeddings are not at the output of the network
• Accurate Ab-initio Neural-network Solutions to Large-Scale Electronic Structure Problems
• VGGT: Visual Geometry Grounded Transformer
• DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers
• ILIAS: Instance-Level Image retrieval At Scale
• ELViS: Efficient Visual Similarity from Local Descriptors that Generalizes Across Domains
• VGGT-Ω
• Revisiting [CLS] and Patch Token Interaction in Vision Transformers
• DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection
• Unlocking Spatial Grounding in Large Audio-Visual Retrieval models
• When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
• Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
• AnyAD: Unified Any-Modality Anomaly Detection in Incomplete Multi-Sequence MRI
• Block-Recurrent Dynamics in Vision Transformers
• SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models
• Enhancing Medical Large Vision-Language Models via Alignment Distillation
• Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentation
• Collaborative Edge-to-Server Inference for Vision-Language Models
• Open Ad-hoc Categorization with Contextualized Feature Learning
• In Pursuit of Pixel Supervision for Visual Pre-training
• Magnification-Aware Distillation (MAD): A Self-Supervised Framework for Unified Representation Learning in Gigapixel Whole-Slide Images
• Unlocking Generalization in Polyp Segmentation with DINO Self-Attention "keys"
• Patch-wise Retrieval: A Bag of Practical Techniques for Instance-level Matching
• VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
• Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving
• Bridging Code Graphs and Large Language Models for Better Code Understanding
• A Geometric Unification of Concept Learning with Concept Cones
• Multi-view Pyramid Transformer: Look Coarser to See Broader
• Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
• On Memory: A comparison of memory mechanisms in world models
• ShaRP: SHAllow-LayeR Pruning for Video Large Language Models Acceleration
• When Do Domain-Specific Foundation Models Justify Their Cost? A Systematic Evaluation Across Retinal Imaging Tasks
• Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion
• Vision Foundry: A System for Training Foundational Vision AI Models
• Flexible Gravitational-Wave Parameter Estimation with Transformers
• CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
• AVGGT: Rethinking Global Attention for Accelerating VGGT
• Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
• FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention
• Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
• FOM-Nav: Frontier-Object Maps for Object Goal Navigation
• TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
• Rethinking Cross-Generator Image Forgery Detection through DINOv3
• RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models
• Patch-Level Glioblastoma Subregion Classification with a Contrastive Learning-Based Encoder
• ADNet: A Large-Scale and Extensible Multi-Domain Benchmark for Anomaly Detection Across 380 Real-World Categories
• DinoLizer: Separating VAE and Diffusion Artifacts in Generative Inpainting Localization
• V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs
• LookSharp: Attention Entropy Minimization for Test-Time Adaptation
• NeAR: Coupled Neural Asset-Renderer Stack
• NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering
• RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models
• DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecture
• Trajectory-guided discharge stratification for heart failure using short-context electronic health record sequence modeling
• Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective
• Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensembles
• A Dataset and Baseline for Deep Learning-Based Visual Quality Inspection in Remanufacturing
• Attention Grounded Enhancement for Visual Document Retrieval
• Training-free Detection of AI-generated images via Cropping Robustness
• SOMA: Feature Gradient Enhanced Affine-Flow Matching for SAR-Optical Registration
• Direct Visual Grounding by Directing Attention of Visual Tokens
• C3Net: Context-Contrast Network for Camouflaged Object Detection
• OPFormer: Object Pose Estimation leveraging foundation model with geometric encoding
• Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
• Φeat: Physically Grounded Material Feature Representation
• EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance
• TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models
• Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
• Towards Cellular-Scale Interpretability in Pathology Foundation Models for Biomarker Assessment
• ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology
• A Cognitive Process-Inspired Architecture for Subject-Agnostic Brain Visual Decoding
• HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation
• Real-IAD Variety: Pushing Industrial Anomaly Detection Dataset to a Modern Era
• Knowledge-guided Disentanglement with Atomic Actions for Action Recognition
• PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstruction
• DINOSim: Zero-Shot Object Detection and Semantic Segmentation on Microscopy Images
• SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense
• What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
• Understanding Multi-View Transformers
• Differential Privacy: Gradient Leakage Attacks in Federated Learning Environments
• Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentation
• Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
• Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
• Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers
• Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detection
• ShortcutBreaker: Low-Rank Noisy Bottleneck and Frequency Filtering Block for Multi-Class Unsupervised Anomaly Detection
• Adapting a global plant identification model to detect invasive alien plant species in high-resolution road side images
• Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs
• Adapting Self-Supervised Representations as a Latent Space for Efficient Generation
• Scaling Vision Transformers for Functional MRI with Flat Maps
• The Mechanistic Emergence of Symbol Grounding in Language Models
• Unconditional Human Motion and Shape Generation via Balanced Score-Based Diffusion
• Diffusion Transformers with Representation Autoencoders
• Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
• ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training
• Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers
• FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging with Diffusion Decoding
• ProteinAE: Protein Diffusion Autoencoders for Structure Encoding
• Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
• Enhancing Concept Localization in CLIP-based Concept Bottleneck Models
• Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imaging
• Visual Representations inside the Language Model
• Activation Quantization of Vision Encoders Needs Prefixing Registers
• Scaling Sequence-to-Sequence Generative Neural Rendering
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
• CLASP: Adaptive Spectral Clustering for Unsupervised Per-Image Segmentation
• ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
• Training Agents Inside of Scalable World Models
• LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transport
• RefAM: Attention Magnets for Zero-Shot Referral Segmentation
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation
• Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
• On the Status of Foundation Models for SAR Imagery
• RetoVLA: Reusing Register Tokens for Spatial Reasoning in Vision-Language-Action Models
• Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
• SiNGER: A Clearer Voice Distills Vision Transformers Further
• Large Pre-Trained Models for Bimanual Manipulation in 3D
• Anatomically Constrained Transformers for Cardiac Amyloidosis Classification
• Enhancing Transformer-Based Vision Models: Addressing Feature Map Anomalies Through Novel Optimization Strategies
• Interpreting ResNet-based CLIP via Neuron-Attention Decomposition
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
• Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
• Driving on Registers
• Isolating to Harness: Cross-Division Distillation for Fully Unsupervised Anomaly Detection
• Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training
• Visual Instruction Pretraining for Domain-Specific Foundation Models
• Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
• Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
• Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few
• Lynx: Towards High-Fidelity Personalized Video Generation
• Language Modeling with Learned Meta-Tokens
• Communication Efficient Split Learning of ViTs with Attention-based Double Compression
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding
• HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models
• Towards Foundational Models for Single-Chip Radar
• DRAG: Data Reconstruction Attack using Guided Diffusion
• Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
• ViewSparsifier: Killing Redundancy in Multi-View Plant Phenotyping
• Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening
• Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models
• Dimensionally Reduced Open-World Clustering: DROWCULA
• Patch-Level Kernel Alignment for Dense Self-Supervised Learning
• Towards Efficient Pixel Labeling for Industrial Anomaly Detection and Localization
• COGITAO: A Visual Reasoning Framework To Study Compositionality & Generalization
• Vision encoders should be image size agnostic and task driven
• Correlates of Image Memorability in Vision Encoders: Activations, Attention Entropy, Patch Uniformity and Autoencoder Losses
• Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation
• Text-Driven 3D Hand Motion Generation from Sign Language Data
• SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass
• DNP-Guided Contrastive Reconstruction with a Reverse Distillation Transformer for Medical Anomaly Detection
• Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
• UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
• UniDCF: A Foundation Model for Comprehensive Dentocraniofacial Hard Tissue Reconstruction
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception
• StyleMM: Stylized 3D Morphable Face Model via Text-Driven Aligned Image Translation
• Multi-Label Plant Species Prediction with Metadata-Enhanced Multi-Head Vision Transformers
• Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing
• Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer
• Toward Context-Aware Exoskeleton Assistance: Integrating Computer Vision Payload Estimation with a User-Centric Optimization Space
• Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens
• Neuro-MoBRE: Exploring Multi-subject Multi-task Intracranial Decoding via Explicit Heterogeneity Resolving
• Hidden Dynamics of Massive Activations in Transformer Training
• Cropping outperforms dropout as an augmentation strategy for training self-supervised text embeddings
• FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
• Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
• What are you sinking? A geometric approach on attention sink
• Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models
• Self-Navigated Residual Mamba for Universal Industrial Anomaly Detection
• Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers
• CoProU-VO: Combining Projected Uncertainty for End-to-End Unsupervised Monocular Visual Odometry
• Representation Shift: Unifying Token Compression with FlashAttention
• Unveiling Super Experts in Mixture-of-Experts Large Language Models
• FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
• Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction
• Temporally Consistent Unsupervised Segmentation for Mobile Robot Perception
• Multi-View Reconstruction with Global Context for 3D Anomaly Detection
• Compositional Video Synthesis by Temporal Object-Centric Learning
• Back to the Features: DINO as a Foundation for Video World Models
• FOCUS: Fused Observation of Channels for Unveiling Spectra
• Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning
• SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensing
• ParallelTime: Dynamically Weighting the Balance of Short- and Long-Term Temporal Dependencies
• Attention (as Discrete-Time Markov) Chains
• Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution
• AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulation
• Multi-Class-Token Transformer for Multitask Self-supervised Music Information Retrieval
• Unsupervised Part Discovery via Descriptor-Based Masked Image Restoration with Optimized Constraints
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• Using Multiple Input Modalities Can Improve Data-Efficiency and O.O.D. Generalization for ML with Satellite Imagery
• Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation
• Inherently Faithful Attention Maps for Vision Transformers
• MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning
• UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
• ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inference
• Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
• Efficient Molecular Conformer Generation with SO(3)-Averaged Flow Matching and Reflow
• Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation
• Know Your Attention Maps: Class-specific Token Masking for Weakly Supervised Semantic Segmentation
• LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion
• OTAS: Open-vocabulary Token Alignment for Outdoor Segmentation
• ReMoE: Report-Guided Mixture-of-Experts for Multimodal OCT/OCTA Anomaly Detection
• Is Visual in-Context Learning for Compositional Medical Tasks within Reach?
• Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models
• GViT: Representing Images as Gaussians for Visual Recognition
• Token Activation Map to Visually Explain Multimodal LLMs
• StableCodec: Taming One-Step Diffusion for Extreme Image Compression
• FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
• scMamba: A Scalable Foundation Model for Single-Cell Multi-Omics Integration Beyond Highly Variable Feature Selection
• Vision Transformers Don't Need Trained Registers
• Light of Normals: Unified Feature Representation for Universal Photometric Stereo
• Image Reconstruction as a Tool for Feature Analysis
• DIP: Unsupervised Dense In-Context Post-training of Visual Representations
• Damba-ST: Domain-Adaptive Mamba for Efficient Urban Spatio-Temporal Prediction
• Emergent Temporal Correspondences from Video Diffusion Transformers
• Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation
• From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
• Pro-AD: Learning Comprehensive Prototypes with Prototype-based Constraint for Multi-class Unsupervised Anomaly Detection
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
• RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors
• DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
• Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
• Random Registers for Cross-Domain Few-Shot Learning
• Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization
• Low-Rank Head Avatar Personalization with Registers
• RAW Image Reconstruction from RGB on Smartphones. NTIRE 2025 Challenge Report
• Efficiency without Compromise: CLIP-aided Text-to-Image GANs with Increased Diversity
• TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
• QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining
• DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
• Object Concepts Emerge from Motion
• Vision Transformers with Self-Distilled Registers
• RoBiS: Robust Binary Segmentation for High-Resolution Industrial Images
• GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution
• The Missing Point in Vision Transformers for Universal Image Segmentation
• VORTA: Efficient Video Diffusion via Routing Sparse Attention
• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
• MoCRA: Mixture of Compositional Rank-1 Atoms for 4K All-in-One Video Restoration
• The emergence of sparse attention: impact of data distribution and benefits of repetition
• Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
• Adaptive Semantic Token Communication for Transformer-based Edge Inference
• Dynamic Resolution Routing for Efficient Egocentric Grounding
• SSR: Similarity-Shift Refinement for Training-Free Object-Centric Masks
• Enhancing Latent Computation in Transformers with Latent Tokens
• Multi-Token Prediction Needs Registers
• Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
• MrTrack: Register Mamba for Needle Tracking with Rapid Reciprocating Motion during Ultrasound-Guided Aspiration Biopsy
• Automated ECG Interval Measurement and Wave Delineation Using Fast Fourier Convolution ResNet
• Vision Foundation Model Embedding-Based Semantic Anomaly Detection
• Discovering Fine-Grained Visual-Concept Relations by Disentangled Optimal Transport Concept Bottleneck Models
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Register and [CLS] tokens yield a decoupling of local and global features in large ViTs
• Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes
• CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment
• Transferable Adversarial Attacks on Black-Box Vision-Language Models
• Taming Outlier Tokens in Diffusion Transformers
• Designing a Good Virtual Node: Addressable and Cardinality-Preserving Global Memory for Message Passing Architectures
• V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
• Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
• Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook
• Vision Transformers Need More Than Registers
• Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
• Learning Multi-view Multi-class Anomaly Detection
• Hallucination in World Models is Predictable and Preventable
• Let ViT Speak: Generative Language-Image Pre-training
• LTX-2: Efficient Joint Audio-Visual Foundation Model
• C-RADIOv4 (Tech Report)
• UniNav: A Unified World-Action Diffusion Model for Visual Navigation
• PhysioSync: Temporal and Cross-Modal Contrastive Learning Inspired by Physiological Synchronization for EEG-Based Emotion Recognition
• When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
• An Empirical Study on Prompt Compression for Large Language Models
• RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection
• CountingDINO: A Training-free Pipeline for Class-Agnostic Counting using Unsupervised Backbones
• Boosting Generative Image Modeling via Joint Image-Feature Synthesis
• IXGS-Intraoperative 3D Reconstruction from Sparse, Arbitrarily Posed Real X-rays
• Search is All You Need for Few-shot Anomaly Detection
• HERA: Historical Evidence Routing Adapter for Physical Prediction in Latent World Models
• Hypergraph Vision Transformers: Images are More than Nodes, More than Edges
• MARS: a Multimodal Alignment and Ranking System for Few-Shot Segmentation
• Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomalies
• Detect All-Type Deepfake Audio: Wavelet Prompt Tuning for Enhanced Auditory Perception
• Earth-Adapter: Bridge the Geospatial Domain Gaps with Mixture of Frequency Adaptation
• Generating ensembles of spatially-coherent in-situ forecasts using flow matching
• Refining CLIP's Spatial Awareness: A Visual-Centric Perspective
• Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets

Discussions

• Vision Transformers Need Registers [hn, 94 points, 9 comments]
• Vision Transformers Need Registers [hn, 22 points, 0 comments]
• Vision transformers need registers! Or at least, it seems they 𝘸𝘢𝘯𝘵 some… ViTs have artifacts in attention maps. It’s due to the model using these patches as “registers”. Just add new tokens (“[re [bsky, 11 points, 0 comments]
• Similar phenomena in vision models. arxiv.org/abs/2309.16588 [bsky, 1 points, 2 comments]
• Vision Transformers Need Registers (extra tokens discarded after last layer) #HackerNews https://arxiv.org/abs/2309.16588 [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 [comments] [70 points] [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3lnvku27r2ud2 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2309.16588 [bsky, 0 points, 1 comments]
• Vision Transformers Need Registers [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 (https://news.ycombinator.com/item?id=43823485) [bsky, 0 points, 0 comments]
• Vision Transformers Need Registers https://arxiv.org/abs/2309.16588 (https://news.ycombinator.com/item?id=43823485) [bsky, 0 points, 0 comments]

Related

• What do Vision Transformers Learn? A Visual Exploration
• Where are my Neighbors? Exploiting Patches Relations in Self-Supervised Vision Transformer
• Vision Transformers with Self-Distilled Registers
• Teaching Matters: Investigating the Role of Supervision in Vision Transformers
• Do Vision Transformers See Like Convolutional Neural Networks?