Do Vision Transformers See Like Convolutional Neural Networks?

2021/08/19 by Maithra Raghu, Raghu, Maithra, Thomas Unterthiner +7 · 2 voices · 109 citations
Computer Science · Engineering · Mathematics · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Computer science #Convolutional neural network #De facto #Domain Adaptation and Few-Shot Learning #Engineering #Feature learning #Machine learning #Pattern recognition (psychology) #Residual #Transfer of learning #Transformer #cs.AI #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2108.08810

published in arXiv (Cornell University) 34 (Cornell University)

openalex publication_date 2021/08/19 · arxiv created 2022/03/03 · arxiv updated 2022/03/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• ImageNet classification with deep convolutional neural networks
• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Similarity of Neural Network Representations Revisited
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Stand-Alone Self-Attention in Vision Models
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• Investigating Multilingual NMT Representations at Scale
• Emerging Cross-lingual Structure in Pretrained Language Models
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Supervised Transfer Learning at Scale for Medical Imaging
• MLP-Mixer: An all-MLP Architecture for Vision
• Emerging Properties in Self-Supervised Vision Transformers
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• Intriguing Properties of Vision Transformers

Cited by

• Rethinking Multi-Branch and Cross-Backbone Fusion for Vehicle Re-Identification in the Foundation-Model Era
• Norm or Direction? Decoding Vision Mambas for High-Resolution Vision
• Position: Explanation Stability Is a Property of the Model Method Pair, Not the Model
• The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers
• What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?
• Geometric 2D Scene Graph Generation
• Representational Similarity via Interpretable Visual Concepts
• Detect Before You Leap: Mirage Detection in Vision-Language Models
• Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
• Fast SAM2 with Text-Driven Token Pruning
• PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding
• Block-Recurrent Dynamics in Vision Transformers
• CALM: A CKA-Guided Adaptive Layer-Wise Modularization Framework for LLM Quantization
• Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models
• Semantic Mismatch and Perceptual Degradation: A New Perspective on Image Editing Immunity
• From Tokens to Photons: Test-Time Physical Prompting for Vison-Language Models
• MKSNet: Advanced Small Object Detection in Remote Sensing Imagery with Multi-Kernel and Dual Attention Mechanisms
• NAS-LoRA: Empowering Parameter-Efficient Fine-Tuning for Visual Foundation Models with Searchable Adaptation
• Hierarchical Attention for Sparse Volumetric Anomaly Detection in Subclinical Keratoconus
• Frequency-Aware Token Reduction for Efficient Vision Transformer
• CountXplain: Interpretable Cell Counting with Prototype-Based Density Map Estimation
• LookSharp: Attention Entropy Minimization for Test-Time Adaptation
• Enhancing Adversarial Transferability through Block Stretch and Shrink
• DetailSemNet: Elevating Signature Verification through Detail-Semantic Integration
• TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing
• Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
• FiCABU: A Fisher-Based, Context-Adaptive Machine Unlearning Processor for Edge AI
• SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentation
• MapSAM2: Adapting SAM2 for Automatic Segmentation of Historical Map Images and Time Series
• FPS: Feedforward-based Parameter Selection For Efficient Fine-Tuning
• Privacy-Aware Continual Self-Supervised Learning on Multi-Window Chest Computed Tomography for Domain-Shift Robustness
• Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods
• Representation Trajectories Matters: Complementary Evidence for OOD Detection and Image Classification
• Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection
• Attentive multilayer fusion for vision transformers
• FreeFuse: Multi-Subject LoRA Fusion via Adaptive Token-Level Routing at Test Time
• Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction
• Model Merging with Functional Dual Anchors
• I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMs
• Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
• LayerSync: Self-aligning Intermediate Layers
• FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging with Diffusion Decoding
• Task-Aware Resolution Optimization for Visual Large Language Models
• A Denoising Framework for Real-World Ultra-Low-Dose Lung CT Images Based on an Image Purification Strategy
• DADO: A Depth-Attention framework for Object Discovery
• There is More to Attention: Statistical Filtering Enhances Explanations in Vision Transformers
• A Comparative Study of Vision Transformers and CNNs for Few-Shot Rigid Transformation and Fundamental Matrix Estimation
• Representation Potentials of Foundation Models for Multimodal Alignment: A Survey
• CoPA: Hierarchical Concept Prompting and Aggregating Network for Explainable Diagnosis
• Generalist Multi-Class Anomaly Detection via Distillation to Two Heterogeneous Student Networks
• Predicting ventilation from single breathing phase non‐contrast CT using Swin Transformers
• Latent Danger Zone: Distilling Unified Attention for Cross-Architecture Black-box Attacks
• Deep Learning Inductive Biases for fMRI Time Series Classification during Resting-state and Movie-watching
• Which Direction to Choose? An Analysis on the Representation Power of Self-Supervised ViTs in Downstream Tasks
• LadderSym: A Multimodal Interleaved Transformer for Music Practice Error Detection
• CNN-ViT Hybrid for Pneumonia Detection: Theory and Empiric on Limited Data without Pretraining
• Enhancing Robustness in Post-Processing Watermarking: An Ensemble Attack Network Using CNNs and Transformers
• SKGE-SWIN: End-To-End Autonomous Vehicle Waypoint Prediction and Navigation Using Skip Stage Swin Transformer
• Beyond Imaging: Vision Transformer Digital Twin Surrogates for 3D+T Biological Tissue Dynamics
• ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Models
• Efficient Identification of Critical Transitions via Flow Matching: A Scalable Generative Approach for Many-Body Systems
• Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints
• Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment
• DSS-Prompt: Dynamic-Static Synergistic Prompting for Few-Shot Class-Incremental Learning
• MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization
• Deeper Inside Deep ViT
• Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance
• I Am Big, You Are Little; I Am Right, You Are Wrong
• Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment
• Mitigating Spurious Correlations in Weakly Supervised Semantic Segmentation via Cross-architecture Consistency Regularization
• Vision Transformer attention alignment with human visual perception in aesthetic object evaluation
• Attention (as Discrete-Time Markov) Chains
• Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation
• Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation
• Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation
• Multiple Streams of Knowledge Retrieval: Enriching and Recalling in Transformers
• Image Reconstruction as a Tool for Feature Analysis
• Boosting Adversarial Transferability via Commonality-Oriented Gradient Optimization
• DuoFormer: Leveraging Hierarchical Representations by Local and Global Attention Vision Transformer
• Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs
• Interpretable Classification of Levantine Ceramic Thin Sections via Neural Networks
• Aligning Multimodal Representations through an Information Bottleneck
• Sparse Imagination for Efficient Visual World Model Planning
• MangoLeafViT: Leveraging Lightweight Vision Transformer with Runtime Augmentation for Efficient Mango Leaf Disease Classification
• Representational Difference Explanations
• TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization
• Spiking Neural Networks Need High Frequency Information
• Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs
• Learning to Adapt to Position Bias in Vision Transformer Classifiers
• Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts
• Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
• BrainSegDMlF: A Dynamic Fusion-enhanced SAM for Brain Lesion Segmentation
• Taming Outlier Tokens in Diffusion Transformers
• Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images
• Vision Transformers in Precision Agriculture: A Comprehensive Survey
• LM-MCVT: A Lightweight Multi-modal Multi-view Convolutional-Vision Transformer Approach for 3D Object Recognition
• Clinically-Grounded Hierarchical Classification for Consistent Chest X-ray Interpretation
• Seeking Physics in Diffusion Noise
• ECViT: Efficient Convolutional Vision Transformer with Local-Attention and Multi-scale Stages
• Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction
• Resilience of Vision Transformers for Domain Generalisation in the Presence of Out-of-Distribution Noisy Images
• Vision transformer [wikipedia]

Discussions

• Do vision transformers see like convolutional neural networks? [hn, 110 points, 42 comments]
• Sure, different architectures being different, represent those same things in their own way (some learn faster) e.g. VITs use more global information in lower levels and skip connections make layers s [bsky, 2 points, 1 comments]

Related

• What do Vision Transformers Learn? A Visual Exploration
• ConvNets vs. Transformers: Whose Visual Representations are More Transferable?
• DeepViT: Towards Deeper Vision Transformer
• Intriguing Properties of Vision Transformers
• CvT: Introducing Convolutions to Vision Transformers