The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers

2025/12/08 by Kanishk Awadhiya, Awadhiya, Kanishk
Neuroscience · Computer Science · #Face Recognition and Perception #Generative Adversarial Networks and Image Synthesis #Advanced Neural Network Applications

paper · pdf · doi:10.48550/arxiv.2512.07331

Abstract

Vision Transformers (ViTs) lack the hierarchical inductive biases inherent to Convolutional Neural Networks (CNNs), theoretically allowing them to maintain high-dimensional representations throughout all layers. However, recent observations suggest ViTs often spontaneously manifest a "U-shaped" entropy profile-compressing information in middle layers before expanding it for the final classification. In this work, we demonstrate that this "Inductive Bottleneck" is not an architectural artifact, but a data-dependent adaptation. By analyzing the layer-wise Effective Encoding Dimension (EED) of DINO-trained ViTs across datasets of varying compositional complexity (UC Merced, Tiny ImageNet, and CIFAR-100), we show that the depth of the bottleneck correlates strongly with the semantic abstraction required by the task. We find that while texture-heavy datasets preserve high-rank representations throughout, object-centric datasets drive the network to dampen high-frequency information in middle layers, effectively "learning" a bottleneck to isolate semantic features.

Citations

• Do Vision Transformers See Like Convolutional Neural Networks?
• Deep Residual Learning for Image Recognition
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• Do Vision Transformers See Like Convolutional Neural Networks?
• IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism
• Deep Transformers Thirst for Comprehensive-Frequency Data
• From Colors to Classes: Emergence of Concepts in Vision Transformers
• DeepViT: Towards Deeper Vision Transformer