2024/01/23 by Jianqiao Zheng, Zheng, Jianqiao, Xue-Qian Li +3
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Enhancement Techniques #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2401.12511
openalex publication_date 2024/01/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Training vision transformer networks on small datasets poses challenges. In contrast, convolutional neural networks (CNNs) can achieve state-of-the-art performance by leveraging their architectural inductive bias. In this paper, we investigate whether this inductive bias can be reinterpreted as an initialization bias within a vision transformer network. Our approach is motivated by the finding that random impulse filters can achieve almost comparable performance to learned filters in CNNs. We introduce a novel initialization strategy for transformer networks that can achieve comparable performance to CNNs on small datasets while preserving its architectural flexibility.