Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model

2021/05/31 by Jiangning Zhang, Chao Xu, Zhang, Jiangning +17
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Metaheuristic Optimization Algorithms Research

paper · pdf · doi:10.48550/arxiv.2105.15089

openalex publication_date 2021/05/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Inspired by biological evolution, we explain the rationality of Vision Transformer by analogy with the proven practical Evolutionary Algorithm (EA) and derive that both of them have consistent mathematical representation. Analogous to the dynamic local population in EA, we improve the existing transformer structure and propose a more efficient EAT model, and design task-related heads to deal with different tasks more flexibly. Moreover, we introduce the spatial-filling curve into the current vision transformer to sequence image data into a uniform sequential format. Thus we can design a unified EAT framework to address multi-modal tasks, separating the network architecture from the data format adaptation. Our approach achieves state-of-the-art results on the ImageNet classification task compared with recent vision transformer works while having smaller parameters and greater throughput. We further conduct multi-modal tasks to demonstrate the superiority of the unified EAT, e.g., Text-Based Image Retrieval, and our approach improves the rank-1 by +3.7 points over the baseline on the CSS dataset.

Citations

• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Reformer: The Efficient Transformer
• Language Models are Few-Shot Learners
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Linformer: Self-Attention with Linear Complexity
• Space-filling Curves for High-performance Data Mining
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Pre-Trained Image Processing Transformer
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Is Space-Time Attention All You Need for Video Understanding?
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• Transformer in Transformer
• DeepViT: Towards Deeper Vision Transformer
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Conditional Positional Encodings for Vision Transformers
• CvT: Introducing Convolutions to Vision Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
• CoAtNet: Marrying Convolution and Attention for All Data Sizes

Related

• EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm
• GLiT: Neural Architecture Search for Global and Local Image Transformer
• Three things everyone should know about Vision Transformers
• Patch Is Not All You Need
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition