Attention, please! A survey of Neural Attention Models in Deep Learning

2021/03/31 by Alana de Santana Correia, Correia, Alana de Santana, Esther Luna Colombini +1 · 9 citations
Computer Science · Neuroscience · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #EEG and Brain-Computer Interfaces #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO) #Visual Attention and Saliency Detection #cs.AI #cs.CV #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2103.16775

66 pages, 24 figures

arxiv created 2021/03/31 · openalex publication_date 2021/03/31 · arxiv updated 2021/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In humans, Attention is a core property of all perceptual and cognitive operations. Given our limited ability to process competing sources, attention mechanisms select, modulate, and focus on the information most relevant to behavior. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. For the last six years, this property has been widely explored in deep neural networks. Currently, the state-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models. We systematically reviewed hundreds of architectures in the area, identifying and discussing those in which attention has shown a significant impact. We also developed and made public an automated methodology to facilitate the development of reviews in the area. By critically analyzing 650 works, we describe the primary uses of attention in convolutional, recurrent networks and generative models, identifying common subgroups of uses and applications. Furthermore, we describe the impact of attention in different application domains and their impact on neural networks' interpretability. Finally, we list possible trends and opportunities for further research, hoping that this review will provide a succinct overview of the main attentional models in the area and guide researchers in developing future approaches that will drive further improvements.

Cited by

• From generative AI to the brain: five takeaways
• Enhancing Time-Series Anomaly Detection by Integrating Spectral-Residual Bottom-Up Attention with Reservoir Computing
• AI-Based Stroke Rehabilitation Domiciliary Assessment System with ST_GCN Attention
• PAN: Pillars-Attention-Based Network for 3D Object Detection
• Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networks
• Enhancing Learning Path Recommendation via Multi-task Learning
• LIMAO: A Framework for Lifelong Modular Learned Query Optimization
• Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
• An Improved ResNet50 Model for Predicting Pavement Condition Index (PCI) Directly from Pavement Images

Related

• Neural Attention Models in Deep Learning: Survey and Taxonomy
• An Attentive Survey of Attention Models
• Understanding Attention: In Minds and Machines
• Visual Attention Methods in Deep Learning: An In-Depth Survey
• Trends, Applications, and Challenges in Human Attention Modelling