Deepening Neural Networks Implicitly and Locally via Recurrent Attention Strategy

2022/10/27 by Shanshan Zhong, Wushao Wen, Zhong, Shanshan +5
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2210.15676

openalex publication_date 2022/10/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

More and more empirical and theoretical evidence shows that deepening neural networks can effectively improve their performance under suitable training settings. However, deepening the backbone of neural networks will inevitably and significantly increase computation and parameter size. To mitigate these problems, we propose a simple-yet-effective Recurrent Attention Strategy (RAS), which implicitly increases the depth of neural networks with lightweight attention modules by local parameter sharing. The extensive experiments on three widely-used benchmark datasets demonstrate that RAS can improve the performance of neural networks at a slight addition of parameter size and computation, performing favorably against other existing well-known attention modules.

Related

• A Generic Shared Attention Mechanism for Various Backbone Neural Networks
• Efficient Attention Network: Accelerate Attention by Searching Where to Plug
• BAM: Bottleneck Attention Module
• Attention-based Conditioning Methods for External Knowledge Integration
• Untangling tradeoffs between recurrence and self-attention in neural networks