Not All Attention Is All You Need

2021/04/10 by Hongqiu Wu, Hai Zhao, Wu, Hongqiu +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2104.04692

openalex publication_date 2021/04/10 · arxiv created 2021/06/01 · arxiv updated 2021/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Beyond the success story of pre-trained language models (PrLMs) in recent natural language processing, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrLMs. In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks.

Citations

• Variational Dropout Sparsifies Deep Neural Networks
• DARTS: Differentiable Architecture Search
• Auto-Sizing Neural Networks: With Applications to n-gram Language Models
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Linformer: Self-Attention with Linear Complexity
• Layer Normalization
• Alleviating the Inequality of Attention Heads for Neural Machine Translation
• Code Summarization with Structure-induced Transformer

Related

• AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuning
• Drop Dropout on Single-Epoch Language Model Pretraining
• Scheduled DropHead: A Regularization Method for Transformer Models
• Gradient-based Intra-attention Pruning on Pre-trained Language Models
• Self-training Large Language Models through Knowledge Detection