Pay Less Attention with Lightweight and Dynamic Convolutions

2019/01/29 by Felix Wu, Wu, Felix, Angela Fan +7 · 35 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1901.10430

openalex publication_date 2019/01/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

Cited by

• cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs
• Multi-refined Feature Enhanced Sentiment Analysis Using Contextual Instruction
• On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
• Resurrecting Submodularity for Neural Text Generation
• Adaptively Sparse Transformers
• Recurrent Graph Syntax Encoder for Neural Machine Translation
• Shallow-to-Deep Training for Neural Machine Translation
• DLA-Net: Learning Dual Local Attention Features for Semantic Segmentation of Large-Scale Building Facade Point Clouds
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• Beyond English-Centric Multilingual Machine Translation
• Language Models are Good Translators
• Stand-Alone Self-Attention in Vision Models
• The University of Sydney's Machine Translation System for WMT19
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
• Not All Memories are Created Equal: Learning to Forget by Expiring
• Understanding the Difficulty of Training Transformers
• PDE-Transformer: A Continuous Dynamical Systems Approach to Sequence Modeling
• Revisit Systematic Generalization via Meaningful Learning
• Hierarchical Attention Transformer Architecture For Syntactic Spell\n Correction
• Strategies for Structuring Story Generation
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Exploring Self-attention for Image Recognition
• On the Importance of Local Information in Transformer Based Models
• Dynamic Convolution for 3D Point Cloud Instance Segmentation
• Long Context Automated Essay Scoring with Language Models
• Do Transformers Need Deep Long-Range Memory
• Probing Word Translations in the Transformer and Trading Decoder for Encoder Layers
• Efficiently Modeling Long Sequences with Structured State Spaces
• Data Diversification: A Simple Strategy For Neural Machine Translation
• Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
• Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
• Training Flexible Depth Model by Multi-Task Learning for Neural Machine Translation
• Mask Attention Networks: Rethinking and Strengthen Transformer

Related

• Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
• On the Integration of Self-Attention and Convolution
• Multiscale Self Attentive Convolutions for Vision and Language Modeling
• Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
• Translution: Unifying Self-attention and Convolution for Adaptive and Relative Modeling