Pay Attention to MLPs

2021/05/17 by Hanxiao Liu, Zihang Dai, Liu, Hanxiao +5 · 3 voices · 25 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2105.08050

arxiv published 2021/05/17 · arxiv updated 2021/06/01

Abstract

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.

Cited by

• Raven: High-Recall Sequence Modeling with Sparse Memory Routing
• "Why the face?": Exploring Robot Error Detection Using Instrumented Bystander Reactions
• DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentation
• STAMP: Spatial-Temporal Adapter with Multi-Head Pooling
• Generating Sketches in a Hierarchical Auto-Regressive Process for Flexible Sketch Drawing Manipulation at Stroke-Level
• SAGS: Self-Adaptive Alias-Free Gaussian Splatting for Dynamic Surgical Endoscopic Reconstruction
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• Flow-Matching Guided Deep Unfolding for Hyperspectral Image Reconstruction
• Short window attention enables long-term memorization
• CURA: Size Isnt All You Need -- A Compact Universal Architecture for On-Device Intelligence
• A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation
• JaneEye: A 12-nm 2K-FPS 18.9-μJ/Frame Event-based Eye Tracking Accelerator
• SV-Mixer: Replacing the Transformer Encoder with Lightweight MLPs for Self-Supervised Model Compression in Speaker Verification
• Token-based Attractors and Cross-attention in Spoof Diarization
• SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification
• Multi-level SSL Feature Gating for Audio Deepfake Detection
• Scaling Neuro-symbolic Problem Solving: Solver-Free Learning of Constraints and Objectives
• Image Quality Assessment for Machines: Paradigm, Large-scale Database, and Models
• Learning to See Through Flare
• Keyword Mamba: Spoken Keyword Spotting with State Space Models
• ClinicalFMamba: Advancing Clinical Assessment using Mamba-based Multimodal Neuroimaging Fusion
• MLP Memory: A Retriever-Pretrained Memory for Large Language Models
• Learning Unified System Representations for Microservice Tail Latency Prediction
• From Individuals to Crowds: Dual-Level Public Response Prediction in Social Media
• Implicit Counterfactual Learning for Audio-Visual Segmentation

Discussions

• Pay Attention to MLPs [hn, 3 points, 0 comments]
• Pay Attention to MLPs [hn, 2 points, 0 comments]
• Gated MLP performs on par with transformers [hn, 1 points, 0 comments]

Related

• Transformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problems
• MLP Can Be A Good Transformer Learner
• Wide Attention Is The Way Forward For Transformers?
• Three things everyone should know about Vision Transformers
• Rethinking the shape convention of an MLP