Efficient Transformers: A Survey

2020/09/14 by Yi Tay, Mostafa Dehghani, Tay, Yi +5 · 2 voices · 96 citations
Computer Science · #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Speech Recognition and Synthesis #cs.AI #cs.CL #cs.CV #cs.IR #cs.LG

paper · pdf · doi:10.48550/arxiv.2009.06732

Version 2: 2022 edition

arxiv published 2020/09/14 · arxiv created 2022/03/14 · arxiv updated 2022/03/15

Abstract

Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of "X-former" models have been proposed - Reformer, Linformer, Performer, Longformer, to name a few - which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid researcher navigate this flurry, this paper characterizes a large and thoughtful selection of recent efficiency-flavored "X-former" models, providing an organized and comprehensive overview of existing work and models across multiple domains.

Cited by

• A new adaptive two-layer model for opinion spread in hypergraphs: parameter sensitivity and estimation
• Context-Aware Intelligent Chatbot Framework Leveraging Mobile Sensing
• DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generation
• Hyperion: Low-Latency Ultra-HD Video Analytics via Collaborative Vision Transformer Inference
• GraviBERT: Transformer-based inference for gravitational-wave time series
• FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
• Sliced ReLU attention: Quasi-linear contextual expressivity via sorting
• HybridToken-VLM: Hybrid Token Compression for Vision-Language Models
• TextMamba: Scene Text Detector with Mamba
• Energy-Efficient Vision Transformer Inference for Edge-AI Deployment
• Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
• In-Context Compositional Learning via Sparse Coding Transformer
• Controllability Analysis of State Space-based Language Model
• DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams
• InEKFormer: A Hybrid State Estimator for Humanoid Robots
• Zero-Training Task-Specific Model Synthesis for Few-Shot Medical Image Classification
• EchoLSTM: A Self-Reflective Recurrent Network for Stabilizing Long-Range Memory
• X-VMamba: Explainable Vision Mamba
• Enhancing Binary Encoded Crime Linkage Analysis Using Siamese Network
• PRAGMA: A Profiling-Reasoned Multi-Agent Framework for Automatic Kernel Optimization
• Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principle
• Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
• Transformers from Compressed Representations
• ARC-Encoder: learning compressed text representations for large language models
• Task-Aware Reduction for Scalable LLM-Database Systems
• Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
• T-T: Table Transformer for Tagging-based Aspect Sentiment Triplet Extraction
• DeRainMamba: A Frequency-Aware State Space Model with Detail Enhancement for Image Deraining
• CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension
• On Structured State-Space Duality
• Exact Causal Attention with 10% Fewer Operations
• BladderFormer: A Streaming Transformer for Real-Time Urological State Monitoring
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
• Optimizing Privacy-Preserving Primitives to Support LLM-Scale Applications
• Where to Add PDE Diffusion in Transformers
• Mamba Modulation: On the Length Generalization of Mamba
• A Unified Transformer Architecture for Low-Latency and Scalable Wireless Signal Processing
• Lightweight Vision Transformer with Window and Spatial Attention for Food Image Classification
• Neural Speech Separation with Parallel Amplitude and Phase Spectrum Estimation
• SV-Mixer: Replacing the Transformer Encoder with Lightweight MLPs for Self-Supervised Model Compression in Speaker Verification
• AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs
• Comparative Analysis of Large Language Models for the Machine-Assisted Resolution of User Intentions
• Exploring Scaling Laws of CTR Model for Online Performance Improvement
• MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
• DeCAL Tokenwise Compression
• FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilities
• Crisp Attention: Regularizing Transformers via Structured Sparsity
• Intelligent Sampling of Extreme-Scale Turbulence Datasets for Accurate and Efficient Spatiotemporal Model Training
• FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
• Foundation Models and Transformers for Anomaly Detection: A Survey
• MeshMamba: State Space Models for Articulated 3D Mesh Generation and Reconstruction
• Efficient Attention Mechanisms for Large Language Models: A Survey
• On Inductive Biases for Machine Learning in Data Constrained Settings
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
• Foundation Models in Medical Imaging: A Review and Outlook
• Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)
• LISTEN: Lightweight Industrial Sound-representable Transformer for Edge Notification
• GNN-CNN: An Efficient Hybrid Model of Convolutional and Graph Neural Networks for Text Representation
• A Survey on Prompt Tuning
• TransX: Scaling Transformer-based Recommendation via Behavioral and Serving Stream Crossings
• Graph Neural Networks as a Substitute for Transformers in Single-Cell Transcriptomics
• Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptation
• From Large-scale Audio Tagging to Real-Time Explainable Emergency Vehicle Sirens Detection
• A Survey of Human-in-the-loop for Machine Learning
• Towards a Small Language Model Lifecycle Framework
• STAR-Pose: Efficient Low-Resolution Video Human Pose Estimation via Spatial-Temporal Adaptive Super-Resolution
• When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
• Intrinsic and Extrinsic Organized Attention: Softmax Invariance and Network Sparsity
• Two Heads Are Better than One: Simulating Large Transformers with Small Ones
• A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysis
• Don't Pay Attention
• Towards Multimodal Graph Large Language Model
• Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers
• FAR: Function-preserving Attention Replacement for IMC-friendly Inference
• Selection Mechanisms for Sequence Modeling using Linear State Space Models
• Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization
• Balanced and Elastic End-to-end Training of Dynamic LLMs
• TxPert: Leveraging Biochemical Relationships for Out-of-Distribution Transcriptomic Perturbation Prediction
• SPKLIP: Aligning Spike Video Streams with Natural Language
• ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking
• Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments
• Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition
• ParalESN: Enabling parallel information processing in Reservoir Computing
• Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces
• Comparison of Different Deep Neural Network Models in the Cultural Heritage Domain
• KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference
• Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
• A General Survey on Attention Mechanisms in Deep Learning
• CheckOne: Lightweight Fault Detection and Mitigation for Vision Transformers
• Combating Toxic Language: A Review of LLM-Based Strategies for Software Engineering
• Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
• LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm
• Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
• PCA-RAG: Principal Component Analysis for Efficient Retrieval-Augmented Generation
• Lattice: Learning to Efficiently Compress the Memory
• Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation

Discussions

• Efficient Transformers: A Survey [hn, 2 points, 0 comments]
• Efficient Transformers: A Survey [hn, 2 points, 0 comments]

Related

• Introduction to Transformers: an NLP Perspective
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks
• Wide Attention Is The Way Forward For Transformers?
• A Quantitative Review on Language Model Efficiency Research
• Primer: Searching for Efficient Transformers for Language Modeling