Long Range Arena: A Benchmark for Efficient Transformers

2020/11/08 by Yi Tay, Mostafa Dehghani, Tay, Yi +17 · 195 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Aerospace engineering #Benchmark (surveying) #Cartography #Computer science #Electrical engineering #Engineering #Ferroelectric and Negative Capacitance Devices #Geography #Power Line Inspection Robots #Range (aeronautics) #Transformer #Voltage #cs.AI #cs.CL #cs.CV #cs.IR #cs.LG

paper · pdf · doi:10.48550/arxiv.2011.04006

published in arXiv (Cornell University) (Cornell University)

arxiv created 2020/11/08 · openalex publication_date 2020/11/08 · arxiv updated 2020/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from 1K to 16K tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.

Cited by

• WHTMix: Efficient Stereo Depth Estimation via Walsh-Hadamard Token Mixing
• Sliced ReLU attention: Quasi-linear contextual expressivity via sorting
• PathMamba: A Hybrid Mamba-Transformer for Topologically Coherent Road Segmentation in Satellite Imagery
• LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Model
• Pretrained Transformers as Universal Computation Engines
• EchoLSTM: A Self-Reflective Recurrent Network for Stabilizing Long-Range Memory
• Semantic Multiplexing
• Belief Net: A Filter-Based Framework for Learning Hidden Markov Models from Observations
• Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off
• Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders
• BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models
• Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• The Curious Case of In-Training Compression of State Space Models
• Metadata Shaping: Natural Language Annotations for the Tail
• Hankel Singular Value Regularization for Highly Compressible State Space Models
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• A Deep State-Space Model Compression Method using Upper Bound on Output Error
• Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
• Design Principles for Sequence Models via Coefficient Dynamics
• Eigenvalues as a Metric for Memory Dynamics in Sequence Models
• OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Wave-PDE Nets: Trainable Wave-Equation Layers as an Alternative to Attention
• RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts
• Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
• Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation
• Where to Add PDE Diffusion in Transformers
• Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models
• Aligning Inductive Bias for Data-Efficient Generalization in State Space Models
• Myosotis: structured computation for attention like layer
• Mamba Modulation: On the Length Generalization of Mamba
• Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• An overview of neural architectures for self-supervised audio representation learning from masked spectrograms
• CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density
• Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modeling
• Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking
• Holographic Transformers for Complex-Valued Signal Processing: Integrating Phase Interference into Self-Attention
• Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
• Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• Uncovering the Spectral Bias in Diagonal State Space Models
• Revisiting associative recall in modern recurrent models
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• FLARE: Fast Low-rank Attention Routing Engine
• Parity Requires Unified Input Dependence and Negative Eigenvalues in SSMs
• Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision
• Small transformer architectures for task switching
• Long-Range Transformers for Dynamic Spatiotemporal Forecasting
• Systolic Array-based Accelerator for Structured State-Space Models
• Modality Agnostic Efficient Long Range Encoder
• SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models
• Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
• Ripple Attention for Visual Perception with Sub-quadratic Complexity
• The Benchmark Lottery
• FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training
• CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Model
• Compression Method for Deep Diagonal State Space Model Based on H² Optimal Reduction
• MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses
• QS4D: Quantization-aware training for efficient hardware deployment of structured state-space sequential models
• Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by nano-GPT
• mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling
• W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling
• Uncovering the Functional Roles of Nonlinearity in Memory
• ContextCache: Context-Aware Semantic Cache for Multi-Turn Queries in Large Language Models
• Long-Sequence Memory with Temporal Kernels and Dense Hopfield Functionals
• Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention
• The Effect of Depth on the Expressivity of Deep Linear State-Space Models
• Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba
• Improving Genomic Models via Task-Specific Self-Pretraining
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
• A Quantile Regression Approach for Remaining Useful Life Estimation with State Space Models
• From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
• SKOLR: Structured Koopman Operator Linear RNN for Time-Series Forecasting
• A Scalable Hybrid Training Approach for Recurrent Spiking Neural Networks
• Scaling Algorithm Distillation for Continuous Control with Mamba
• LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions
• Improving the Efficiency of Long Document Classification using Sentence Ranking Approach
• The Efficiency Misnomer
• Revisiting Transformers with Insights from Image Filtering and Boosting
• On Measuring Long-Range Interactions in Graph Neural Networks
• Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models
• Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions
• Context Is Not Comprehension
• From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems
• SiLIF: Structured State Space Model Dynamics and Parametrization for Spiking Neural Networks
• Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention
• Weight-Space Linear Recurrent Neural Networks
• Mamba Drafters for Speculative Decoding
• SHORING: Design Provable Conditional High-Order Interaction Network via Symbolic Testing
• BLAB: Brutally Long Audio Bench
• ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations
• Neuromorphic Sequential Arena: A Benchmark for Neuromorphic Temporal Processing
• Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries
• Sparsified State-Space Models are Efficient Highway Networks
• Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers
• Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis
• Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answering
• Revisiting Glorot Initialization for Long-Range Linear Recurrences
• Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• Enhancing Transformers Through Conditioned Embedded Tokens
• SchoenbAt: Rethinking Attention with Polynomial basis
• Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain
• Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments
• LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
• ParalESN: Enabling parallel information processing in Reservoir Computing
• Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures
• Parallelizable memory recurrent units
• Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
• Long-context Non-factoid Question Answering in Indic Languages
• Sparse Factorization of Large Square Matrices
• MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models

Related

• On the Long Range Abilities of Transformers
• On the locality bias and results in the Long Range Arena
• Efficient Transformers: A Survey
• Sparse is Enough in Scaling Transformers
• Transformer Quality in Linear Time