Modality Agnostic Efficient Long Range Encoder

2025/07/25 by Toufiq Parag, Ahmed Elgammal, Parag, Toufiq +1
Computer Science · #Advanced Neural Network Applications #Big Data and Digital Economy #Computational complexity theory #Computer Vision and Pattern Recognition (cs.CV) #Encoder #Encoding (memory) #FOS: Computer and information sciences #Inference #Memory footprint #Parallel Computing and Optimization Techniques #Quadratic equation #Security token #Transformer

paper · pdf · doi:10.48550/arxiv.2507.19409

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The long-context capability of recent large transformer models can be surmised to rely on techniques such as attention/model parallelism, as well as hardware-level optimizations. While these strategies allow input lengths to scale to millions of tokens, they do not fundamentally mitigate the quadratic computational and memory complexity of the core attention mechanism. In this paper, we address the challenge of long-context processing on a single device using generic implementations by reducing the quadratic memory footprint and inference cost. Existing approaches to extend the context length for generic single device implementations -- such as token merging and modified attentions -- are often modality specific and attain a suboptimal tradeoff between accuracy and efficiency. To overcome these limitations, we propose MAELRE (Modality Agnostic Efficient Long Range Encoder), a unified and efficient transformer architecture designed for long-range encoding across diverse modalities. MAELRE integrates token merging with attention approximation, progressively merging tokens at different stages of internal computational blocks. It employs a lightweight attention approximation when the number of tokens is large, and switches to standard dot-product attention as the sequence becomes shorter through successive aggregation. We demonstrate that MAELRE achieves superior accuracy while reducing computational cost compared to existing long-context models on classification tasks spanning multiple modalities, including text, time series, audio, and vision.

Citations

• Scaling Granite Code Models to 128K Context
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Effective Long-Context Scaling of Foundation Models
• FLatten Transformer: Vision Transformer using Focused Linear Attention
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models
• Scaling Transformer to 1M tokens and beyond with RMT
• Multiscale Audio Spectrogram Transformer for Efficient Audio Classification
• Hungry Hungry Hippos: Towards Language Modeling with State Space Models
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
• Token Merging: Your ViT But Faster
• Contrastive Audio-Visual Masked Autoencoder
• Investigating Efficiently Extending Transformers for Long Input Summarization
• EfficientFormer: Vision Transformers at MobileNet Speed
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• OPT: Open Pre-trained Transformer Language Models
• Flamingo: a Visual Language Model for Few-Shot Learning
• MeSHup: A Corpus for Full Text Biomedical Document Indexing
• Locating and Editing Factual Associations in GPT
• Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks
• LongT5: Efficient Text-To-Text Transformer for Long Sequences
• AdaViT: Adaptive Tokens for Efficient Vision Transformer
• Self-attention Does Not Need O(n²) Memory
• MViTv2: Improved Multiscale Vision Transformers for Classification and Detection
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• MetaFormer Is Actually What You Need for Vision
• MetaFormer is Actually What You Need for Vision
• Masked Autoencoders Are Scalable Vision Learners
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Sequence Parallelism: Long Sequence Training from System Perspective
• Multiscale Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Rethinking Spatial Dimensions of Vision Transformers
• Scalable Vision Transformers with Hierarchical Pooling
• Learning Transferable Visual Models From Natural Language Supervision
• Long Range Arena: A Benchmark for Efficient Transformers
• Rethinking Attention with Performers
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Linformer: Self-Attention with Linear Complexity
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• VGGSound: A Large-scale Audio-Visual Dataset
• Longformer: The Long-Document Transformer
• Sparse Sinkhorn Attention
• Reformer: The Efficient Transformer
• Fast Transformer Decoding: One Write-Head is All You Need
• RandAugment: Practical automated data augmentation with a reduced search space
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Generating Long Sequences with Sparse Transformers
• mixup: Beyond Empirical Risk Minimization
• Attention Is All You Need
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Deep Networks with Stochastic Depth
• Rethinking the Inception Architecture for Computer Vision
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• A Unified View of Long-Sequence Models towards Modeling Million-Scale Dependencies
• CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling
• Recurrent Memory-Augmented Transformers with Chunked Attention for Long-Context Language Modeling
• Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
• Long-Short Transformer: Efficient Transformers for Language and Vision