ENA: Efficient N-dimensional Attention

2025/08/16 by Zhong, Yibo
Computer Science · Neuroscience · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Brain Tumor Detection and Classification #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2508.11921

openalex publication_date 2025/08/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Efficient modeling of long sequences of high-order data requires a more efficient architecture than Transformer. In this paper, we investigate two key aspects of extending linear recurrent models, especially those originally designed for language modeling, to high-order data (1D to ND): scanning strategies and attention-hybrid architectures. Empirical results suggest that scanning provides limited benefits, while attention-hybrid models yield promising results. Focusing on the latter, we further evaluate types of attention and find that tiled high-order sliding window attention (SWA) is efficient in both theory and practice. We term the resulting hybrid architecture of linear recurrence and high-order SWA as Efficient N-dimensional Attention (ENA). We then conduct several experiments to demonstrate its effectiveness. The intuition behind ENA is that linear recurrence compresses global information into a state, while SWA complements it by enforcing strict local modeling. Together, they form a simple framework that offers a promising and practical solution for ultra-long high-order data modeling.

Citations

• Test-Time Training Done Right
• One-Minute Video Generation with Test-Time Training
• RWKV-7 "Goose" with Expressive Dynamic State Evolution
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
• Fast Video Generation with Sliding Tile Attention
• CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up
• Gated Delta Networks: Improving Mamba2 with Delta Rule
• Flex Attention: A Programming Model for Generating Optimized Attention Kernels
• Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think
• Gated Slot Attention for Efficient Linear-Time Sequence Modeling
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length
• DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention
• HGRN2: Gated Linear RNNs with State Expansion
• Simple linear attention language models balance the recall-throughput tradeoff
• SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers
• Gated Linear Attention Transformers with Hardware-Efficient Training
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Retentive Network: A Successor to Transformer for Large Language Models
• RWKV: Reinventing RNNs for the Transformer Era
• CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• Higher-order Linear Attention
• Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences
• CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling
• Efficient Attention Mechanisms for Large Language Models: A Survey
• Adaptive Multi-Resolution Attention with Linear Complexity