ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoning

2025/10/02 by Haochen You, You, Haochen, Baojing Liu +1
Computer Science · #AI-based Problem Solving and Planning #Bounded function #Computation and Language (cs.CL) #Encoder #FLOPS #FOS: Computer and information sciences #Generalization #Inference #Iterative refinement #Logic, Reasoning, and Knowledge #Networking and Internet Architecture (cs.NI) #Scalability #Semantic Web and Ontologies #Transformer

paper · pdf · doi:10.48550/arxiv.2510.01585

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While Transformer architectures have demonstrated impressive scalability across domains, they continue to face challenges in long-context reasoning, computational efficiency, and structural generalization - largely due to rigid layer stacking, dense attention, and reliance on positional encodings. We present ReSSFormer, a Recursive Sparse Structured Transformer that integrates three complementary innovations: Recurrent Reasoning & Memory Unit (R2MU) for iterative reasoning with bounded depth, Adaptive Sparse Attention Module (ASAM) for efficient and focused context selection, and Self-Organizing Encoder Structure (SOES) for position-free structure induction. ReSSFormer replaces conventional depth stacking with recurrent inference, substitutes full attention with token- and expert-level sparsity, and models latent token topology directly from content. Across language modeling, multi-hop QA, and structure-sensitive tasks, ReSSFormer consistently outperforms strong baselines under comparable FLOPs and parameter budgets, highlighting its scalability, efficiency, and structural flexibility.

Citations

• Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modeling
• MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization
• Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting
• SepPrune: Structured Pruning for Efficient Deep Speech Separation
• Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
• TRA: Better Length Generalisation with Threshold Relative Attention
• Enhancing RWKV-based Language Models for Long-Sequence Text Generation
• SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models
• MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
• Efficient Sparse Attention needs Adaptive Token Release
• Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
• COMAE: COMprehensive Attribute Exploration for Zero-shot Hashing
• Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention
• On the Stability of Expressive Positional Encodings for Graphs
• Enhancing Graph Transformers with Hierarchical Distance Structural Encoding
• CoLT5: Faster Long-Range Transformers with Conditional Computation
• Multimodal Learning with Transformers: A Survey
• Training Verifiers to Solve Math Word Problems
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings
• Rethinking Attention with Performers
• Longformer: The Long-Document Transformer
• Scaling Laws for Neural Language Models
• Reformer: The Efficient Transformer
• Compressive Transformers for Long-Range Sequence Modelling
• Syntax-aware Neural Semantic Role Labeling
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Self-Attention with Relative Position Representations
• The NarrativeQA Reading Comprehension Challenge

Related

• RecurFormer: Not All Transformer Heads Need Self-Attention
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
• Scaling Linear Attention with Sparse State Expansion
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
• Rectified Sparse Attention