2025/10/21 by Felix Michalak, Steven Abreu, Michalak, Felix +1
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #FOS: Computer and information sciences #Information Retrieval and Search Behavior #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2510.19861
openalex publication_date 2025/10/21 · openalex created_date 2025/10/25 · openalex updated_date 2026/07/28
We demonstrate complete functional segregation in hybrid SSM-Transformer architectures: retrieval depends exclusively on self-attention layers. Across RecurrentGemma-2B/9B and Jamba-Mini-1.6, attention ablation causes catastrophic retrieval failure (0% accuracy), while SSM layers show no compensatory mechanisms even with improved prompting. Conversely, sparsifying attention to just 15% of heads maintains near-perfect retrieval while preserving 84% MMLU performance, suggesting self-attention specializes primarily for retrieval tasks. We identify precise mechanistic requirements for retrieval: needle tokens must be exposed during generation and sufficient context must be available during prefill or generation. This strict functional specialization challenges assumptions about redundancy in hybrid architectures and suggests these models operate as specialized modules rather than integrated systems, with immediate implications for architecture optimization and interpretability.