2025/05/09 by Andrew Kiruluta, Preethi Raju, Kiruluta, Andrew +3 · 6 voices · 1 citation
Computer Science · #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Computation #Context (archaeology) #Convolution (computer science) #Dilation (metric space) #Image and Object Detection Techniques #Quadratic equation #Security token #Sequence (biology) #State (computer science) #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2506.01963
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/05/09 · openalex created_date 2025/10/14 · openalex updated_date 2026/08/05
We present a novel non attention based architecture for large language models (LLMs) that efficiently handles very long context windows, on the order of hundreds of thousands to potentially millions of tokens. Unlike traditional Transformer designs, which suffer from quadratic memory and computation overload due to the nature of the self attention mechanism, our model avoids token to token attention entirely. Instead, it combines the following complementary components: State Space blocks (inspired by S4) that learn continuous time convolution kernels and scale near linearly with sequence length, Multi Resolution Convolution layers that capture local context at different dilation levels, a lightweight Recurrent Supervisor to maintain a global hidden state across sequential chunks, and Retrieval Augmented External Memory that stores and retrieves high-level chunk embeddings without reintroducing quadratic operations.