2025/05/27 by Mónika Farsang, Farsang, Mónika, Ramin Hasani +5 · 4 citations
Computer Science · Materials Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science #Model Reduction and Neural Networks #Neural and Evolutionary Computing (cs.NE) #Parallel Computing and Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2505.21717
openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
We present LrcSSM, a non-linear recurrent model that processes long sequences as fast as today's linear state-space layers. By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel, giving O(TD) computational work and memory and only O(log T) sequential depth, for input-sequence length T and a state dimension D. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Importantly, the diagonal Jacobian structure of our model results in no performance loss compared to the original model with dense Jacobian, and the approach can be generalized to other non-linear recurrent models, demonstrating broader applicability. On a suite of long-range forecasting tasks, we demonstrate that LrcSSM outperforms Transformers, LRU, S5, and Mamba.