2024/07/09 by Kai Biegun, Biegun, Kai, Rares Dolga +5 · 1 citation
Computer Science · Earth and Planetary Sciences · #Computational Physics and Python Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Medical Image Segmentation Techniques #Seismic Imaging and Inversion Techniques
paper · pdf · doi:10.48550/arxiv.2407.07239
openalex publication_date 2024/07/09 · openalex created_date 2024/07/13 · openalex updated_date 2026/07/28
Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, their empirical performance is not well understood and they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN -- a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple and efficient model with a robust normalisation procedure, and a practical implementation that remains faithful to its theoretical derivation. RotRNN also achieves competitive performance to state-of-the-art linear recurrent models on several long sequence modelling datasets.