WaLRUS: Wavelets for Long-range Representation Using SSMs

2025/05/17 by Hossein Babaei, Babaei, Hossein, White, Mel +4
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Fault Diagnosis Techniques #Machine Learning (cs.LG) #Machine Learning in Healthcare #Signal Processing (eess.SP) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.12161

openalex publication_date 2025/05/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

State-Space Models (SSMs) have proven to be powerful tools for modeling long-range dependencies in sequential data. While the recent method known as HiPPO has demonstrated strong performance, and formed the basis for machine learning models S4 and Mamba, it remains limited by its reliance on closed-form solutions for a few specific, well-behaved bases. The SaFARi framework generalized this approach, enabling the construction of SSMs from arbitrary frames, including non-orthogonal and redundant ones, thus allowing an infinite diversity of possible "species" within the SSM family. In this paper, we introduce WaLRUS (Wavelets for Long-range Representation Using SSMs), a new implementation of SaFARi built from Daubechies wavelets.

Citations

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Liquid Structural State-Space Models
• Simplified State Space Layers for Sequence Modeling
• How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections
• On the Parameterization and Initialization of Diagonal State Space Models
• Diagonal State Spaces are as Effective as Structured State Spaces
• Efficiently Modeling Long Sequences with Structured State Spaces
• Wearing a MASK: Compressed Representations of Variable-Length Sequences\n Using Recurrent Neural Tangent Kernels
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Google Speech Commands-Musan test set
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries
• Framewise phoneme classification with bidirectional LSTM and other neural network architectures
• Long Short-Term Memory
• Bidirectional recurrent neural networks
• Ideal spatial adaptation by wavelet shrinkage
• Finding Structure in Time
• Orthonormal bases of compactly supported wavelets
• The M4 Competition: 100,000 time series and 61 forecasting methods

Related

• W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling
• SaFARi: State-Space Models for Frame-Agnostic Representation
• From S4 to Mamba: A Comprehensive Survey on Structured State Space Models
• How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections
• Vision Mamba: A Comprehensive Survey and Taxonomy