Effectively Modeling Time Series with Simple Discrete State Spaces

2023/03/16 by Michael Zhang, Khaled K. Saab, Zhang, Michael +9 · 10 citations
Computer Science · Decision Sciences · #Advanced Text Analysis Techniques #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stock Market Forecasting Methods #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2303.09489

openalex publication_date 2023/03/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations cannot express autoregressive time series processes. We thus introduce SpaceTime, a new state-space time series architecture that improves all three criteria. For expressivity, we propose a new SSM parameterization based on the companion matrix -- a canonical representation for discrete-time processes -- which enables SpaceTime's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SpaceTime to predict many future time-steps by generating its own layer-wise inputs. For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix. With sequence length ℓ and state-space size d, we go from O(d ℓ) naïvely to O(d + ℓ). In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks, with best or second-best AUROC on 6 / 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SpaceTime (1) fits AR(p) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art, and (3) speeds up training on real-world ETTh1 data by 73% and 80% relative wall-clock time over Transformers and LSTMs.

Cited by

• The Seismic Wavefield Common Task Framework
• KOSS: Kalman-Optimal Selective State Spaces for Long-Term Sequence Modeling
• Delays in Spiking Neural Networks: A State Space Model Approach
• VitalBench: A Rigorous Multi-Center Benchmark for Long-Term Vital Sign Prediction in Intraoperative Care
• Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis
• Common Task Framework For a Critical Evaluation of Scientific Machine Learning Algorithms
• StructuralDecompose: A Modular Framework for Robust Time Series Decomposition in R
• State Space Models over Directed Graphs
• Uncovering the Spectral Bias in Diagonal State Space Models
• Hypergraph Neural Network with State Space Models for Node Classification

Related

• Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting
• SDE: A Simplified and Disentangled Dependency Encoding Framework for State Space Models in Time Series Forecasting
• Stanza: A Nonlinear State Space Model for Probabilistic Inference in Non-Stationary Time Series
• Deep Latent State Space Models for Time-Series Generation
• Tailored Architectures for Time Series Forecasting: Evaluating Deep Learning Models on Gaussian Process-Generated Data