Long-term Forecasting using Higher Order Tensor RNNs

2017/10/31 by Rose Yu, Stephan Zheng, Yu, Rose +5 · 6 citations
Computer Science · Engineering · Mathematics · #Computational Physics and Python Applications #Energy Load and Power Forecasting #FOS: Computer and information sciences #Machine Learning (cs.LG) #Tensor decomposition and applications

paper · pdf · doi:10.48550/arxiv.1711.00073

openalex publication_date 2017/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Higher-Order Tensor RNN (HOT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher-order moments and higher-order state transition functions. Furthermore, we decompose the higher-order structure using the tensor-train decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation guarantees and the variance bound for HOT-RNN for general sequence inputs. We also demonstrate 5% ~ 12% improvements for long-term prediction over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world time series data.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Tensorizing Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Long Short-Term Memory
• Deep Learning in Neural Networks: An Overview
• Nonparametric Estimation of Multi-View Latent Variable Models
• Unsupervised Learning of Video Representations using LSTMs
• Sequence to Sequence Learning with Neural Networks
• Higher Order Recurrent Neural Networks
• Hierarchical Multiscale Recurrent Neural Networks
• DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks
• Learning to Reason with Third-Order Tensor Products
• Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large\n Vocabulary Speech Recognition
• Exponential Families for Conditional Random Fields

Cited by

• A Scalable Factorization Approach for High-Order Structured Tensor Recovery
• Foundations of Sequence-to-Sequence Modeling for Time Series
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• LETS Forecast: Learning Embedology for Time Series Forecasting
• Tensorized Transformer for Dynamical Systems Modeling
• Block Hankel Tensor ARIMA for Multiple Short Time Series Forecasting

Related

• ReNF: Rethinking the Design of Neural Long-Term Time Series Forecasters
• SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting
• Backpropagation Algorithms and Reservoir Computing in Recurrent Neural\n Networks for the Forecasting of Complex Spatiotemporal Dynamics
• Multivariate Long-term Time Series Forecasting with Fourier Neural Filter
• Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems