DeepSITH: Efficient Learning via Decomposition of What and When Across\n Time Scales

2021/04/09 by Brandon G. Jacques, Zoran Tiganj, Jacques, Brandon +5 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Music and Audio Processing #Neural Networks and Applications #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2104.04646

openalex publication_date 2021/04/09 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Extracting temporal relationships over a range of scales is a hallmark of\nhuman perception and cognition -- and thus it is a critical feature of machine\nlearning applied to real-world problems. Neural networks are either plagued by\nthe exploding/vanishing gradient problem in recurrent neural networks (RNNs) or\nmust adjust their parameters to learn the relevant time scales (e.g., in\nLSTMs). This paper introduces DeepSITH, a network comprising\nbiologically-inspired Scale-Invariant Temporal History (SITH) modules in series\nwith dense connections between layers. SITH modules respond to their inputs\nwith a geometrically-spaced set of time constants, enabling the DeepSITH\nnetwork to learn problems along a continuum of time-scales. We compare DeepSITH\nto LSTMs and other recent RNNs on several time series prediction and decoding\ntasks. DeepSITH achieves state-of-the-art performance on these problems.\n

Citations

• On the difficulty of training Recurrent Neural Networks
• Long Short-Term Memory

Cited by

• Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows

Related

• A deep convolutional neural network that is invariant to time rescaling
• A Survey: Time Travel in Deep Learning Space: An Introduction to Deep Learning Models and How Deep Learning Models Evolved from the Initial Ideas
• Scale-invariant temporal history (SITH): optimal slicing of the past in an uncertain world
• TimeNet: Pre-trained deep recurrent neural network for time series classification
• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers