2020/10/27 by Matthew Evanusa, Snehesh Shrestha, Evanusa, Matthew +7
Computer Science · Engineering · #Advanced Memory and Neural Computing #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #Neural and Evolutionary Computing (cs.NE) #Optical Network Technologies #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.2010.14611
arxiv created 2020/10/27 · openalex publication_date 2020/10/27 · arxiv updated 2020/10/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In many real-world applications, fully-differentiable RNNs such as LSTMs and GRUs have been widely deployed to solve time series learning tasks. These networks train via Backpropagation Through Time, which can work well in practice but involves a biologically unrealistic unrolling of the network in time for gradient updates, are computationally expensive, and can be hard to tune. A second paradigm, Reservoir Computing, keeps the recurrent weight matrix fixed and random. Here, we propose a novel hybrid network, which we call Hybrid Backpropagation Parallel Echo State Network (HBP-ESN) which combines the effectiveness of learning random temporal features of reservoirs with the readout power of a deep neural network with batch normalization. We demonstrate that our new network outperforms LSTMs and GRUs, including multi-layer "deep" versions of these networks, on two complex real-world multi-dimensional time series datasets: gesture recognition using skeleton keypoints from ChaLearn, and the DEAP dataset for emotion recognition from EEG measurements. We show also that the inclusion of a novel meta-ring structure, which we call HBP-ESN M-Ring, achieves similar performance to one large reservoir while decreasing the memory required by an order of magnitude. We thus offer this new hybrid reservoir deep learning paradigm as a new alternative direction for RNN learning of temporal or sequential data.