Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained Recurrent Neural Networks

2025/08/28 by Pinna, Matteo, Ceni, Andrea, Gallicchio, Claudio
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.21172

Abstract

Echo State Networks (ESNs) are a particular type of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular for their fast and efficient learning. However, traditional ESNs often struggle with long-term information processing. In this paper, we introduce a novel class of deep untrained RNNs based on temporal residual connections, called Deep Residual Echo State Networks (DeepResESNs). We show that leveraging a hierarchy of untrained residual recurrent layers significantly boosts memory capacity and long-term temporal modeling. For the temporal residual connections, we consider different orthogonal configurations, including randomly generated and fixed-structure configurations, and we study their effect on network dynamics. A thorough mathematical analysis outlines necessary and sufficient conditions to ensure stable dynamics within DeepResESN. Our experiments on a variety of time series tasks showcase the advantages of the proposed approach over traditional shallow and deep RC.

Citations

• Edge of stability echo state networks
• Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks
• The UEA multivariate time series classification archive, 2018
• The UCR Time Series Archive
• Dilated Recurrent Neural Networks
• Attention Is All You Need
• Deep Residual Learning for Image Recognition
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Optimization and applications of echo state networks with leaky- integrator neurons
• Learning long-term dependencies with gradient descent is difficult
• Minimum Complexity Echo State Network

Related

• Deep Echo State Network (DeepESN): A Brief Survey
• Residual Reservoir Memory Networks
• Deep-ESN: A Multiple Projection-encoding Hierarchical Reservoir Computing Framework
• Residual Memory Networks: Feed-forward approach to learn long temporal dependencies
• Towards a Comprehensive Theory of Reservoir Computing