Bayesian optimization of hyper-parameters in reservoir computing

2016/11/16 by J. Yperman, Jan Yperman, Yperman, Jan +2 · 4 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Neural Networks and Reservoir Computing #cs.LG

paper · pdf · doi:10.48550/arxiv.1611.05193

openalex publication_date 2016/11/16 · arxiv created 2017/06/14 · arxiv updated 2017/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We describe a method for searching the optimal hyper-parameters in reservoir computing, which consists of a Gaussian process with Bayesian optimization. It provides an alternative to other frequently used optimization methods such as grid, random, or manual search. In addition to a set of optimal hyper-parameters, the method also provides a probability distribution of the cost function as a function of the hyper-parameters. We apply this method to two types of reservoirs: nonlinear delay nodes and echo state networks. It shows excellent performance on all considered benchmarks, either matching or significantly surpassing results found in the literature. In general, the algorithm achieves optimal results in fewer iterations when compared to other optimization methods. We have optimized up to six hyper-parameters simultaneously, which would have been infeasible using, e.g., grid search. Due to its automated nature, this method significantly reduces the need for expert knowledge when optimizing the hyper-parameters in reservoir computing. Existing software libraries for Bayesian optimization, such as Spearmint, make the implementation of the algorithm straightforward. A fork of the Spearmint framework along with a tutorial on how to use it in practice is available at https://bitbucket.org/uhasseltmachinelearning/spearmint/

Citations

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Bidirectional recurrent neural networks
• New results on recurrent network training: unifying the algorithms and accelerating convergence
• Bayesian Optimization with Exponential Convergence

Cited by

• Empirical Investigation into Configuring Echo State Networks for Representative Benchmark Problem Domains
• Tri-Learn Graph Fusion Network for Attributed Graph Clustering
• Attractor reconstruction in attracting subspaces: Slow-spectrum preshaping for reservoir computing under partial observation
• Multifunctionality in a reservoir computer
• Boosting Reservoir Computing with Brain-inspired Adaptive Dynamics

Related

• Efficient Optimisation of Physical Reservoir Computers using only a Delayed Input
• Bayesian optimisation of large-scale photonic reservoir computers
• Optimal nonlinear information processing capacity in delay-based reservoir computers
• Reservoir optimization and Machine Learning methods
• Tuning the activation function to optimize the forecast horizon of a reservoir computer