Low-Cost Recurrent Neural Network Expected Performance Evaluation

2018/05/18 by Andrés Camero, Camero, Andrés, Jamal Toutouh +3 · 5 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Machine Learning and Data Classification #Neural Networks and Applications #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1805.07159

arxiv created 2019/03/11 · arxiv updated 2019/03/12

Abstract

Recurrent neural networks are a powerful tool, but they are very sensitive to their hyper-parameter configuration. Moreover, training properly a recurrent neural network is a tough task, therefore selecting an appropriate configuration is critical. Varied strategies have been proposed to tackle this issue. However, most of them are still impractical because of the time/resources needed. In this study, we propose a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples of the weights. We empirically validate our proposal using three use cases. The results suggest that this is a promising alternative to reduce the cost of exploration for hyper-parameter optimization.

Cited by

• DLOPT: Deep Learning Optimization Library
• An Empirical Exploration of Deep Recurrent Connections and Memory Cells Using Neuro-Evolution
• Reliable and Fast Recurrent Neural Network Architecture Optimization
• GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks
• An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

Related

• Accelerating Neural Architecture Search using Performance Prediction
• Hyperparameter Optimisation with Early Termination of Poor Performers
• Reliable and Fast Recurrent Neural Network Architecture Optimization
• Efficient Neural Architecture Search with Performance Prediction
• HyperPower: Power- and Memory-Constrained Hyper-Parameter Optimization for Neural Networks