A Two-Stage Subspace Trust Region Approach for Deep Neural Network Training

2018/05/23 by Dudar, Viacheslav, Chierchia, Giovanni, Chouzenoux, Emilie +2
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1805.09430

Abstract

In this paper, we develop a novel second-order method for training feed-forward neural nets. At each iteration, we construct a quadratic approximation to the cost function in a low-dimensional subspace. We minimize this approximation inside a trust region through a two-stage procedure: first inside the embedded positive curvature subspace, followed by a gradient descent step. This approach leads to a fast objective function decay, prevents convergence to saddle points, and alleviates the need for manually tuning parameters. We show the good performance of the proposed algorithm on benchmark datasets.

Related

• A non-monotone trust-region method with noisy oracles and additional sampling
• Efficient Second-Order Neural Network Optimization via Adaptive Trust Region Methods
• Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods
• Data-Parallel Neural Network Training via Nonlinearly Preconditioned Trust-Region Method
• Training Deep Neural Networks via Branch-and-Bound