EA-CG: An Approximate Second-Order Method for Training Fully-Connected Neural Networks

2018/02/19 by Sheng-Wei Chen, Chen, Sheng-Wei, Chun‐Nan Chou +3
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1802.06502

openalex publication_date 2018/02/19 · openalex created_date 2018/12/11 · openalex updated_date 2026/07/28

Abstract

For training fully-connected neural networks (FCNNs), we propose a practical approximate second-order method including: 1) an approximation of the Hessian matrix and 2) a conjugate gradient (CG) based method. Our proposed approximate Hessian matrix is memory-efficient and can be applied to any FCNNs where the activation and criterion functions are twice differentiable. We devise a CG-based method incorporating one-rank approximation to derive Newton directions for training FCNNs, which significantly reduces both space and time complexity. This CG-based method can be employed to solve any linear equation where the coefficient matrix is Kronecker-factored, symmetric and positive definite. Empirical studies show the efficacy and efficiency of our proposed method.

Related

• Block-diagonal Hessian-free Optimization for Training Neural Networks
• Second-order Neural Network Training Using Complex-step Directional Derivative
• Practical Quasi-Newton Methods for Training Deep Neural Networks
• A Tutorial on Neural Networks and Gradient-free Training
• A Neural Network Training Method Based on Neuron Connection Coefficient Adjustments