2019/01/31 by Jarek Duda, Duda, Jarek · 2 citations
Physics and Astronomy · Computer Science · #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.1901.11457
Deep neural networks are usually trained with stochastic gradient descent\n(SGD), which minimizes objective function using very rough approximations of\ngradient, only averaging to the real gradient. Standard approaches like\nmomentum or ADAM only consider a single direction, and do not try to model\ndistance from extremum - neglecting valuable information from calculated\nsequence of gradients, often stagnating in some suboptimal plateau. Second\norder methods could exploit these missed opportunities, however, beside\nsuffering from very large cost and numerical instabilities, many of them\nattract to suboptimal points like saddles due to negligence of signs of\ncurvatures (as eigenvalues of Hessian).\n Saddle-free Newton method is a rare example of addressing this issue -\nchanges saddle attraction into repulsion, and was shown to provide essential\nimprovement for final value this way. However, it neglects noise while\nmodelling second order behavior, focuses on Krylov subspace for numerical\nreasons, and requires costly eigendecomposion.\n Maintaining SFN advantages, there are proposed inexpensive ways for\nexploiting these opportunities. Second order behavior is linear dependence of\nfirst derivative - we can optimally estimate it from sequence of noisy\ngradients with least square linear regression, in online setting here: with\nweakening weights of old gradients. Statistically relevant subspace is\nsuggested by PCA of recent noisy gradients - in online setting it can be made\nby slowly rotating considered directions toward new gradients, gradually\nreplacing old directions with recent statistically relevant. Eigendecomposition\ncan be also performed online: with regularly performed step of QR method to\nmaintain diagonal Hessian. Outside the second order modeled subspace we can\nsimultaneously perform gradient descent.\n