2020/01/14 by Dominic Kafka, Kafka, Dominic, Daniël N. Wilke +1
Computer Science · #49M05 #65K05 #90C15 #90C26 #FOS: Computer and information sciences #FOS: Mathematics #I.2.6 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2001.05113
openalex publication_date 2020/01/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Learning rates in stochastic neural network training are currently determined\na priori to training, using expensive manual or automated iterative tuning.\nThis study proposes gradient-only line searches to resolve the learning rate\nfor neural network training algorithms. Stochastic sub-sampling during training\ndecreases computational cost and allows the optimization algorithms to progress\nover local minima. However, it also results in discontinuous cost functions.\nMinimization line searches are not effective in this context, as they use a\nvanishing derivative (first order optimality condition), which often do not\nexist in a discontinuous cost function and therefore converge to\ndiscontinuities as opposed to minima from the data trends. Instead, we base\ncandidate solutions along a search direction purely on gradient information, in\nparticular by a directional derivative sign change from negative to positive (a\nNon-negative Associative Gradient Projection Point (NN- GPP)). Only considering\na sign change from negative to positive always indicates a minimum, thus\nNN-GPPs contain second order information. Conversely, a vanishing gradient is\npurely a first order condition, which may indicate a minimum, maximum or saddle\npoint. This insight allows the learning rate of an algorithm to be reliably\nresolved as the step size along a search direction, increasing convergence\nperformance and eliminating an otherwise expensive hyperparameter.\n