Learning-Rate-Free Learning: Dissecting D-Adaptation and Probabilistic Line Search

2023/08/06 by McGuinness, Max
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2308.03102

Abstract

This paper explores two recent methods for learning rate optimisation in stochastic gradient descent: D-Adaptation (arXiv:2301.07733) and probabilistic line search (arXiv:1502.02846). These approaches aim to alleviate the burden of selecting an initial learning rate by incorporating distance metrics and Gaussian process posterior estimates, respectively. In this report, I provide an intuitive overview of both methods, discuss their shared design goals, and devise scope for merging the two algorithms.

Related

• Explainable Learning Rate Regimes for Stochastic Optimization
• A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
• Statistical Adaptive Stochastic Gradient Methods
• Gradient-only line searches: An Alternative to Probabilistic Line Searches
• Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms