Adaptive Hierarchical Hyper-gradient Descent

2020/08/17 by Renlong Jie, Jie, Renlong, Junbin Gao +7
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2008.07277

openalex publication_date 2020/08/17 · arxiv created 2021/05/11 · arxiv updated 2021/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this study, we investigate learning rate adaption at different levels based on the hyper-gradient descent framework and propose a method that adaptively learns the optimizer parameters by combining multiple levels of learning rates with hierarchical structures. Meanwhile, we show the relationship between regularizing over-parameterized learning rates and building combinations of adaptive learning rates at different levels. The experiments on several network architectures, including feed-forward networks, LeNet-5 and ResNet-18/34, show that the proposed multi-level adaptive approach can outperform baseline adaptive methods in a variety of circumstances.

Citations

• Gradient-based learning applied to document recognition
• An overview of gradient descent optimization algorithms
• Learned Optimizers that Scale and Generalize
• L4: Practical loss-based stepsize adaptation for deep learning
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• On the Convergence of AdaBound and its Connection to SGD
• Learning Multiple Layers of Features from Tiny Images

Related

• Differentiable Self-Adaptive Learning Rate
• Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
• Disentangling Adaptive Gradient Methods from Learning Rates
• Online Learning Rate Adaptation with Hypergradient Descent
• AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio