Structured second-order methods via natural gradient descent

2021/07/22 by Lin, Wu, Nielsen, Frank, Khan, Mohammad Emtiyaz +1 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2107.10884

Abstract

In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces. Natural-gradient descent is an attractive approach to design new algorithms in many settings such as gradient-free, adaptive-gradient, and second-order methods. Our structured methods not only enjoy a structural invariance but also admit a simple expression. Finally, we test the efficiency of our proposed methods on both deterministic non-convex problems and deep learning problems.

Cited by

• Optimization Guarantees for Square-Root Natural-Gradient Variational Inference

Related

• Tractable structured natural gradient descent using local parameterizations
• Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis
• First-order and second-order variants of the gradient descent in a unified framework
• Revisiting Natural Gradient for Deep Networks
• Natural Gradient Works Efficiently in Learning