A Common Framework for Natural Gradient and Taylor based Optimisation using Manifold Theory

2018/03/26 by Adnan Haider, Haider, Adnan
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1803.09791

arxiv created 2018/10/03 · arxiv updated 2018/10/04

Abstract

This technical report constructs a theoretical framework to relate standard Taylor approximation based optimisation methods with Natural Gradient (NG), a method which is Fisher efficient with probabilistic models. Such a framework will be shown to also provide mathematical justification to combine higher order methods with the method of NG.

Related

• Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis
• New insights and perspectives on the natural gradient method
• Inversion-Free Natural Gradient Descent on Riemannian Manifolds
• A Formalization of The Natural Gradient Method for General Similarity Measures
• Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems