2023/07/20 by Nihat Ay, Ay, Nihat, Jesse van Oostrum +2 · 1 citation
Computer Science · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.2307.11249
openalex publication_date 2023/07/20 · openalex created_date 2023/07/26 · openalex updated_date 2026/08/02
This article studies the Fisher-Rao gradient, also referred to as the natural gradient, of the evidence lower bound (ELBO) which plays a central role in generative machine learning. It reveals that the gap between the evidence and its lower bound, the ELBO, has essentially a vanishing natural gradient within unconstrained optimization. As a result, maximization of the ELBO is equivalent to minimization of the Kullback-Leibler divergence from a target distribution, the primary objective function of learning. Building on this insight, we derive a condition under which this equivalence persists even when optimization is constrained to a model. This condition yields a geometric characterization, which we formalize through the notion of a cylindrical model.