2019/04/03 by Navid Azizan, Babak Hassibi, Azizan, Navid +1
Physics and Astronomy · Mathematics · Computer Science · #Statistical Mechanics and Entropy #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.1904.01855
Stochastic mirror descent (SMD) is a fairly new family of algorithms that has\nrecently found a wide range of applications in optimization, machine learning,\nand control. It can be considered a generalization of the classical stochastic\ngradient algorithm (SGD), where instead of updating the weight vector along the\nnegative direction of the stochastic gradient, the update is performed in a\n"mirror domain" defined by the gradient of a (strictly convex) potential\nfunction. This potential function, and the mirror domain it yields, provides\nconsiderable flexibility in the algorithm compared to SGD. While many\nproperties of SMD have already been obtained in the literature, in this paper\nwe exhibit a new interpretation of SMD, namely that it is a risk-sensitive\noptimal estimator when the unknown weight vector and additive noise are\nnon-Gaussian and belong to the exponential family of distributions. The\nanalysis also suggests a modified version of SMD, which we refer to as\nsymmetric SMD (SSMD). The proofs rely on some simple properties of Bregman\ndivergence, which allow us to extend results from quadratics and Gaussians to\ncertain convex functions and exponential families in a rather seamless way.\n