2019/02/11 by Ehsan Amid, Amid, Ehsan, Manfred K. Warmuth +1
Computer Science · #Gaussian Processes and Bayesian Inference #Bayesian Modeling and Causal Inference #Target Tracking and Data Fusion in Sensor Networks
paper · pdf · doi:10.48550/arxiv.1902.04107
Expectation-Maximization (EM) is a prominent approach for parameter\nestimation of hidden (aka latent) variable models. Given the full batch of\ndata, EM forms an upper-bound of the negative log-likelihood of the model at\neach iteration and updates to the minimizer of this upper-bound. We first\nprovide a "model level" interpretation of the EM upper-bound as sum of relative\nentropy divergences to a set of singleton models, induced by the set of\nobservations. Our alternative motivation unifies the "observation level" and\nthe "model level" view of the EM. As a result, we formulate an online version\nof the EM algorithm by adding an analogous inertia term which corresponds to\nthe relative entropy divergence to the old model. Our motivation is more widely\napplicable than the previous approaches and leads to simple online updates for\nmixture of exponential distributions, hidden Markov models, and the first known\nonline update for Kalman filters. Additionally, the finite sample form of the\ninertia term lets us derive online updates when there is no closed-form\nsolution. Finally, we extend the analysis to the distributed setting where we\nmotivate a systematic way of combining multiple hidden variable models.\nExperimentally, we validate the results on synthetic as well as real-world\ndatasets.\n