Expectation-maximization for logistic regression

2013/05/31 by Scott, James G., Sun, Liang
#Computation (stat.CO) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.1306.0040

Abstract

We present a family of expectation-maximization (EM) algorithms for binary and negative-binomial logistic regression, drawing a sharp connection with the variational-Bayes algorithm of Jaakkola and Jordan (2000). Indeed, our results allow a version of this variational-Bayes approach to be re-interpreted as a true EM algorithm. We study several interesting features of the algorithm, and of this previously unrecognized connection with variational Bayes. We also generalize the approach to sparsity-promoting priors, and to an online method whose convergence properties are easily established. This latter method compares favorably with stochastic-gradient descent in situations with marked collinearity.

Related

• Parameter-Expanded ECME Algorithms for Logistic and Penalized Logistic Regression
• Logistic Variational Bayes Revisited
• The normalized expectation-maximization (N-EM) algorithm
• EM algorithm and variants: an informal tutorial
• Optimal and computationally tractable lower bounds for logistic log-likelihoods