A Tutorial on MM Algorithms
2004/02/01 by David R Hunter, David R. Hunter, Kenneth Lange · 112 citations
Mathematics · Decision Sciences · #Advanced Statistical Methods and Models #Statistical Methods and Bayesian Inference #Optimal Experimental Design Methods
paper · doi:10.1198/0003130042836
Abstract
Most problems in frequentist statistics involve optimization of a function such as a likelihood or a sum of squares. EM algorithms are among the most effective algorithms for maximum likelihood estimation because they consistently drive the likelihood uphill by maximizing a simple surrogate function for the log-likelihood. Iterative optimization of a surrogate function as exemplified by an EM algorithm does not necessarily require missing data. Indeed, every EM algorithm is a special case of the more general class of MM optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. In our opinion, MM algorithms deserve to be part of the standard toolkit of professional statisticians. This article explains the principle behind MM algorithms, suggests some methods for constructing them, and discusses some of their attractive features. We include numerous examples throughout the article to illustrate the concepts described. In addition to surveying previous work on MM algorithms, this article introduces some new material on constrained optimization and standard error estimation.
Citations
Cited by
- Universal Filtered Multi-Carrier Transmission With Adaptive Active Interference Cancellation
- A Note on the Convergence of the Gaussian Mean Shift Algorithm
- Efficient Low-Rank Semidefinite Programming with Robust Loss Functions
- Robust Maximum Likelihood Estimation of Sparse Vector Error Correction\n Model
- Pseudo-Bayesian Robust PCA: Algorithms and Analyses
- The Dynamic ECME Algorithm
- Symbol-Level Precoding for Integrated Sensing and Covert Communication
- Simultaneous penalized M-estimation of covariance matrices using\n geodesically convex optimization
- Secure Beamforming Design for IRS-ISAC Systems with a Hardware-Efficient Hybrid Beamforming Architecture
- Sparse Distance Weighted Discrimination
- Regularized Reduced Rank Regression for mixed predictor and response variables
- Accurate, Interpretable, and Fast Animation: An Iterative, Sparse, and Nonconvex Approach
- Constant-Modulus Secure Analog Beamforming for an IRS-Assisted Communication System with Large-Scale Antenna Array
- Conic Quadratic Formulations for Wireless Communications Design
- Kernel Density Estimation-Based Markov Models with Hidden State
- Fast Semidifferential-based Submodular Function Optimization
- A redescending-weight curvature regularization model for edge-preserving signal denoising
- Near-optimal Linear Predictive Clustering in Non-separable Spaces via Mixed Integer Programming and Quadratic Pseudo-Boolean Reductions
- Beamforming Design for Joint Localization and Data Transmission in Distributed Antenna System
- A Majorization-Minimization Algorithm for Computing the Karcher Mean of Positive Definite Matrices
- A Latent-Variable Formulation of the Poisson Canonical Polyadic Tensor Model: Maximum Likelihood Estimation and Fisher Information
- Rates of Convergence of Maximum Smoothed Log-Likelihood Estimators for Semi-Parametric Multivariate Mixtures
- Sinusoidal Parameter Estimation from Signed Measurements via Majorization-Minimization Based RELAX
- Theoretical Grounding for Estimation in Conditional Independence Multivariate Finite Mixture Models
- Demand Estimation from Sales Transaction Data -- Practical Extensions
- A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing
- Hybrid Transceivers Design for Large-Scale Antenna Arrays Using Majorization-Minimization Algorithms
- Multiresolution Tensor Decomposition for Multiple Spatial Passing Networks
- Novel Algorithms based on Majorization Minimization for Nonnegative Matrix Factorization
- Block-wise Minimization-Majorization algorithm for Huber's criterion: sparse learning and applications
- PrivateMail: Supervised Manifold Learning of Deep Features With Differential Privacy for Image Retrieval
- Design of polyphase sequence sets with good correlation properties under spectral distortion via majorization-minimization framework
- Provably Convergent Working Set Algorithm for Non-Convex Regularized Regression
- I-LAMM for Sparse Learning: Simultaneous Control of Algorithmic Complexity and Statistical Error
- Nonsmooth Analysis and Subgradient Methods for Averaging in Dynamic Time\n Warping Spaces
- Gaining Outlier Resistance With Progressive Quantiles: Fast Algorithms and Theoretical Studies
- Fast Image Recovery Using Variable Splitting and Constrained Optimization
- Penalized Cox’s proportional hazards model for high-dimensional survival data with grouped predictors
- Sequence Design to Minimize the Weighted Integrated and Peak Sidelobe Levels
- Computational methods for large-scale inverse problems: a survey on hybrid projection methods
- SolarBoost: Distributed Photovoltaic Power Forecasting Amid Time-varying Grid Capacity
- Hierarchical Overlapping Group Lasso for GMANOVA Model
- Non-Orthogonal Pilot Sequence Design for Multi-Cells Interference Networks
- Robust Design for Intelligent Reflecting Surfaces Assisted MISO Systems
- Randomized flexible Krylov methods for ℓp regularization
- Zero-Temperature Limit of a Convergent Algorithm to Minimize the Bethe\n Free Energy
- Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
- Generalized simultaneous component analysis of binary and quantitative data
- Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors
- Truncated Inference for Latent Variable Optimization Problems: Application to Robust Estimation and Learning
- Private Posterior distributions from Variational approximations
- Model-Based Clustering of Nonparametric Weighted Networks With Application to Water Pollution Analysis
- Algorithms for nonnegative matrix factorization with the beta-divergence
- A Unified Approach for Learning the Parameters of Sum-Product Networks
- Classification Filtering
- Non-orthogonal Multiple Access Assisted Multi-Region Geocast
- Revisiting Bayesian Blind Deconvolution
- Clustering Data with Nonignorable Missingness using Semi-Parametric Mixture Models
- Genomic Region Detection via Spatial Convex Clustering
- Pilot Spoofing Attack by Multiple Eavesdroppers
- Adaptively Robust Geographically Weighted Regression
- Channel Estimation and Data Detection in DS-Spread Channels: A Unified Framework, Novel Algorithms, and Waveform Comparison
- Efficient Rank Minimization via Solving Non-convexPenalties by Iterative Shrinkage-Thresholding Algorithm
- On Minimization/Maximization of the Generalized Multi-Order Complex Quadratic Form With Constant-Modulus Constraints
- New Methods for MLE of Toeplitz Structured Covariance Matrices with\n Applications to RADAR Problems
- Variational Algorithms for Marginal MAP
- Proactive Constrained Policy Optimization with Preemptive Penalty
- Energy-Efficient Cell Activation, User Association, and Spectrum Allocation in Heterogeneous Networks
- SparseStep: Approximating the Counting Norm for Sparse Regularization
- Robust Binary Fused Compressive Sensing using Adaptive Outlier Pursuit
- Binary Fused Compressive Sensing: 1-Bit Compressive Sensing meets Group Sparsity
- Iterative Shrinkage Approach to Restoration of Optical Imagery
- Hash Function Learning via Codewords
- MIMO Transmit Beampattern Matching Under Waveform Constraints
- Group Sparse Beamforming for Green Cloud-RAN
- Overcomplete Independent Component Analysis via SDP
- Network Slicing for Service-Oriented Networks Under Resource Constraints
- A Fast Algorithm for Convolutional Structured Low-Rank Matrix Recovery
- Maximum likelihood estimation of burst-merging kernels for bursty time series
- Regularized k-POD: Sparse k-means clustering for high-dimensional missing data
- Escaping Poor Local Minima in Large Scale Robust Estimation
- Partially Observed Markov Decision Processes. Problem Sets and Internet\n Supplement
- Total Variation, Adaptive Total Variation and Nonconvex Smoothly Clipped Absolute Deviation Penalty for Denoising Blocky Images
- Robust Simultaneous Localization of Nodes and Targets in Sensor Networks Using Range-Only Measurements
- Iteratively Reweighted \ℓ1 Approaches to Sparse Composite\n Regularization
- Exploiting Cognition in ISAR Processing for Spectral Compatibility Applications
- Latent Association Mining in Binary Data
- Cluster and propensity based approximation of a network. [europepmc]
- A method for calling copy number polymorphism using haplotypes. [europepmc]
- Convex clustering: an attractive alternative to hierarchical clustering. [europepmc]
- PAFit: A Statistical Method for Measuring Preferential Attachment in Temporal Complex Networks. [europepmc]
- Structure-adaptive CBCT reconstruction using weighted total variation and Hessian penalties. [europepmc]
- Link predication based on matrix factorization by fusion of multi class organizations of the network. [europepmc]
- Variance Component Selection With Applications to Microbiome Taxonomic Data. [europepmc]
- Genomic region detection via Spatial Convex Clustering. [europepmc]
- Markov Model-Based Method to Analyse Time-Varying Networks in EEG Task-Related Data. [europepmc]
- Robust Relative Error Estimation. [europepmc]
- Informed Weighted Non-Negative Matrix Factorization Using αβ -Divergence Applied to Source Apportionment. [europepmc]
- Efficient Low-PAR Waveform Design Method for Extended Target Estimation Based on Information Theory in Cognitive Radar. [europepmc]
- Robust Bayesian Regression with Synthetic Posterior Distributions. [europepmc]
- A Unified Formulation of k-Means, Fuzzy c-Means and Gaussian Mixture Model by the Kolmogorov-Nagumo Average. [europepmc]
- Efficient ReML inference in variance component mixed models using a Min-Max algorithm. [europepmc]
- VCSEL: PRIORITIZING SNP-SET BY PENALIZED VARIANCE COMPONENT SELECTION. [europepmc]
- Model-based clustering for random hypergraphs. [europepmc]
- A genome-wide analysis suggests pleiotropic effects of Green Revolution genes on shade avoidance in wheat. [europepmc]
- Multisensor Estimation Fusion on Statistical Manifold. [europepmc]
- Poisson Phase Retrieval in Very Low-count Regimes. [europepmc]
- High-Performance Statistical Computing in the Computing Environments of the 2020s. [europepmc]
- A Legacy of EM Algorithms. [europepmc]
- The Convex Mixture Distribution: Granger Causality for Categorical Time Series. [europepmc]
- Improved nonparametric survival prediction using CoxPH, Random Survival Forest & DeepHit Neural Network. [europepmc]
- MMORF-FSL's MultiMOdal Registration Framework. [europepmc]
Related