A Tutorial on MM Algorithms
2004/02/01 by David R Hunter, David R. Hunter, Kenneth Lange · 1,807 citations
Decision Sciences · Mathematics · #Advanced Statistical Methods and Models #Algorithm #Artificial intelligence #Bayesian inference #Bayesian probability #Computer science #Estimation theory #Frequentist inference #Function (biology) #Likelihood function #Machine learning #Mathematical optimization #Mathematics #Missing data #Optimal Experimental Design Methods #Statistical Methods and Bayesian Inference
paper · doi:10.1198/0003130042836
published in The American Statistician 58(1), 30-37 (Taylor & Francis)
openalex publication_date 2004/02/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04
Abstract
Most problems in frequentist statistics involve optimization of a function such as a likelihood or a sum of squares. EM algorithms are among the most effective algorithms for maximum likelihood estimation because they consistently drive the likelihood uphill by maximizing a simple surrogate function for the log-likelihood. Iterative optimization of a surrogate function as exemplified by an EM algorithm does not necessarily require missing data. Indeed, every EM algorithm is a special case of the more general class of MM optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. In our opinion, MM algorithms deserve to be part of the standard toolkit of professional statisticians. This article explains the principle behind MM algorithms, suggests some methods for constructing them, and discusses some of their attractive features. We include numerous examples throughout the article to illustrate the concepts described. In addition to surveying previous work on MM algorithms, this article introduces some new material on constrained optimization and standard error estimation.
Citations
Cited by
- Universal Filtered Multi-Carrier Transmission With Adaptive Active Interference Cancellation
- A Note on the Convergence of the Gaussian Mean Shift Algorithm
- Efficient Low-Rank Semidefinite Programming with Robust Loss Functions
- Robust Maximum Likelihood Estimation of Sparse Vector Error Correction\n Model
- Pseudo-Bayesian Robust PCA: Algorithms and Analyses
- The Dynamic ECME Algorithm
- Symbol-Level Precoding for Integrated Sensing and Covert Communication
- Simultaneous penalized M-estimation of covariance matrices using geodesically convex optimization
- Secure Beamforming Design for IRS-ISAC Systems with a Hardware-Efficient Hybrid Beamforming Architecture
- Sparse Distance Weighted Discrimination
- Regularized Reduced Rank Regression for mixed predictor and response variables
- Accurate, Interpretable, and Fast Animation: An Iterative, Sparse, and Nonconvex Approach
- Constant-Modulus Secure Analog Beamforming for an IRS-Assisted Communication System with Large-Scale Antenna Array
- Conic Quadratic Formulations for Wireless Communications Design
- Kernel Density Estimation-Based Markov Models with Hidden State
- Fast Semidifferential-based Submodular Function Optimization
- A redescending-weight curvature regularization model for edge-preserving signal denoising
- Near-optimal Linear Predictive Clustering in Non-separable Spaces via Mixed Integer Programming and Quadratic Pseudo-Boolean Reductions
- Beamforming Design for Joint Localization and Data Transmission in Distributed Antenna System
- A Majorization-Minimization Algorithm for Computing the Karcher Mean of Positive Definite Matrices
- A Latent-Variable Formulation of the Poisson Canonical Polyadic Tensor Model: Maximum Likelihood Estimation and Fisher Information
- Rates of Convergence of Maximum Smoothed Log-Likelihood Estimators for Semi-Parametric Multivariate Mixtures
- Sinusoidal Parameter Estimation from Signed Measurements via Majorization-Minimization Based RELAX
- Theoretical Grounding for Estimation in Conditional Independence Multivariate Finite Mixture Models
- Demand Estimation from Sales Transaction Data -- Practical Extensions
- A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing
- Hybrid Transceivers Design for Large-Scale Antenna Arrays Using Majorization-Minimization Algorithms
- Multiresolution Tensor Decomposition for Multiple Spatial Passing Networks
- Novel Algorithms based on Majorization Minimization for Nonnegative Matrix Factorization
- Block-wise Minimization-Majorization algorithm for Huber's criterion: sparse learning and applications
- PrivateMail: Supervised Manifold Learning of Deep Features With Differential Privacy for Image Retrieval
- Design of polyphase sequence sets with good correlation properties under spectral distortion via majorization-minimization framework
- Provably Convergent Working Set Algorithm for Non-Convex Regularized Regression
- I-LAMM for Sparse Learning: Simultaneous Control of Algorithmic Complexity and Statistical Error
- Nonsmooth Analysis and Subgradient Methods for Averaging in Dynamic Time Warping Spaces
- Gaining Outlier Resistance With Progressive Quantiles: Fast Algorithms and Theoretical Studies
- Fast Image Recovery Using Variable Splitting and Constrained Optimization
- Penalized Cox’s proportional hazards model for high-dimensional survival data with grouped predictors
- Sequence Design to Minimize the Weighted Integrated and Peak Sidelobe Levels
- Computational methods for large-scale inverse problems: a survey on hybrid projection methods
- SolarBoost: Distributed Photovoltaic Power Forecasting Amid Time-varying Grid Capacity
- Hierarchical Overlapping Group Lasso for GMANOVA Model
- Non-Orthogonal Pilot Sequence Design for Multi-Cells Interference Networks
- Robust Design for Intelligent Reflecting Surfaces Assisted MISO Systems
- Randomized flexible Krylov methods for ℓp regularization
- Zero-Temperature Limit of a Convergent Algorithm to Minimize the Bethe Free Energy
- Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
- Generalized simultaneous component analysis of binary and quantitative data
- Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors
- Truncated Inference for Latent Variable Optimization Problems: Application to Robust Estimation and Learning
- Private Posterior distributions from Variational approximations
- Model-Based Clustering of Nonparametric Weighted Networks with Application to Water Pollution Analysis
- Algorithms for nonnegative matrix factorization with the beta-divergence
- A Unified Approach for Learning the Parameters of Sum-Product Networks
- Classification Filtering
- Non-orthogonal Multiple Access Assisted Multi-Region Geocast
- Revisiting Bayesian Blind Deconvolution
- Clustering Data with Nonignorable Missingness using Semi-Parametric Mixture Models
- Genomic Region Detection via Spatial Convex Clustering
- Pilot Spoofing Attack by Multiple Eavesdroppers
- Adaptively Robust Geographically Weighted Regression
- Channel Estimation and Data Detection in DS-Spread Channels: A Unified Framework, Novel Algorithms, and Waveform Comparison
- Efficient Rank Minimization via Solving Non-convexPenalties by Iterative Shrinkage-Thresholding Algorithm
- On Minimization/Maximization of the Generalized Multi-Order Complex Quadratic Form With Constant-Modulus Constraints
- New Methods for MLE of Toeplitz Structured Covariance Matrices with Applications to RADAR Problems
- Variational Algorithms for Marginal MAP
- Proactive Constrained Policy Optimization with Preemptive Penalty
- Energy-Efficient Cell Activation, User Association, and Spectrum Allocation in Heterogeneous Networks
- SparseStep: Approximating the Counting Norm for Sparse Regularization
- Robust Binary Fused Compressive Sensing using Adaptive Outlier Pursuit
- Binary Fused Compressive Sensing: 1-Bit Compressive Sensing meets Group Sparsity
- Iterative Shrinkage Approach to Restoration of Optical Imagery
- Hash Function Learning via Codewords
- MIMO Transmit Beampattern Matching Under Waveform Constraints
- Group Sparse Beamforming for Green Cloud-RAN
- Overcomplete Independent Component Analysis via SDP
- Network Slicing for Service-Oriented Networks Under Resource Constraints
- A Fast Algorithm for Convolutional Structured Low-Rank Matrix Recovery
- Maximum likelihood estimation of burst-merging kernels for bursty time series
- Regularized k-POD: Sparse k-means clustering for high-dimensional missing data
- Escaping Poor Local Minima in Large Scale Robust Estimation
- On the Linear Programming Model for Dynamic Stochastic Matching and Its Application to Pricing
- Partially Observed Markov Decision Processes. Problem Sets and Internet\n Supplement
- Total Variation, Adaptive Total Variation and Nonconvex Smoothly Clipped Absolute Deviation Penalty for Denoising Blocky Images
- Robust Simultaneous Localization of Nodes and Targets in Sensor Networks Using Range-Only Measurements
- Iteratively Reweighted \ℓ1 Approaches to Sparse Composite\n Regularization
- Exploiting Cognition in ISAR Processing for Spectral Compatibility Applications
- Latent Association Mining in Binary Data
- Trust Region Policy Optimization
- Energy Efficiency Optimization for Millimeter Wave System with Resolution-Adaptive ADCs
- Solving High-Order Portfolios via Successive Convex Approximation Algorithms
- Inertial Quadratic Majorization Minimization with Application to Kernel Regularized Learning
- Some Theoretical Results Regarding the Polygonal Distribution
- A fast unified algorithm for solving group-lasso penalize learning problems
- Fusing heterogeneous data sets
- Learning Hash Function through Codewords
- DistStat.jl: Towards Unified Programming for High-Performance Statistical Computing Environments in Julia
- Epitope profiling via mixture modeling of ranked data
- Fronthaul-Aware User-Centric Generalized Cell-Free Massive MIMO Systems
- Efficient Network Automatic Relevance Determination
- Generalized Quadratic Matrix Programming: A Unified Framework for Linear Precoding With Arbitrary Input Distributions
- Efficient Hierarchical Bayesian Inference for Spatio-temporal Regression\n Models in Neuroimaging
- Another Look at DWD: Thrifty Algorithm and Bayes Risk Consistency in RKHS
- Distributed Learning of Finite Gaussian Mixtures
- Robust sparse Gaussian graphical modeling
- Space Alternating Penalized Kullback Proximal Point Algorithms for Maximizing Likelihood with Nondifferentiable Penalty
- Joint MOO of Transmit Precoding and Receiver Design in a Downlink Time Switching MISO SWIPT System
- Grouped Variable Selection via Nested Spike and Slab Priors
- DISCOMAX: A Proximity-Preserving Distance Correlation Maximization Algorithm
- Multivariate Multinomial Logit Model with ANOVA Decomposition for Correlated Categorical Outcomes
- Relaxed Majorization-Minimization for Non-smooth and Non-convex Optimization
- Robust and Sparse Regression in GLM by Stochastic Optimization
- A monotonic MM-type algorithm for estimation of nonparametric finite mixture models with dependent marginals
- A quasi-Newton acceleration for high-dimensional optimization algorithms
- Robust Transmission for Massive MIMO Downlink with Imperfect CSI
- Structured Channel Covariance Estimation from Limited Samples for Large Antenna Arrays
- Accelerated MM Algorithms for Ranking Scores Inference from Comparison Data
- Iterative proportional scaling revisited: a modern optimization perspective
- Efficient edge-preserving methods for dynamic inverse problems
- Fast Iteratively Reweighted Least Squares Algorithms for Analysis-Based Sparsity Reconstruction
- Spatially Informed Independent Vector Analysis
- Flexible High-Dimensional Unsupervised Learning with Missing Data
- A Distributed Method for Fitting Laplacian Regularized Stratified Models
- A Robust Design for BackCom Assisted Hybrid NOMA
- On-Policy Trust Region Policy Optimisation with Replay Buffers
- Achievable Rate maximization by Passive Intelligent Mirrors
- Negative Binomial Matrix Factorization for Recommender Systems
- Priorconditioned Sparsity-Promoting Projection Methods for Deterministic and Bayesian Linear Inverse Problems
- A systematic optimization approach for a class of statistical inference problems utilizing data augmentation
- DCOOL-NET: Distributed cooperative localization for sensor networks
- PALMNUT: An Enhanced Proximal Alternating Linearized Minimization Algorithm with Application to Separate Regularization of Magnitude and Phase
- SPIRAL out of convexity: sparsity-regularized algorithms for photon-limited imaging
- Influence network reconstruction from discrete time-series of count data modelled by multidimensional Hawkes processes
- Rethinking the Trust Region in LLM Reinforcement Learning
- Physical Layer Security for Integrated Sensing and Communication: A Survey
- Nonconcave penalized composite conditional likelihood estimation of sparse Ising models
- PIANO: A Fast Parallel Iterative Algorithm for Multinomial and Sparse Multinomial Logistic Regression
- Regularized Tyler's Scatter Estimator: Existence, Uniqueness, and Algorithms
- Joint Design of Transmit Waveforms and Receive Filters for MIMO Radar via Manifold Optimization
- Dominating Hyperplane Regularization for Variable Selection in Multivariate Count Regression
- Robust Estimation of Sparse Precision Matrix using Adaptive Weighted Graphical Lasso Approach
- Sparsity Within and Across Overlapping Groups
- PAFit: An R Package for the Non-Parametric Estimation of Preferential Attachment and Node Fitness in Temporal Complex Networks
- Majorization Minimization Methods for Distributed Pose Graph Optimization with Convergence Guarantees
- Generalized mean shift with triangular kernel profile
- Logistic principal component analysis via non-convex singular value thresholding
- Robust subspace recovery by Tyler's M-estimator
- Coordinate descent algorithms for lasso penalized regression
- A majorization-minimization approach to design of power transmission networks
- A Unified Probabilistic View on Spatially Informed Source Separation and Extraction Based on Independent Vector Analysis
- Nonconvex Optimization via MM Algorithms: Convergence Theory
- Separating common (global and local) and distinct variation in multiple mixed types data sets
- Cluster and propensity based approximation of a network. [europepmc]
- A method for calling copy number polymorphism using haplotypes. [europepmc]
- Convex clustering: an attractive alternative to hierarchical clustering. [europepmc]
- PAFit: A Statistical Method for Measuring Preferential Attachment in Temporal Complex Networks. [europepmc]
- Structure-adaptive CBCT reconstruction using weighted total variation and Hessian penalties. [europepmc]
- Link predication based on matrix factorization by fusion of multi class organizations of the network. [europepmc]
- Variance Component Selection With Applications to Microbiome Taxonomic Data. [europepmc]
- Genomic region detection via Spatial Convex Clustering. [europepmc]
- Markov Model-Based Method to Analyse Time-Varying Networks in EEG Task-Related Data. [europepmc]
- Robust Relative Error Estimation. [europepmc]
- Informed Weighted Non-Negative Matrix Factorization Using αβ -Divergence Applied to Source Apportionment. [europepmc]
- Efficient Low-PAR Waveform Design Method for Extended Target Estimation Based on Information Theory in Cognitive Radar. [europepmc]
- Robust Bayesian Regression with Synthetic Posterior Distributions. [europepmc]
- A Unified Formulation of k-Means, Fuzzy c-Means and Gaussian Mixture Model by the Kolmogorov-Nagumo Average. [europepmc]
- Efficient ReML inference in variance component mixed models using a Min-Max algorithm. [europepmc]
- VCSEL: PRIORITIZING SNP-SET BY PENALIZED VARIANCE COMPONENT SELECTION. [europepmc]
- Model-based clustering for random hypergraphs. [europepmc]
- A genome-wide analysis suggests pleiotropic effects of Green Revolution genes on shade avoidance in wheat. [europepmc]
- Multisensor Estimation Fusion on Statistical Manifold. [europepmc]
- Poisson Phase Retrieval in Very Low-count Regimes. [europepmc]
- High-Performance Statistical Computing in the Computing Environments of the 2020s. [europepmc]
- A Legacy of EM Algorithms. [europepmc]
- The Convex Mixture Distribution: Granger Causality for Categorical Time Series. [europepmc]
- Improved nonparametric survival prediction using CoxPH, Random Survival Forest & DeepHit Neural Network. [europepmc]
- MMORF-FSL's MultiMOdal Registration Framework. [europepmc]
Related