Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts

2023/05/12 by Huy Nguyen, TrungTin Nguyen, Nguyen, Huy +5 · 6 citations
Computer Science · Mathematics · #Algorithm #Applied mathematics #Artificial intelligence #Artificial neural network #Bayesian Methods and Mixture Models #Complement (music) #Computer science #Convergence (economics) #Estimation theory #FOS: Computer and information sciences #Gaussian #Gaussian Processes and Bayesian Inference #Gaussian process #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematical optimization #Mathematics #Mixture model #Target Tracking and Data Fusion in Sensor Networks

paper · pdf · doi:10.48550/arxiv.2305.07572

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Originally introduced as a neural network for ensemble learning, mixture of experts (MoE) has recently become a fundamental building block of highly successful modern deep neural networks for heterogeneous data analysis in several applications of machine learning and statistics. Despite its popularity in practice, a satisfactory level of theoretical understanding of the MoE model is far from complete. To shed new light on this problem, we provide a convergence analysis for maximum likelihood estimation (MLE) in the Gaussian-gated MoE model. The main challenge of that analysis comes from the inclusion of covariates in the Gaussian gating functions and expert networks, which leads to their intrinsic interaction via some partial differential equations with respect to their parameters. We tackle these issues by designing novel Voronoi loss functions among parameters to accurately capture the heterogeneity of parameter estimation rates. Our findings reveal that the MLE has distinct behaviors under two complement settings of location parameters of the Gaussian gating functions, namely when all these parameters are non-zero versus when at least one among them vanishes. Notably, these behaviors can be characterized by the solvability of two different systems of polynomial equations. Finally, we conduct a simulation study to empirically verify our theoretical results.

Cited by

• Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
• Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
• Approximation rates for finite mixtures of location-scale models
• Mixture of Experts in Large Language Models
• Modifications of the BIC for order selection in finite mixture models
• Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures

Related

• Convergence Rates for Gaussian Mixtures of Experts
• Demystifying Softmax Gating Function in Gaussian Mixture of Experts
• Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures
• Convergence Rates for Softmax Gating Mixture of Experts
• On Least Square Estimation in Softmax Gating Mixture of Experts