Robust machine learning by median-of-means: Theory and practice

2020/04/01 by Guillaume Lecué, Matthieu Lerasle · 1 voice · 6 citations
Decision Sciences · Mathematics · #Advanced Statistical Methods and Models #Advanced Statistical Process Monitoring #Statistical Methods and Inference

paper · pdf · doi:10.1214/19-aos1828

openalex publication_date 2020/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Median-of-means (MOM) based procedures have been recently introduced in learning theory (Lugosi and Mendelson (2019); Lecué and Lerasle (2017)). These estimators outperform classical least-squares estimators when data are heavy-tailed and/or are corrupted. None of these procedures can be implemented, which is the major issue of current MOM procedures (Ann. Statist. 47 (2019) 783–794). In this paper, we introduce minmax MOM estimators and show that they achieve the same sub-Gaussian deviation bounds as the alternatives (Lugosi and Mendelson (2019); Lecué and Lerasle (2017)), both in small and high-dimensional statistics. In particular, these estimators are efficient under moments assumptions on data that may have been corrupted by a few outliers. Besides these theoretical guarantees, the definition of minmax MOM estimators suggests simple and systematic modifications of standard algorithms used to approximate least-squares estimators and their regularized versions. As a proof of concept, we perform an extensive simulation study of these algorithms for robust versions of the LASSO.

Citations

• Robust Estimation of a Location Parameter
• The Future of Data Analysis
• Regularization and the small-ball method II: complexity dependent error rates
• Learning subgaussian classes : Upper and minimax bounds
• Towards Robust Deep Neural Networks

Cited by

• Improved Concentration for Mean Estimators via Shrinkage
• Robust Variational Bayes by Min-Max Median Aggregation
• Exponential Lasso: robust sparse penalization under heavy-tailed noise and outliers with exponential-type loss
• Tuning free Catoni type joint robust estimation
• Convex Clustering Redefined: Robust Learning with the Median of Means Estimator
• Asymptotics of constrained M-estimation under convexity
• Outlier-robust estimation of a sparse linear model using ℓ₁-penalized Huber's M-estimator

Discussions

• doi.org/10.1214/19-A... [bsky, 2 points, 1 comments]

Related

• Robust machine learning by median-of-means : theory and practice
• On the Optimality of the Median-of-Means Estimator under Adversarial Contamination
• Learning from MOM's principles: Le Cam's approach
• Robust classification via MOM minimization
• Error bounds of Median-of-means estimators with VC-dimension