Robust variance-regularized risk minimization with concomitant scaling

2023/01/27 by Matthew J. Holland, Holland, Matthew J.
Decision Sciences · Engineering · Mathematics · #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Risk and Portfolio Optimization #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2301.11584

openalex publication_date 2023/01/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Under losses which are potentially heavy-tailed, we consider the task of minimizing sums of the loss mean and standard deviation, without trying to accurately estimate the variance. By modifying a technique for variance-free robust mean estimation to fit our problem setting, we derive a simple learning procedure which can be easily combined with standard gradient-based solvers to be used in traditional machine learning workflows. Empirically, we verify that our proposed approach, despite its simplicity, performs as well or better than even the best-performing candidates derived from alternative criteria such as CVaR or DRO risks on a variety of datasets.

Related

• One Method to Rule Them All: Variance Reduction for Data, Parameters and Many New Methods
• Variance-based regularization with convex objectives
• Empirical Risk Minimization for Losses without Variance
• Robust high dimensional learning for Lipschitz and convex losses
• Stochastic dual averaging methods using variance reduction techniques for regularized empirical risk minimization problems