Diametrical Risk Minimization: Theory and Computations

2019/10/24 by Norton, Matthew, Royset, Johannes O. · 1 citation
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.1910.10844

Abstract

The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in sharp empirical risk landscapes from benchmark neural network classification problems with corrupted labels.

Cited by

• Flat Minima and Generalization: Insights from Stochastic Convex Optimization

Related

• Black-Box Uniform Stability for Non-Euclidean Empirical Risk Minimization
• Differentially Private Empirical Risk Minimization Revisited: Faster and More General
• On the Fine-Grained Complexity of Empirical Risk Minimization: Kernel\n Methods and Neural Networks
• On the Local Minima of the Empirical Risk
• General risk measures for robust machine learning