How regularization affects the geometry of loss functions

2023/07/28 by Nathaniel Bottman, Bottman, Nathaniel, Yaim Cooper +3 · 1 citation
Physics and Astronomy · Computer Science · #Model Reduction and Neural Networks #Neural Networks and Applications #Medical Image Segmentation Techniques

paper · pdf · doi:10.48550/arxiv.2307.15744

Abstract

What neural networks learn depends fundamentally on the geometry of the underlying loss function. We study how different regularizers affect the geometry of this function. One of the most basic geometric properties of a smooth function is whether it is Morse or not. For nonlinear deep neural networks, the unregularized loss function L is typically not Morse. We consider several different regularizers, including weight decay, and study for which regularizers the regularized function L_ε becomes Morse.

Cited by

• When majority rules, minority loses: bias amplification of gradient descent

Related

• A topological description of loss surfaces based on Betti Numbers
• Numerical Exploration of Training Loss Level-Sets in Deep Neural\n Networks
• Loss Barcode: A Topological Measure of Escapability in Loss Landscapes
• Pure and Spurious Critical Points: a Geometric Study of Linear Networks
• Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks