How to induce regularization in linear models: A guide to reparametrizing gradient flow

2023/08/09 by Hung-Hsu Chou, Chou, Hung-Hsu, Johannes Maly +3
Engineering · #FOS: Mathematics #Numerical Analysis (math.NA) #Optimization and Control (math.OC) #Reservoir Engineering and Simulation Methods

paper · pdf · doi:10.48550/arxiv.2308.04921

openalex publication_date 2023/08/09 · openalex created_date 2023/08/11 · openalex updated_date 2026/07/28

Abstract

In this work, we analyze the relation between reparametrizations of gradient flow and the induced implicit bias in linear models, which encompass various basic regression tasks. In particular, we study how reparametrization, loss function, and link function influence the convergence behavior and implicit bias of gradient flow. Our results provide conditions under which the implicit bias can be well-described and convergence of the flow is guaranteed. We furthermore show how to use these insights for designing reparametrization functions that lead to specific implicit biases which are closely connected to ℓ_p- or trigonometric regularizers.

Related

• Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
• Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent
• Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias
• Linear regression with overparameterized linear neural networks: Tight upper and lower bounds for implicit ℓ¹-regularization
• Implicit Bias in Deep Linear Classification: Initialization Scale vs\n Training Accuracy