On Margin Maximization in Linear and ReLU Networks

2021/10/06 by Gal Vardi, Ohad Shamir, Vardi, Gal +3 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2110.02732

openalex publication_date 2021/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li [2019] showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in the parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.

Citations

• On implicit regularization: Morse functions and applications to matrix\n factorization
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Inductive Bias of Multi-Channel Linear Convolutional Networks with\n Bounded Weight Norm

Cited by

• Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers
• PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
• SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

Related

• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias
• Gradient Methods Provably Converge to Non-Robust Networks
• When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?
• Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks