Benign overfitting without concentration

2021/01/04 by Zong Shang, Shang, Zong
Engineering · Mathematics · #FOS: Mathematics #Sparse and Compressive Sensing Techniques #Statistics Theory (math.ST) #math.ST #stat.TH

paper · pdf · doi:10.48550/arxiv.2101.00914

arxiv created 2021/01/04 · openalex publication_date 2021/01/04 · arxiv updated 2021/01/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We obtain a sufficient condition for benign overfitting of linear regression problem. Our result does not rely on concentration argument but on small-ball assumption and thus can holds in heavy-tailed case. The basic idea is to establish a coordinate small-ball estimate in terms of effective rank so that we can calibrate the balance of epsilon-Net and exponential probability. Our result indicates that benign overfitting is not depending on concentration property of the input vector. Finally, we discuss potential difficulties for benign overfitting beyond linear model and a benign overfitting result without truncated effective rank.

Citations

• Understanding deep learning requires rethinking generalization
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Benign overfitting in ridge regression
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve

Related

• A Classical View on Benign Overfitting: The Role of Sample Size
• Benign Overfitting in Out-of-Distribution Generalization of Linear Models
• Benign overfitting and adaptive nonparametric regression
• Benign Overfitting in Linear Classifiers with a Bias Term
• Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression