2022/02/20 by Cheng Chen, John C. Duchi, Cheng, Chen +3 · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Statistical Methods and Inference #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.2202.09889
openalex publication_date 2022/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We examine the necessity of interpolation in overparameterized models, that is, when achieving optimal predictive risk in machine learning problems requires (nearly) interpolating the training data. In particular, we consider simple overparameterized linear regression y = X θ+ w with random design X ∈ ℝn × d under the proportional asymptotics d/n → γ∈ (1, ∞). We precisely characterize how prediction (test) error necessarily scales with training error in this setting. An implication of this characterization is that as the label noise variance σ2 → 0, any estimator that incurs at least cσ4 training error for some constant c is necessarily suboptimal and will suffer growth in excess prediction error at least linear in the training error. Thus, optimal performance requires fitting training data to substantially higher accuracy than the inherent noise floor of the problem.