Nonparametric regression with modified ReLU networks

2022/07/17 by Aleksandr Beknazaryan, Beknazaryan, Aleksandr, Hailin Sang +1
Engineering · Mathematics · Computer Science · #Control Systems and Identification #Statistical Methods and Inference #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2207.08306

Abstract

We consider regression estimation with modified ReLU neural networks in which network weight matrices are first modified by a function α before being multiplied by input vectors. We give an example of continuous, piecewise linear function α for which the empirical risk minimizers over the classes of modified ReLU networks with l₁ and squared l₂ penalties attain, up to a logarithmic factor, the minimax rate of prediction of unknown β-smooth function.

Related

• Robust nonparametric regression based on deep ReLU neural networks
• Nearly Optimal Learning using Sparse Deep ReLU Networks in Regularized Empirical Risk Minimization with Lipschitz Loss
• Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes
• Benign Overfitting for Regression with Trained Two-Layer ReLU Networks
• Nonparametric Learning of Two-Layer ReLU Residual Units