Nonparametric regression using deep neural networks with ReLU activation function

2017/08/31 by Johannes Schmidt-Hieber · 6 citations
Computer Science · Mathematics · #Activation function #Artificial neural network #Estimator #Feedforward neural network #Flexibility (engineering) #Function (biology) #Machine Learning and ELM #Minimax #Nonparametric regression #Nonparametric statistics #Regression #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques #cs.LG #math.ST #msc:62G08 #stat.ML #stat.TH

paper · pdf · doi:10.1214/19-aos1875

published as Article: Annals of Statistics, Volume 48, Number 4, 1875-1897, 2020, Rejoinder: Annals of Statistics, Volume 48, Number 4, 1916-1921, 2020 · article, rejoinder and supplementary material

openalex created_date 2017/08/31 · openalex publication_date 2020/08/01 · arxiv created 2020/09/13 · arxiv updated 2020/09/15 · openalex updated_date 2026/08/06

Abstract

Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to log n-factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential network parameters exceeding the sample size. The analysis gives some insights into why multilayer feedforward neural networks perform well in practice. Interestingly, for ReLU activation function the depth (number of layers) of the neural network architectures plays an important role, and our theory suggests that for nonparametric regression, scaling the network depth with the sample size is natural. It is also shown that under the composition assumption wavelet estimators can only achieve suboptimal rates.

Citations

• Nonparametric estimation of composite functions
• Approximation by superpositions of a sigmoidal function
• Multilayer feedforward networks are universal approximators
• On the Number of Linear Regions of Deep Neural Networks
• ImageNet classification with deep convolutional neural networks
• Comparison of non-linear activation functions for deep neural networks on MNIST classification task

Cited by

• Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
• ICODEN: Ordinary Differential Equation Neural Networks for Interval-Censored Data
• A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications
• Deep learning of point processes for modeling high-frequency data
• Deep Learning Meets Sparse Regularization: A signal processing perspective
• Deep learning with missing data