A function space analysis of finite neural networks with insights from sampling theory

2020/04/15 by Raja Giryes, Giryes, Raja
Computer Science · #62D05 #68Q32 #FOS: Computer and information sciences #FOS: Mathematics #Functional Analysis (math.FA) #I.2.6 #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2004.06989

openalex publication_date 2020/04/15 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

This work suggests using sampling theory to analyze the function space represented by neural networks. First, it shows, under the assumption of a finite input domain, which is the common case in training neural networks, that the function space generated by multi-layer networks with non-expansive activation functions is smooth. This extends over previous works that show results for the case of infinite width ReLU networks. Then, under the assumption that the input is band-limited, we provide novel error bounds for univariate neural networks. We analyze both deterministic uniform and random sampling showing the advantage of the former.

Related

• Finite Sample Identification of Wide Shallow Neural Networks with Biases
• Finite Samples for Shallow Neural Networks
• Sampling Complexity of Deep Approximation Spaces
• Sampling weights of deep neural networks
• Towards a General Theory of Infinite-Width Limits of Neural Classifiers