Topological properties of the set of functions generated by neural networks of fixed size

2018/06/21 by Philipp Petersen, Petersen, Philipp, Mones Raslan +3 · 15 citations
Computer Science · Mathematics · #52A30 #54H99 #68T05 #FOS: Mathematics #Functional Analysis (math.FA) #General Topology (math.GN) #Machine Learning and ELM #Neural Networks and Applications #Topological and Geometric Data Analysis #math.FA #math.GN #msc:52A30 #msc:54H99 #msc:68T05

paper · pdf · doi:10.48550/arxiv.1806.08459

openalex publication_date 2018/06/21 · arxiv created 2020/01/23 · arxiv updated 2020/01/24 · openalex created_date 2022/09/27 · openalex updated_date 2026/07/28

Abstract

We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties. It is highly non-convex, except possibly for a few exotic activation functions. Moreover, the set is not closed with respect to L^p-norms, 0 < p < ∞, for all practically-used activation functions, and also not closed with respect to the L^∞-norm for all practically-used activation functions except for the ReLU and the parametric ReLU. Finally, the function that maps a family of weights to the function computed by the associated network is not inverse stable for every practically used activation function. In other words, if f₁, f₂ are two functions realized by neural networks and if f₁, f₂ are close in the sense that ‖f₁ - f₂‖_L^∞ ≤ ε for ε > 0, it is, regardless of the size of ε, usually not possible to find weights w₁, w₂ close together such that each f_i is realized by a neural network with weights w_i. Overall, our findings identify potential causes for issues in the training procedure of deep learning such as no guaranteed convergence, explosion of parameters, and slow convergence.

Cited by

• On Parameter Identification in Three-Dimensional Elasticity and Discretisation with Physics-Informed Neural Networks
• A discontinuous Galerkin plane wave neural network method for Helmholtz equation and Maxwell's equations
• Complexity Dependent Error Rates for Physics-informed Statistical Learning via the Small-ball Method
• Neural network approximation in discrete dual norms with adaptive test spaces
• A convergence framework for energy minimisation of linear self-adjoint elliptic PDEs in nonlinear approximation spaces
• Multimodal learning of melt pool dynamics in laser powder bed fusion
• PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
• Convergence rates for Tikhonov regularization on compact sets: application to neural networks
• Computational Math with Neural Networks is Hard
• Discontinuous hybrid neural networks for the one-dimensional partial differential equations
• SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures
• Neural network methods for Neumann series problems of Perron-Frobenius operators
• An r-adaptive finite element method using neural networks for parametric self-adjoint elliptic problem
• Non-identifiability distinguishes Neural Networks among Parametric Models
• An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications

Related

• Deep, Skinny Neural Networks are not Universal Approximators
• An in-depth look at approximation via deep and narrow neural networks
• Data Topology-Dependent Upper Bounds of Neural Network Widths
• Deep Function Machines: Generalized Neural Networks for Topological Layer Expression
• On the Expressive Power of Neural Networks