Topological properties of the set of functions generated by neural\n networks of fixed size

2018/06/21 by Philipp Petersen, Petersen, Philipp, Mones Raslan +3 · 5 citations
Computer Science · #52A30 #54H99 #68T05 #FOS: Mathematics #Functional Analysis (math.FA) #General Topology (math.GN) #Machine Learning and ELM #Neural Networks and Applications #Topological and Geometric Data Analysis

paper · pdf · doi:10.48550/arxiv.1806.08459

openalex publication_date 2018/06/21 · openalex created_date 2022/09/27 · openalex updated_date 2026/07/28

Abstract

We analyze the topological properties of the set of functions that can be\nimplemented by neural networks of a fixed size. Surprisingly, this set has many\nundesirable properties. It is highly non-convex, except possibly for a few\nexotic activation functions. Moreover, the set is not closed with respect to\nL^p-norms, 0 < p < \∞, for all practically-used activation functions,\nand also not closed with respect to the L^\∞-norm for all\npractically-used activation functions except for the ReLU and the parametric\nReLU. Finally, the function that maps a family of weights to the function\ncomputed by the associated network is not inverse stable for every practically\nused activation function. In other words, if f₁, f₂ are two functions\nrealized by neural networks and if f₁, f₂ are close in the sense that\n\‖f₁ - f₂\‖_L^\∞ \≤ \ε for \ε > 0, it is,\nregardless of the size of \ε, usually not possible to find weights\nw₁, w₂ close together such that each f_i is realized by a neural network\nwith weights w_i. Overall, our findings identify potential causes for issues\nin the training procedure of deep learning such as no guaranteed convergence,\nexplosion of parameters, and slow convergence.\n

Cited by

• On Parameter Identification in Three-Dimensional Elasticity and Discretisation with Physics-Informed Neural Networks
• Complexity Dependent Error Rates for Physics-informed Statistical Learning via the Small-ball Method
• Neural network approximation in discrete dual norms with adaptive test spaces
• A convergence framework for energy minimisation of linear self-adjoint elliptic PDEs in nonlinear approximation spaces
• Multimodal learning of melt pool dynamics in laser powder bed fusion

Related

• Deep, Skinny Neural Networks are not Universal Approximators
• An in-depth look at approximation via deep and narrow neural networks
• Data Topology-Dependent Upper Bounds of Neural Network Widths
• Deep Function Machines: Generalized Neural Networks for Topological Layer Expression
• On the Expressive Power of Neural Networks