Non-identifiability distinguishes Neural Networks among Parametric Models

2025/04/25 by Sourav Chatterjee, Chatterjee, Sourav, Timothy Sudijono +1 · 1 voice
Computer Science · Mathematics · Social Sciences · #Adversarial Robustness in Machine Learning #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistics Theory (math.ST) #cs.LG #math.ST #stat.ML

paper · pdf · doi:10.48550/arxiv.2504.18017

openalex publication_date 2025/04/25 · arxiv published 2025/04/25 · arxiv updated 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

One of the enduring problems surrounding neural networks is to identify the factors that differentiate them from traditional statistical models. We prove a pair of results which distinguish feedforward neural networks among parametric models at the population level, for regression tasks. Firstly, we prove that for any pair of random variables (X,Y), neural networks always learn a nontrivial relationship between X and Y, if one exists. Secondly, we prove that for reasonable smooth parametric models, under local and global identifiability conditions, there exists a nontrivial (X,Y) pair for which the parametric model learns the constant predictor 𝔼[Y]. Together, our results suggest that a lack of identifiability distinguishes neural networks among the class of smooth parametric models.

Citations

• The Connection Between Approximation, Depth Separation and Learnability\n in Neural Networks
• Deep Learning is Singular, and That's Good
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• An Investigation into Neural Net Optimization via Hessian Eigenvalue\n Density
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Topological properties of the set of functions generated by neural\n networks of fixed size
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• On the ability of neural nets to express distributions
• The Power of Depth for Feedforward Neural Networks
• Representation Benefits of Deep Feedforward Networks
• Approximation by superpositions of a sigmoidal function
• Multilayer feedforward networks are universal approximators

Discussions

• File under things you always suspected, but it's nice to have a proof: "Non-identifiability distinguishes Neural Networks among Parametric Models" arxiv.org/abs/2504.18017 I had a vibes-based sense fo [bsky, 4 points, 2 comments]

Related

• Neural Networks and the Bias/Variance Dilemma
• Neural network identifiability for a family of sigmoidal nonlinearities
• On Linear Identifiability of Learned Representations
• Understanding Neural Networks as Statistical Tools
• Asymptotics for regression models under loss of identifiability