2023/02/27 by Antoine Maillard, Afonso S. Bandeira, Maillard, Antoine +7 · 4 citations
Computer Science · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Mathematics #FOS: Physical sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Probability (math.PR) #Topological and Geometric Data Analysis
paper · pdf · doi:10.48550/arxiv.2302.14112
openalex publication_date 2023/02/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
When can the input of a ReLU neural network be inferred from its output? In other words, when is the network injective? We consider a single layer, x ↦ ReLU(Wx), with a random Gaussian m × n matrix W, in a high-dimensional setting where n, m → ∞. Recent work connects this problem to spherical integral geometry giving rise to a conjectured sharp injectivity threshold for α= (m)/(n) by studying the expected Euler characteristic of a certain random set. We adopt a different perspective and show that injectivity is equivalent to a property of the ground state of the spherical perceptron, an important spin glass model in statistical physics. By leveraging the (non-rigorous) replica symmetry-breaking theory, we derive analytical equations for the threshold whose solution is at odds with that from the Euler characteristic. Furthermore, we use Gordon's min--max theorem to prove that a replica-symmetric upper bound refutes the Euler characteristic prediction. Along the way we aim to give a tutorial-style introduction to key ideas from statistical physics in an effort to make the exposition accessible to a broad audience. Our analysis establishes a connection between spin glasses and integral geometry but leaves open the problem of explaining the discrepancies.