Understanding over-parameterized deep networks by geometrization

2019/02/11 by Xiao Dong, Ling Zhou, Dong, Xiao +1 · 1 citation
Computer Science · Engineering · #3D Shape Modeling and Analysis #Advanced Numerical Analysis Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Manufacturing Process and Optimization #cs.LG

paper · pdf · doi:10.48550/arxiv.1902.03793

6 pages

arxiv created 2019/02/11 · openalex publication_date 2019/02/11 · arxiv updated 2019/02/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A complete understanding of the widely used over-parameterized deep networks is a key step for AI. In this work we try to give a geometric picture of over-parameterized deep networks using our geometrization scheme. We show that the Riemannian geometry of network complexity plays a key role in understanding the basic properties of over-parameterizaed deep networks, including the generalization, convergence and parameter sensitivity. We also point out deep networks share lots of similarities with quantum computation systems. This can be regarded as a strong support of our proposal that geometrization is not only the bible for physics, it is also the key idea to understand deep learning systems.

Citations

• Emergent General Relativity from Fisher Information Metric
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• How deep learning works --The geometry of deep learning
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Spacetime as the optimal generative network of quantum states: a roadmap to QM=GR?
• The loss landscape of overparameterized neural networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Geometrization of deep networks for the interpretability of deep learning systems

Cited by

• Deep network as memory space: complexity, generalization, disentangled representation and interpretability

Related

• Geometrization of deep networks for the interpretability of deep learning systems
• How deep learning works --The geometry of deep learning
• Deep network as memory space: complexity, generalization, disentangled representation and interpretability
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• On the Geometry of Deep Learning