Deep network as memory space: complexity, generalization, disentangled representation and interpretability

2019/07/12 by X. Dong, Xiao Dong, L. Zhou +3
Computer Science · Engineering · #Advanced Memory and Neural Computing #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.1907.06572

11 pages, 2 figures

arxiv created 2019/07/12 · openalex publication_date 2019/07/12 · arxiv updated 2019/07/16 · openalex created_date 2019/07/23 · openalex updated_date 2026/07/28

Abstract

By bridging deep networks and physics, the programme of geometrization of deep networks was proposed as a framework for the interpretability of deep learning systems. Following this programme we can apply two key ideas of physics, the geometrization of physics and the least action principle, on deep networks and deliver a new picture of deep networks: deep networks as memory space of information, where the capacity, robustness and efficiency of the memory are closely related with the complexity, generalization and disentanglement of deep networks. The key components of this understanding include:(1) a Fisher metric based formulation of the network complexity; (2)the least action (complexity=action) principle on deep networks and (3)the geometry built on deep network configurations. We will show how this picture will bring us a new understanding of the interpretability of deep learning systems.

Citations

• Emergent General Relativity from Fisher Information Metric
• Understanding Black-box Predictions via Influence Functions
• How deep learning works --The geometry of deep learning
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Dataset Distillation
• Geometrization of deep networks for the interpretability of deep learning systems
• Understanding over-parameterized deep networks by geometrization
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
• Gauge theory and twins paradox of disentangled representations

Related

• Geometrization of deep networks for the interpretability of deep learning systems
• Mathematics of Deep Learning
• Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory
• Understanding over-parameterized deep networks by geometrization
• Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations