An exact mapping between the Variational Renormalization Group and Deep\n Learning

2014/10/14 by Pankaj Mehta, David J. Schwab, Mehta, Pankaj +1 · 8 voices · 100 citations
Computer Science · Materials Science · Mathematics · Physics and Astronomy · #Artificial intelligence #Boltzmann machine #Computer science #Deep learning #Feature learning #Functional renormalization group #Generative Adversarial Networks and Image Synthesis #Granularity #Ising model #Machine Learning in Materials Science #Machine learning #Mathematics #Physics #Quantum many-body systems #Quantum mechanics #Renormalization group #Representation (politics) #Restricted Boltzmann machine #Scheme (mathematics) #Set (abstract data type) #Statistical Mechanics and Entropy #Statistical physics #Theoretical and Computational Physics #Theoretical computer science #cond-mat.stat-mech #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1410.3831

published in arXiv (Cornell University) (Cornell University) · 8 pages, 3 figures

arxiv created 2014/10/14 · openalex publication_date 2014/10/14 · arxiv updated 2014/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning is a broad set of techniques that uses multiple layers of\nrepresentation to automatically learn relevant features directly from\nstructured data. Recently, such techniques have yielded record-breaking results\non a diverse set of difficult machine learning tasks in computer vision, speech\nrecognition, and natural language processing. Despite the enormous success of\ndeep learning, relatively little is understood theoretically about why these\ntechniques are so successful at feature learning and compression. Here, we show\nthat deep learning is intimately related to one of the most important and\nsuccessful techniques in theoretical physics, the renormalization group (RG).\nRG is an iterative coarse-graining scheme that allows for the extraction of\nrelevant features (i.e. operators) as a physical system is examined at\ndifferent length scales. We construct an exact mapping from the variational\nrenormalization group, first introduced by Kadanoff, and deep learning\narchitectures based on Restricted Boltzmann Machines (RBMs). We illustrate\nthese ideas using the nearest-neighbor Ising Model in one and two-dimensions.\nOur results suggests that deep learning algorithms may be employing a\ngeneralized RG-like scheme to learn relevant features from data.\n

Cited by

• Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
• The RG-Flow Transformer: Encoding Scale-Free Dynamics in Scarce EEG
• Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention
• Machine learning and the physical sciences
• Bulk-boundary decomposition of neural networks
• Geometric deep learning: going beyond Euclidean data
• Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations
• RGMem: Renormalization Group-inspired Memory Evolution for Language Agents
• What Really is Deep Learning Doing?
• Feature extraction of machine learning and phase transition point of Ising model
• Cognition all the way down 2.0: neuroscience beyond neurons in the diverse intelligence era
• Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• Coarse-graining as a downward causation mechanism
• Mean Field Theory of Activation Functions in Deep Neural Networks
• Optimizing Latent Dimension Allocation in Hierarchical VAEs: Balancing Attenuation and Information Retention for OOD Detection
• A quantitative definition of organismality and its application to lichen
• The Autodidactic Universe
• Resolution and Relevance Trade-offs in Deep Learning
• Towards meaningful physics from generative models
• Self-regularizing restricted Boltzmann machines
• Dreaming neural networks: forgetting spurious memories and reinforcing pure ones
• Elements of Consciousness and Cognition. Biology, Mathematic, Physics and Panpsychism: an Information Topology Perspective
• Multilevel Anomaly Detection for Mixed Data
• A Perspective on Deep Imaging
• Functional Renormalization for Signal Detection: Dimensional Analysis and Dimensional Phase Transition for Nearly Continuous Spectra Effective Field Theory
• Neural Network Statistical Mechanics
• The many faces of deep learning
• A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability
• On Interpretability of Artificial Neural Networks: A Survey
• Flow of Information in Feed-Forward Deep Neural Networks
• Opening the black box of deep learning
• Generalization Dynamics of Linear Diffusion Models
• Group Convolutional Neural Network Ground State of the Quantum Dimer Model
• The Physics of Machine Learning: An Intuitive Introduction for the Physical Scientist
• PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons
• Super-resolution of spin configurations based on flow-based generative models
• PCANet: An energy perspective
• Optimal Real-Space Renormalization-Group Transformations with Artificial Neural Networks
• Renormalization Group Transformation for Hamiltonian Dynamical Systems in Biological Networks
• Why does Deep Learning work? - A perspective from Group Theory
• Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks
• Non-Gaussian processes and neural networks at finite widths
• Renormalization Group as a Koopman Operator
• Scale-invariant feature extraction of neural network and renormalization group flow
• A Probabilistic Representation of Deep Learning
• Nonperturbative renormalization for the neural network-QFT correspondence
• Machine learning quantum phases of matter beyond the fermion sign problem
• A Two-Phase Perspective on Deep Learning Dynamics
• Quantum Entanglement in Neural Network States
• Pankaj Mehta [wikipedia]

Discussions

• Exact mapping between Variational Renormalization Group and Deep Learning (2014) [hn, 97 points, 13 comments]
• An Exact Mapping Between the Variational Renormalization Group and Deep Learning [hn, 30 points, 7 comments]
• Deep Learning and Renormalization Group Thery [hn, 10 points, 1 comments]
• Trying to understand the implications of Mehta & Schwab (2014): Deep Learning as Renormalization Group — Implications, Reception, and Legacy. Why would there be a tight connection between quantum phys [bsky, 2 points, 0 comments]
• An Exact Mapping Between the Variational Renormalization Group and Deep Learning [hn, 1 points, 0 comments]
• Got in enough dead-ends with the renormalisation flow = deep learning layers process thought, which it turned out thousands of people had had already. Very useful to see it developed though. arxiv.org [bsky, 1 points, 0 comments]
• An exact map between deep learning models and statistical physics models of coarse graining | http://arxiv.org/abs/1410.3831 #dlearn [bsky, 0 points, 0 comments]
• wtf you can map deep learning to the RG group? arxiv.org/pdf/1410.3831 I go a little more mad (angry? insane?) every time I learn something new about the fascinating shoggoth known as "machine learnin [bsky, 0 points, 0 comments]

Related

• Deep learning and the renormalization group
• Interpreting Deep Learning by Establishing a Rigorous Corresponding Relationship with Renormalization Group
• Why Unsupervised Deep Networks Generalize
• Renormalization group for deep neural networks: Universality of learning and scaling laws
• Holographic Renormalization with Machine learning