Scale-invariant feature extraction of neural network and renormalization group flow

2018/01/22 by Satoshi Iso, Shotaro Shiba, Sumito Yokoo · 77 citations
Computer Science · Mathematics · Physics and Astronomy · #Artificial intelligence #Artificial neural network #Boltzmann machine #Computer science #Flow (mathematics) #Generative Adversarial Networks and Image Synthesis #Geometry #Granularity #Invariant (physics) #Ising model #Mathematical physics #Mathematics #Model Reduction and Neural Networks #Pattern recognition (psychology) #Physics #Renormalization group #Restricted Boltzmann machine #Statistical physics #Theoretical and Computational Physics #cond-mat.stat-mech #cs.LG #hep-th #stat.ML

paper · pdf · doi:10.1103/physreve.97.053304

published in Physical review. E 97(5), 053304 (American Physical Society) · 32 pages, 17 figures

arxiv created 2018/01/22 · openalex publication_date 2018/05/08 · arxiv updated 2018/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Theoretical understanding of how a deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse graining. It reminds us of the basic renormalization group (RG) concept in statistical physics. In order to explore possible relations between DNN and RG, we use the restricted Boltzmann machine (RBM) applied to an Ising model and construct a flow of model parameters (in particular, temperature) generated by the RBM. We show that the unsupervised RBM trained by spin configurations at various temperatures from T=0 to T=6 generates a flow along which the temperature approaches the critical value T_c=2.27. This behavior is the opposite of the typical RG flow of the Ising model. By analyzing various properties of the weight matrices of the trained RBM, we discuss why it flows towards T_c and how the RBM learns to extract features of spin configurations.

Citations

• A Practical Guide to Training Restricted Boltzmann Machines
• Why does Deep Learning work? - A perspective from Group Theory
• Deep learning and the renormalization group
• An exact mapping between the Variational Renormalization Group and Deep\n Learning
• Training Products of Experts by Minimizing Contrastive Divergence
• Why does deep and cheap learning work so well?
• Renormalization Group Transformation for Hamiltonian Dynamical Systems in Biological Networks
• Restricted Boltzmann machines for the long range Ising models

Cited by

• Emergence of a finite-size-scaling function in the supervised learning of the Ising phase transition
• Signal detection in nearly continuous spectra and symmetry breaking
• Machine learning for quantum matter
• Thermodynamics and feature extraction by machine learning
• A cautionary tale for machine learning generated configurations in presence of a conserved quantity