Correlation between entropy and generalizability in a neural network

2022/07/05 by Ge Zhang, Zhang, Ge
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Statistical Mechanics (cond-mat.stat-mech) #Statistical Mechanics and Entropy

paper · pdf · doi:10.48550/arxiv.2207.01996

openalex publication_date 2022/07/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Although neural networks can solve very complex machine-learning problems, the theoretical reason for their generalizability is still not fully understood. Here we use Wang-Landau Mote Carlo algorithm to calculate the entropy (logarithm of the volume of a part of the parameter space) at a given test accuracy, and a given training loss function value or training accuracy. Our results show that entropical forces help generalizability. Although our study is on a very simple application of neural networks (a spiral dataset and a small, fully-connected neural network), our approach should be useful in explaining the generalizability of more complicated neural networks in future works.

Related

• High-entropy Advantage in Neural Networks' Generalizability
• Generalisation and the Risk--Entropy Curve
• Towards a theory of machine learning
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning