2019/09/27 by O. Duranthon, Duranthon, O, Matteo Marsili +6
Computer Science · Mathematics · Physics and Astronomy · #Data Analysis #Data Mining Algorithms and Applications #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #FOS: Physical sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications #Quantum many-body systems #Statistical Mechanics (cond-mat.stat-mech) #Statistics and Probability (physics.data-an) #cond-mat.stat-mech #cs.IT #cs.LG #math.IT #physics.data-an
paper · pdf · doi:10.48550/arxiv.1909.12792
28 pages, 9 figures
openalex publication_date 2019/09/27 · arxiv created 2021/01/27 · arxiv updated 2021/01/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We show that the mutual information between the representation of a learning machine and the hidden features that it extracts from data is bounded from below by the relevance, which is the entropy of the model's energy distribution. Models with maximal relevance -- that we call Optimal Learning Machines (OLM) -- are hence expected to extract maximally informative representations. We explore this principle in a range of models. For fully connected Ising models and we show that \em i) OLM are characterised by inhomogeneous distributions of couplings, and that \em ii) their learning performance is affected by sub-extensive features that are elusive to a thermodynamic treatment. On specific learning tasks, we find that likelihood maximisation is achieved by models with maximal relevance. Training of Restricted Boltzmann Machines on the MNIST benchmark shows that learning is associated with a broadening of the spectrum of energy levels and that the internal representation of the hidden layer approaches the maximal relevance that can be achieved in a finite dataset. Finally, we discuss a Gaussian learning machine that clarifies that learning hidden features is conceptually different from parameter estimation.