Storage and Learning phase transitions in the Random-Features Hopfield Model

2023/03/29 by Matteo Negri, Negri, Matteo, Clarissa Lauditi +7 · 7 citations
Computer Science · Mathematics · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Morphological variations and asymmetry #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2303.16880

openalex publication_date 2023/03/29 · openalex created_date 2023/04/05 · openalex updated_date 2026/07/30

Abstract

The Hopfield model is a paradigmatic model of neural networks that has been analyzed for many decades in the statistical physics, neuroscience, and machine learning communities. Inspired by the manifold hypothesis in machine learning, we propose and investigate a generalization of the standard setting that we name Random-Features Hopfield Model. Here P binary patterns of length N are generated by applying to Gaussian vectors sampled in a latent space of dimension D a random projection followed by a non-linearity. Using the replica method from statistical physics, we derive the phase diagram of the model in the limit P,N,D→∞ with fixed ratios α=P/N and α_D=D/N. Besides the usual retrieval phase, where the patterns can be dynamically recovered from some initial corruption, we uncover a new phase where the features characterizing the projection can be recovered instead. We call this phenomena the learning phase transition, as the features are not explicitly given to the model but rather are inferred from the patterns in an unsupervised fashion.

Cited by

• The dimensionality of the Hopfield model
• Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits
• Statistical physics for artificial neural networks
• Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings
• Statistical mechanics of vector Hopfield network near and above saturation
• Hopfield Networks as Models of Emergent Function in Biology
• On the role of non-linear latent features in bipartite generative neural networks

Related

• Random Features Hopfield Networks generalize retrieval to previously unseen examples
• Statistical mechanics of vector Hopfield network near and above saturation
• Prototype Analysis in Hopfield Networks with Hebbian Learning
• A Waddington landscape for prototype learning in generalized Hopfield networks
• Unsupervised prototype learning in an associative-memory network