2018/05/22 by Mirco Milletarí, Thiparat Chotibut, Milletarí, Mirco +3
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Quantum many-body systems
paper · pdf · doi:10.48550/arxiv.1805.08786
openalex publication_date 2018/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We present a Statistical Mechanics (SM) model of deep neural networks, connecting the energy-based and the feed forward networks (FFN) approach. We infer that FFN can be understood as performing three basic steps: encoding, representation validation and propagation. From the meanfield solution of the model, we obtain a set of natural activations -- such as Sigmoid, \tanh and ReLu -- together with the state-of-the-art, Swish; this represents the expected information propagating through the network and tends to ReLu in the limit of zero noise.We study the spectrum of the Hessian on an associated classification task, showing that Swish allows for more consistent performances over a wider range of network architectures.