2024/02/08 by Stojnic, Mihailo
#Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Mathematics #FOS: Physical sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Probability (math.PR)
paper · doi:10.48550/arxiv.2402.05719
Recent progress in studying treelike committee machines (TCM) neural networks (NN) in \citeStojnictcmspnncaprdt23,Stojnictcmspnncapliftedrdt23,Stojnictcmspnncapdiffactrdt23 showed that the Random Duality Theory (RDT) and its a partially lifted(pl RDT) variant are powerful tools that can be used for very precise networks capacity analysis. Here, we consider wide hidden layer networks and uncover that certain aspects of numerical difficulties faced in \citeStojnictcmspnncapdiffactrdt23 miraculously disappear. In particular, we employ recently developed fully lifted (fl) RDT to characterize the wide (d→ ∞) TCM nets capacity. We obtain explicit, closed form, capacity characterizations for a very generic class of the hidden layer activations. While the utilized approach significantly lowers the amount of the needed numerical evaluations, the ultimate fl RDT usefulness and success still require a solid portion of the residual numerical work. To get the concrete capacity values, we take four very famous activations examples: \emphReLU, quadratic, erf, and tanh. After successfully conducting all the residual numerical work for all of them, we uncover that the whole lifting mechanism exhibits a remarkably rapid convergence with the relative improvements no better than ∼ 0.1% happening already on the 3-rd level of lifting. As a convenient bonus, we also uncover that the capacity characterizations obtained on the first and second level of lifting precisely match those obtained through the statistical physics replica theory methods in \citeZavPeh21 for the generic and in \citeBalMalZech19 for the ReLU activations.