2021/11/01 by Soham Dan, Dan, Soham, Anirbit Mukherjee +4
Computer Science · Materials Science · Mathematics · Physics and Astronomy · #Algorithm #Artificial intelligence #Artificial neural network #Computer science #Dynamical Systems (math.DS) #Elasticity (physics) #FOS: Computer and information sciences #FOS: Mathematics #Geometry #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Machine learning #Mathematics #Model Reduction and Neural Networks #Neural Networks and Applications #Physics #Point (geometry) #Property (philosophy) #Regression #Statistics #Training set
paper · pdf · doi:10.48550/arxiv.2111.01166
openalex publication_date 2021/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
In the recent past, a property of neural training trajectories in weight-space had been isolated, that of "local elasticity" (denoted as S\rm rel). Local elasticity attempts to quantify the propagation of the influence of a sampled data point on the prediction at another data. In this work, we embark on a comprehensive study of the existing notion of S\rm rel and also propose a new definition that addresses the limitations that we point out for the original definition in the classification setting. On various state-of-the-art neural network training on SVHN, CIFAR-10 and CIFAR-100 we demonstrate how our new proposal of S\rm rel, as opposed to the original definition, much more sharply detects the property of the weight updates preferring to make prediction changes within the same class as the sampled data. In neural regression experiments we demonstrate that the original S\rm rel reveals a 2-phase behavior -- that the training proceeds via an initial elastic phase when S\rm rel changes rapidly and an eventual inelastic phase when S\rm rel remains large. We show that some of these properties can be analytically reproduced in various instances of doing regression via gradient flows on model predictor classes.