2023/05/01 by Samuel Tovey, Tovey, Samuel, Sven Krippendorf +5 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computational Physics (physics.comp-ph) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2305.00995
openalex publication_date 2023/05/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
A theory of neural networks (NNs) built upon collective variables would provide scientists with the tools to better understand the learning process at every stage. In this work, we introduce two such variables, the entropy and the trace of the empirical neural tangent kernel (NTK) built on the training data passed to the model. We empirically analyze the NN performance in the context of these variables and find that there exists correlation between the starting entropy, the trace of the NTK, and the generalization of the model computed after training is complete. This framework is then applied to the problem of optimal data selection for the training of NNs. To this end, random network distillation (RND) is used as a means of selecting training data which is then compared with random selection of data. It is shown that not only does RND select data-sets capable of outperforming random selection, but that the collective variables associated with the RND data-sets are larger than those of the randomly selected sets. The results of this investigation provide a stable ground from which the selection of data for NN training can be driven by this phenomenological framework.