The distance between the weights of the neural network is meaningful

2021/01/31 by Liqun Yang, Yang, Liqun, Yijun Yang +7
Computer Science · #94-10 (Primary) 47N30(Secondary) #Advanced Neural Network Applications #FOS: Computer and information sciences #I.2.4 #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications #acm:94-10 #cs.LG #msc:94-10

paper · pdf · doi:10.48550/arxiv.2102.00396

10 pages, 13 figure

arxiv created 2021/01/31 · openalex publication_date 2021/01/31 · arxiv updated 2021/02/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In the application of neural networks, we need to select a suitable model based on the problem complexity and the dataset scale. To analyze the network's capacity, quantifying the information learned by the network is necessary. This paper proves that the distance between the neural network weights in different training stages can be used to estimate the information accumulated by the network in the training process directly. The experiment results verify the utility of this method. An application of this method related to the label corruption is shown at the end.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The information bottleneck method
• On Information and Sufficiency
• Long Short-Term Memory
• Gradient-based learning applied to document recognition
• Opening the Black Box of Deep Neural Networks via Information
• YOLOv3: An Incremental Improvement
• Where is the Information in a Deep Neural Network?
• Understanding deep learning requires rethinking generalization
• Shortcut learning in deep neural networks
• Generalizing from a Few Examples
• Learning Multiple Layers of Features from Tiny Images

Related

• Computing the Information Content of Trained Neural Networks
• Where is the Information in a Deep Neural Network?
• A comparative analysis of a neural network with calculated weights and a neural network with random generation of weights based on the training dataset size
• Interpreting Neural Networks through Mahalanobis Distance
• Predicting Neural Network Accuracy from Weights