2019/11/21 by Jindong Gu, Volker Tresp, Gu, Jindong +1 · 6 citations
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Computer science #Deep neural networks #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #Image (mathematics) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematics #Memorization #Music and Audio Processing #Pattern recognition (psychology) #Similarity (geometry) #Simple (philosophy) #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1911.09537
published in arXiv (Cornell University) (Cornell University) · Workshop on Machine Learning with Guarantees, NeurIPS 2019
arxiv created 2019/11/21 · openalex publication_date 2019/11/21 · arxiv updated 2019/11/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Deep neural networks (DNNs) can easily fit a random labeling of the training data with zero training error. What is the difference between DNNs trained with random labels and the ones trained with true labels? Our paper answers this question with two contributions. First, we study the memorization properties of DNNs. Our empirical experiments shed light on how DNNs prioritize the learning of simple input patterns. In the second part, we propose to measure the similarity between what different DNNs have learned and memorized. With the proposed approach, we analyze and compare DNNs trained on data with true labels and random labels. The analysis shows that DNNs have One way to Learn and N ways to Memorize. We also use gradient information to gain an understanding of the analysis results.