The Vanishing Decision Boundary Complexity and the Strong First Component

2022/11/25 by Hengshuai Yao, Yao, Hengshuai
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2211.16209

openalex publication_date 2022/11/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show that unlike machine learning classifiers, there are no complex boundary structures in the decision boundaries for well-trained deep models. However, we found that the complicated structures do appear in training but they vanish shortly after shaping. This is a pessimistic news if one seeks to capture different levels of complexity in the decision boundary for understanding generalization, which works well in machine learning. Nonetheless, we found that the decision boundaries of predecessor models on the training data are reflective of the final model's generalization. We show how to use the predecessor decision boundaries for studying the generalization of deep models. We have three major findings. One is on the strength of the first principle component of deep models, another about the singularity of optimizers, and the other on the effects of the skip connections in ResNets. Code is at https://github.com/hengshu1/decision_boundary_github.

Related

• Investigating Decision Boundaries of Trained Neural Networks
• On the Decision Boundary of Deep Neural Networks
• Unveiling the Hessian's Connection to the Decision Boundary
• Building a Regular Decision Boundary with Deep Networks
• Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective