Neural Collapse: A Review on Modelling Principles and Generalization

2022/06/08 by Vignesh Kothapalli, Kothapalli, Vignesh · 11 citations
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications #Statistical Mechanics and Entropy

paper · pdf · doi:10.48550/arxiv.2206.04041

openalex publication_date 2022/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep classifier neural networks enter the terminal phase of training (TPT) when training error reaches zero and tend to exhibit intriguing Neural Collapse (NC) properties. Neural collapse essentially represents a state at which the within-class variability of final hidden layer outputs is infinitesimally small and their class means form a simplex equiangular tight frame. This simplifies the last layer behaviour to that of a nearest-class center decision rule. Despite the simplicity of this state, the dynamics and implications of reaching it are yet to be fully understood. In this work, we review the principles which aid in modelling neural collapse, followed by the implications of this state on generalization and transfer learning capabilities of neural networks. Finally, we conclude by discussing potential avenues and directions for future research.

Cited by

• An Equivariance Toolbox for Learning Dynamics
• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
• On the Unreasonable Effectiveness of Last-layer Retraining
• From Coefficients to Directions: Rethinking Model Merging with Directional Alignment
• BootOOD: Self-Supervised Out-of-Distribution Detection via Synthetic Sample Exposure under Neural Collapse
• Adversarial Attacks Leverage Interference Between Features in Superposition
• The role of fibration symmetries in geometric deep learning
• MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment
• Why and How Auxiliary Tasks Improve JEPA Representations
• Wormhole Dynamics in Deep Neural Networks
• Feature learning is decoupled from generalization in high capacity neural networks

Related

• Limitations of Neural Collapse for Understanding Generalization in Deep Learning
• Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks
• Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges
• Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
• Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?