Generalized Neural Collapse for a Large Number of Classes

2023/10/09 by Jiachen Jiang, Jiang, Jiachen, Jinxin Zhou +11 · 12 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2310.05351

openalex publication_date 2023/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural collapse provides an elegant mathematical characterization of learned last layer representations (a.k.a. features) and classifier weights in deep classification models. Such results not only provide insights but also motivate new techniques for improving practical deep models. However, most of the existing empirical and theoretical studies in neural collapse focus on the case that the number of classes is small relative to the dimension of the feature space. This paper extends neural collapse to cases where the number of classes are much larger than the dimension of feature space, which broadly occur for language models, retrieval systems, and face recognition applications. We show that the features and classifier exhibit a generalized neural collapse phenomenon, where the minimum one-vs-rest margins is maximized.We provide empirical study to verify the occurrence of generalized neural collapse in practical deep neural networks. Moreover, we provide theoretical study to show that the generalized neural collapse provably occurs under unconstrained feature model with spherical constraint, under certain technical conditions on feature dimension and number of classes.

Cited by

• Heads collapse, features stay: Why Replay needs big buffers
• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
• Mitigating Negative Flips via Margin Preserving Training
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• Explaining Grokking and Information Bottleneck through Neural Collapse Emergence
• Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery
• From Compression to Expression: A Layerwise Analysis of In-Context Learning
• Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
• Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model
• BCE vs. CE in Deep Feature Learning
• Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images
• XtraLight-MedMamba for Classification of Neoplastic Tubular Adenomas

Related

• Limitations of Neural Collapse for Understanding Generalization in Deep Learning
• Neural Collapse: A Review on Modelling Principles and Generalization
• Extended Unconstrained Features Model for Exploring Deep Neural Collapse
• Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
• A Geometric Analysis of Neural Collapse with Unconstrained Features