Neural Collapse with Cross-Entropy Loss

2020/12/15 by Jianfeng Lu, Stefan Steinerberger, Lu, Jianfeng +1 · 2 citations
Biochemistry, Genetics and Molecular Biology · Mathematics · #Classical Analysis and ODEs (math.CA) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Mathematical Approximation and Integration #Numerical methods in inverse problems #RNA Research and Splicing

paper · pdf · doi:10.48550/arxiv.2012.08465

openalex publication_date 2020/12/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the variational problem of cross-entropy loss with n feature vectors on a unit hypersphere in ℝ^d. We prove that when d ≥ n - 1, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that as n → ∞ with fixed d, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.

Citations

• The spectral function of an elliptic operator

Cited by

• SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapse
• Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradient

Related

• Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap
• On the Optimization Landscape of Neural Collapse under MSE Loss: Global\n Optimality with Unconstrained Features
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Are All Losses Created Equal: A Neural Collapse Perspective
• Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data