Neural collapse with unconstrained features

2020/11/23 by Dustin G. Mixon, Mixon, Dustin G., Hans Parshall +3 · 19 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Adversarial Robustness in Machine Learning #Cell Image Analysis Techniques #Neural Networks and Applications #cs.LG

paper · pdf · doi:10.48550/arxiv.2011.11619

arxiv created 2020/11/23 · arxiv updated 2020/11/24

Abstract

Neural collapse is an emergent phenomenon in deep learning that was recently discovered by Papyan, Han and Donoho. We propose a simple "unconstrained features model" in which neural collapse also emerges empirically. By studying this model, we provide some explanation for the emergence of neural collapse in terms of the landscape of empirical risk.

Cited by

• Towards Understanding Steering Strength
• Heads collapse, features stay: Why Replay needs big buffers
• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering
• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
• Geometric Properties of Neural Multivariate Regression
• Explaining Grokking and Information Bottleneck through Neural Collapse Emergence
• Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
• Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradient
• Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery
• Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model
• Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
• Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
• Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model
• Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations
• Enhancing Pre-Trained Model-Based Class-Incremental Learning through Neural Collapse

Related

• Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data
• Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
• Limitations of Neural Collapse for Understanding Generalization in Deep Learning
• Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
• Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data