Neural collapse with unconstrained features

2020/11/23 by Dustin G. Mixon, Mixon, Dustin G., Hans Parshall +3 · 12 citations
Computer Science · Biochemistry, Genetics and Molecular Biology · #Neural Networks and Applications #Adversarial Robustness in Machine Learning #Cell Image Analysis Techniques

paper · pdf · doi:10.48550/arxiv.2011.11619

Abstract

Neural collapse is an emergent phenomenon in deep learning that was recently discovered by Papyan, Han and Donoho. We propose a simple "unconstrained features model" in which neural collapse also emerges empirically. By studying this model, we provide some explanation for the emergence of neural collapse in terms of the landscape of empirical risk.

Cited by

• Towards Understanding Steering Strength
• Heads collapse, features stay: Why Replay needs big buffers
• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering
• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
• Geometric Properties of Neural Multivariate Regression
• Explaining Grokking and Information Bottleneck through Neural Collapse Emergence
• Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
• Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradient
• Consistent Supervised-Unsupervised Alignment for Generalized Category Discovery

Related

• Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data
• Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model
• Limitations of Neural Collapse for Understanding Generalization in Deep Learning
• Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
• Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data