2022/01/21 by Ido Ben-Shaul, Shai Dekel, Ben-Shaul, Ido +1 · 1 citation
Computer Science · #Digital Imaging for Blood Diseases #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2201.08924
openalex publication_date 2022/01/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.