2025/12/11 by Yi Huang, Qingyun Sun, Huang, Yi +9
Computer Science · #Adversarial Robustness in Machine Learning #Bottleneck #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Information bottleneck method #Information theory #Machine Learning (cs.LG) #Machine Learning and Data Classification #Mutual information #Noise (video) #Pooling #Representation (politics) #Robustness (evolution)
paper · pdf · doi:10.48550/arxiv.2512.10573
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/12/11 · openalex created_date 2025/12/13 · openalex updated_date 2026/07/28
The Information Bottleneck (IB) principle facilitates effective representation learning by preserving label-relevant information while compressing irrelevant information. However, its strong reliance on accurate labels makes it inherently vulnerable to label noise, prevalent in real-world scenarios, resulting in significant performance degradation and overfitting. To address this issue, we propose LaT-IB, a novel Label-Noise ResistanT Information Bottleneck method which introduces a "Minimal-Sufficient-Clean" (MSC) criterion. Instantiated as a mutual information regularizer to retain task-relevant information while discarding noise, MSC addresses standard IB's vulnerability to noisy label supervision. To achieve this, LaT-IB employs a noise-aware latent disentanglement that decomposes the latent representation into components aligned with to the clean label space and the noise space. Theoretically, we first derive mutual information bounds for each component of our objective including prediction, compression, and disentanglement, and moreover prove that optimizing it encourages representations invariant to input noise and separates clean and noisy label information. Furthermore, we design a three-phase training framework: Warmup, Knowledge Injection and Robust Training, to progressively guide the model toward noise-resistant representations. Extensive experiments demonstrate that LaT-IB achieves superior robustness and efficiency under label noise, significantly enhancing robustness and applicability in real-world scenarios with label noise.