Robustness Emerges Early in Training Dynamics, but Is Not Preserved

2026/08/05 by Jiangang Yang, Wenhui Shi, Lu Hu +2
Computer Science · #cs.LG #cs.CV

paper · pdf

Accepted by ECCV2026

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Robustness to natural corruptions remains a fundamental challenge for deep neural networks. In this paper, we identify a robustness fading phenomenon where shallow layers spontaneously develop robust representations and flat loss landscapes in early training, yet these properties are not preserved during standard convergence. To address this, we propose a framework that performs strategic interventions on training dynamics to stabilize the empirically identified early-emergent robust priors. Our approach includes two parameter-free strategies: Early-Phase Stabilization~(EPS) and Asymmetric Weight Reversion~(AWR), which stabilize or recover robust shallow configurations without modifying the model architecture or introducing learnable parameters. Extensive experiments demonstrate the efficacy of our framework across various benchmarks and architectures, yielding significant gains in downstream transfer, dynamic adaptation, and diverse computer vision applications.

Citations

• Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness
• Improving robustness to corruptions with multiplicative weight perturbations
• Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning
• MambaOut: Do We Really Need Mamba for Vision?
• Can Biases in ImageNet Models Explain Generalization?
• Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification
• Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?
• Adaptive Sharpness-Aware Pruning for Robust Sparse Networks
• Robust Test-Time Adaptation in Dynamic Scenarios
• A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking
• Towards Stable Test-Time Adaptation in Dynamic Wild World
• Robust Mean Teacher for Continual and Gradual Test-Time Adaptation
• Enhance the Visual Representation via Discrete Adversarial Training
• Understanding Robust Overfitting of Adversarial Training and Beyond
• EfficientFormer: Vision Transformers at MobileNet Speed
• Contrastive Test-Time Adaptation
• 3D Common Corruptions and Data Augmentation
• Improving Robustness by Enhancing Weak Subnets
• OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images
• Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• ResNet strikes back: An improved training procedure in timm
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Improving robustness against common corruptions with frequency biased models
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Overfitting in adversarially robust deep learning
• The Early Phase of Neural Network Training
• Gabor Layers Enhance Network Robustness
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Benchmarking the Robustness of Semantic Segmentation Models
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• A Fourier Perspective on Model Robustness in Computer Vision
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• When Does Label Smoothing Help?
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Adversarial Examples Are Not Bugs, They Are Features
• Similarity of Neural Network Representations Revisited
• GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Do ImageNet Classifiers Generalize to ImageNet?
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Representation Learning with Contrastive Predictive Coding
• On the Spectral Bias of Neural Networks
• How Does Batch Normalization Help Optimization?
• AutoAugment: Learning Augmentation Policies from Data
• Averaging Weights Leads to Wider Optima and Better Generalization
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Measuring the Intrinsic Dimension of Objective Landscapes
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Visualizing the Loss Landscape of Neural Nets
• Adversarial Examples: Attacks and Defenses for Deep Learning
• mixup: Beyond Empirical Risk Minimization
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Understanding intermediate layers using linear classifier probes
• Wide Residual Networks
• Deep Residual Learning for Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks