Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

2019/03/28 by Dan Hendrycks, Thomas G. Dietterich, Hendrycks, Dan +1 · 207 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1903.12261

openalex publication_date 2019/03/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.

Cited by

• VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
• MulRobBench: A Decision-Level Benchmark for Safe and Security-Policy-Compliant Multimodal UAV Agents
• AGVBench: A Reliability-Oriented Benchmark of Data Augmentation for Vein Recognition
• Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
• Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations
• Source-Free Controlled Adaptation of Teachers for Continual Test-Time Adaptation
• Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation
• Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation
• PIcsC: Partitioning-Induced Covariate Shift Correction
• ReLATE: Reliability-Guided Evidence Fusion for Robust UAV--Satellite cross-view Geo-Localization
• Structural Preservation Governs Data Augmentation in Deep Learning-Based Laser Speckle Material Classification
• Trustworthy Medical Segmentation: Uncertainty-Aware U-Net Evaluation Under Clinical Image Degradation
• Granular-ball Guided Masking: Structure-aware Data Augmentation
• Clever Hans in Chemistry: Chemist Style Signals Confound Activity Prediction on Public Benchmarks
• Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking
• Multi-Layer Confidence Scoring for Detection of Out-of-Distribution Samples, Adversarial Attacks, and In-Distribution Misclassifications
• Efficient Jailbreak Mitigation Using Semantic Linear Classification in a Multi-Staged Pipeline
• Can We Test Consciousness Theories on AI? Ablations, Markers, and Robustness
• The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs
• Stylized Synthetic Augmentation further improves Corruption Robustness
• SUPER -- A Framework for Sensitivity-based Uncertainty-aware Performance and Risk Assessment in Visual Inertial Odometry
• From Code to Field: Evaluating the Robustness of Convolutional Neural Networks for Disease Diagnosis in Mango Leaves
• Efficient Vision-Language Reasoning via Adaptive Token Pruning
• Progressive Conditioned Scale-Shift Recalibration of Self-Attention for Online Test-time Adaptation
• Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Video
• On the Bayes Inconsistency of Disagreement Discrepancy Surrogates
• Neural Collapse in Test-Time Adaptation
• Hands-on Evaluation of Visual Transformers for Object Recognition and Detection
• Uncertainty-Aware Subset Selection for Robust Visual Explainability under Distribution Shifts
• Instance-Aware Test-Time Segmentation for Continual Domain Shifts
• Towards Reliable Test-Time Adaptation: Style Invariance as a Correctness Likelihood
• Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models
• Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation
• Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks
• Walking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networks
• RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving
• Adam Simplified: Bias Correction Debunked
• Enhancing Conformal Prediction via Class Similarity
• Benchmarking Corruption Robustness of LVLMs: A Discriminative Benchmark and Robustness Alignment Metric
• DriveFlow: Rectified Flow Adaptation for Robust 3D Object Detection in Autonomous Driving
• LookSharp: Attention Entropy Minimization for Test-Time Adaptation
• PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation
• DHAuDS: A Dynamic and Heterogeneous Audio Benchmark for Test-Time Adaptation
• Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach
• Understanding Private Learning From Feature Perspective
• Vision Language Models are Confused Tourists
• ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation
• A Dataset and Baseline for Deep Learning-Based Visual Quality Inspection in Remanufacturing
• SNAP: Low-Latency Test-Time Adaptation with Sparse Updates
• Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines
• Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures
• RobustGait: Robustness Analysis for Appearance Based Gait Recognition
• A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning
• DINO-Detect: A Simple yet Effective Framework for Blur-Robust AI-Generated Image Detection
• Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation
• Calibrated Adversarial Sampling: Multi-Armed Bandit-Guided Generalization Against Unforeseen Attacks
• Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification
• Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
• "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with VLMs
• Provable Repair of Deep Neural Network Defects by Preimage Synthesis and Property Refinement
• ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora
• RobustA: Robust Anomaly Detection in Multimodal Data
• DARN: Dynamic Adaptive Regularization Networks for Efficient and Robust Foundation Model Adaptation
• Decoupled Entropy Minimization
• Test Time Adaptation Using Adaptive Quantile Recalibration
• Learning with Category-Equivariant Architectures for Human Activity Recognition
• Purrturbed but Stable: Human-Cat Invariant Representations Across CNNs, ViTs and Self-Supervised ViTs
• Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering
• T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis
• ODP-Bench: Benchmarking Out-of-Distribution Performance Prediction
• C-LEAD: Contrastive Learning for Enhanced Adversarial Defense
• VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes
• Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks
• EgoSafe: A First-Person Mobile-Captured Benchmark for Visual Safety Understanding
• MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space
• Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
• Humanoid-inspired Causal Representation Learning for Domain Generalization
• Eigen-Value: Efficient Domain-Robust Data Valuation via Eigenvalue-Based Approach
• The Benchmarking Epistemology: Construct Validity for Evaluating Machine Learning Models
• Prediction-Powered Semi-Supervised Learning with Online Power Tuning
• Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions
• Buffer layers for Test-Time Adaptation
• H-SPLID: HSIC-based Saliency Preserving Latent Information Decomposition
• What Does It Take to Build a Performant Selective Classifier?
• CO-PFL: Contribution-Oriented Personalized Federated Learning for Heterogeneous Networks
• Knowledge Distillation of Uncertainty using Deep Latent Factor Model
• Uncertainty Estimation by Flexible Evidential Deep Learning
• Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
• Domain Generalizable Continual Learning
• Bridging Symmetry and Robustness: On the Role of Equivariance in Enhancing Adversarial Robustness
• SteeringTTA: Guiding Diffusion Trajectories for Robust Test-Time-Adaptation
• Generalist++: A Meta-learning Framework for Mitigating Trade-off in Adversarial Training
• A Function Centric Perspective On Flat and Sharp Minima
• Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation
• Adversarial Attacks Leverage Interference Between Features in Superposition
• Joint Discriminative-Generative Modeling via Dual Adversarial Training
• Deep Edge Filter: Return of the Human-Crafted Layer in Deep Learning
• Efficient Edge Test-Time Adaptation via Latent Feature Coordinate Correction
• Self-Training with Dynamic Weighting for Robust Gradual Domain Adaptation
• SpurBreast: A Curated Dataset for Investigating Spurious Correlations in Real-world Breast MRI Classification
• Tight Robustness Certificates and Wasserstein Distributional Attacks for Deep Neural Networks
• Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals
• SynthID-Image: Image watermarking at internet scale
• Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
• Evaluating the Robustness of a Production Malware Detection System to Transferable Adversarial Attacks
• On the Occurence of Critical Learning Periods in Neural Networks
• Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation
• Injecting Hallucinations in Autonomous Vehicles: A Component-Agnostic Safety Evaluation Framework
• Revisiting Mixout: An Overlooked Path to Robust Finetuning
• LATTA: Langevin-Anchored Test-Time Adaptation for Enhanced Robustness and Stability
• NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering
• Bridging Reasoning to Learning: Unmasking Illusions using Complexity Out of Distribution Generalization
• Spatially Grounded Concept-Based Image Classification
• Technical note on Sequential Test-Time Adaptation via Martingale-Driven Fisher Prompting
• Technical note on Fisher Information for Robust Federated Cross-Validation
• Neon: Negative Extrapolation From Self-Training Improves Image Generation
• Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking
• Sequence-Preserving Dual-FoV Defense for Traffic Sign and Light Recognition in Autonomous Vehicles
• Quantum Probabilistic Label Refining: Enhancing Label Quality for Robust Image Classification
• VLOD-TTA: Test-Time Adaptation of Vision-Language Object Detectors
• CODED-SMOOTHING: Coding Theory Helps Generalization
• The Impact of Scaling Training Data on Adversarial Robustness
• Annotation-Efficient Active Test-Time Adaptation with Conformal Prediction
• SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation
• RDD: Pareto Analysis of the Rate-Distortion-Distinguishability Trade-off
• BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression
• Visual CoT Makes VLMs Smarter but More Fragile
• Evaluating the Impact of Radiographic Noise on Chest X-ray Semantic Segmentation and Disease Classification Using a Scalable Noise Injection Framework
• On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
• Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Adversarial Scheduling
• ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse
• TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
• Zubov-Net: Adaptive Stability for Neural ODEs Reconciling Accuracy with Robustness
• POEM: Explore Unexplored Reliable Samples to Enhance Test-Time Adaptation
• TRiCo: Triadic Game-Theoretic Co-Training for Robust Semi-Supervised Learning
• Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
• Unlocking Noise-Resistant Vision: Key Architectural Secrets for Robust Models
• Feature Dynamics as Implicit Data Augmentation: A Depth-Decomposed View on Deep Neural Network Generalization
• ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research
• Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderation
• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
• The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
• A Validation Strategy for Deep Learning Models: Evaluating and Enhancing Robustness
• MLF-4DRCNet: Multi-Level Fusion with 4D Radar and Camera for 3D Object Detection in Autonomous Driving
• Losing the Plot: How VLM responses degrade on imperfect charts
• Graph Coloring for Multi-Task Learning
• AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
• VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models
• FedERL: Federated Efficient and Robust Learning for Common Corruptions
• An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity
• Cumulative Consensus Score: Label-Free and Model-Agnostic Evaluation of Object Detectors in Deployment
• The Lifecycle Principle: Stabilizing Dynamic Neural Networks with State Memory
• Post-Hoc Split-Point Self-Consistency Verification for Efficient, Unified Quantification of Aleatoric and Epistemic Uncertainty in Deep Learning
• Promoting Shape Bias in CNNs: Frequency-Based and Contrastive Regularization for Corruption Robustness
• A Modern Look at Simplicity Bias in Image Classification Tasks
• Model-Agnostic Open-Set Air-to-Air Visual Object Detection for Reliable UAV Perception
• ACE and Diverse Generalization via Selective Disagreement
• From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
• ConstStyle: Robust Domain Generalization with Unified Style Transformation
• Prior Distribution and Model Confidence
• Extracting Uncertainty Estimates from Mixtures of Experts for Semantic Segmentation
• Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularization
• Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization
• An Investigation of Visual Foundation Models Robustness
• ADVMEM: Adversarial Memory Initialization for Realistic Test-Time Adaptation via Tracklet-Based Benchmarking
• Domain Adaptation via Feature Refinement
• RT-VLM: Re-Thinking Vision Language Model with 4-Clues for Real-World Object Recognition Robustness
• MorphGen: Morphology-Guided Representation Learning for Robust Single-Domain Generalization in Histopathological Cancer Classification
• Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
• Entropy-Based Non-Invasive Reliability Monitoring of Convolutional Neural Networks
• Adaptive Dual Uncertainty Optimization: Boosting Monocular 3D Object Detection under Test-Time Shifts
• Active Learning for Neurosymbolic Program Synthesis
• Image Quality Assessment for Machines: Paradigm, Large-scale Database, and Models
• Gradient Rectification for Robust Calibration under Distribution Shift
• ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimation
• Quantization Robustness to Input Degradations for Object Detection
• Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
• Efficient Multi-Source Knowledge Transfer by Model Merging
• Flatness-aware Curriculum Learning via Adversarial Difficulty
• OASIS: Open-world Adaptive Self-supervised and Imbalanced-aware System
• FOCUS: Frequency-Optimized Conditioning of DiffUSion Models for mitigating catastrophic forgetting during Test-Time Adaptation
• Multi-Plasticity Synergy with Adaptive Mechanism Assignment for Training Spiking Neural Networks
• SNAP-UQ: Self-supervised Next-Activation Prediction for Single-Pass Uncertainty in TinyML
• TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML
• Automated Model Evaluation for Object Detection via Prediction Consistency and Reliability
• Processing and acquisition traces in visual encoders: What does CLIP know about your camera?
• BERTector: An Intrusion Detection Framework Constructed via Joint-dataset Learning Based on Language Model
• On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness
• Harnessing Input-Adaptive Inference for Efficient VLN
• A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking
• Hierarchical Adaptive networks with Task vectors for Test-Time Adaptation
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey
• Keep It Real: Challenges in Attacking Compression-Based Adversarial Purification
• DocVCE: Diffusion-based Visual Counterfactual Explanations for Document Image Classification
• Slice or the Whole Pie? Utility Control for AI Models
• Selection-Based Vulnerabilities: Clean-Label Backdoor Attacks in Active Learning
• Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation
• The Power of Many: Synergistic Unification of Diverse Augmentations for Efficient Adversarial Robustness
• RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging
• Test-Time Model Adaptation for Quantized Neural Networks
• Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models
• Evading Data Provenance in Deep Neural Networks
• Towards Robust Semantic Correspondence: A Benchmark and Insights
• Robust 3D Object Detection using Probabilistic Point Clouds from Single-Photon LiDARs
• Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks

Related

• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Real World Robustness from Systematic Noise
• A simple way to make neural networks robust against diverse image corruptions
• A Survey of Neural Network Robustness Assessment in Image Recognition
• A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking