Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions?

2025/05/07 by Shashank Agnihotri, Agnihotri, Shashank, David Schader +7 · 2 citations
Computer Science · Social Sciences · #Computer Vision and Pattern Recognition (cs.CV) #Corruption and Economic Development #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2505.04835

openalex publication_date 2025/05/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning (DL) models are widely used in real-world applications but remain vulnerable to distribution shifts, especially due to weather and lighting changes. Collecting diverse real-world data for testing the robustness of DL models is resource-intensive, making synthetic corruptions an attractive alternative for robustness testing. However, are synthetic corruptions a reliable proxy for real-world corruptions? To answer this, we conduct the largest benchmarking study on semantic segmentation models, comparing performance on real-world corruptions and synthetic corruptions datasets. Our results reveal a strong correlation in mean performance, supporting the use of synthetic corruptions for robustness evaluation. We further analyze corruption-specific correlations, providing key insights to understand when synthetic corruptions succeed in representing real-world corruptions. Open-source Code: https://github.com/shashankskagnihotri/benchmarking_robustness/tree/segmentation_david/semantic_segmentation

Citations

• Towards Class-wise Robustness Analysis
• FAIR-TAT: Improving Model Fairness Using Targeted Adversarial Training
• How Do Training Methods Influence the Utilization of Vision Models?
• Improving 2D Feature Representations by 3D-Aware Fine-Tuning
• Beware of Aliases -- Signal Preservation is Crucial for Robust Image Restoration
• Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive
• Detection Defenses: An Empty Promise against Adversarial Patch Attacks on Optical Flow
• Classification robustness to common optical aberrations
• On the unreasonable vulnerability of transformers for image restoration -- and an easy fix
• Intra- & Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization
• Differentiable Sensor Layouts for End-to-End Learning of Task-Specific Camera Parameters
• Robustifying Token Attention for Vision Transformers
• CosPGD: an efficient white-box adversarial attack for pixel-wise prediction tasks
• InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
• Attacking Motion Estimation with Adversarial Snow
• Intra-Source Style Augmentation for Improved Domain Generalization
• Robust Models are less Over-Confident
• FrequencyLowCut Pooling -- Plug & Play against Catastrophic Overfitting
• A Perturbation-Constrained Adversarial Attack for Evaluating the Robustness of Optical Flow
• 3D Common Corruptions and Data Augmentation
• Masked-attention Mask Transformer for Universal Image Segmentation
• BEiT: BERT Pre-Training of Image Transformers
• SegFormer: Simple and Efficient Design for Semantic Segmentation with\n Transformers
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
• RobustBench: a standardized adversarial robustness benchmark
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Unified Perceptual Parsing for Scene Understanding
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• On the Robustness of Semantic Segmentation Models to Adversarial Attacks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Pyramid Scene Parsing Network
• Xception: Deep Learning with Depthwise Separable Convolutions
• Semantic Understanding of Scenes through the ADE20K Dataset
• Semantic Understanding of Scenes Through the ADE20K Dataset
• Cityscapes dataset for semantic urban scene understanding
• Deep Residual Learning for Image Recognition
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Hypercolumns for Object Segmentation and Fine-grained Localization
• Microsoft COCO: Common Objects in Context
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Cited by

• RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo
• DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions

Related

• Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts
• Benchmarking Corruption Robustness of LVLMs: A Discriminative Benchmark and Robustness Alignment Metric
• Using the Overlapping Score to Improve Corruption Benchmarks
• Frequency-Based Vulnerability Analysis of Deep Learning Models against Image Corruptions
• 3D Common Corruptions and Data Augmentation