AVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time

2025/05/31 by Maharana, Sarthak Kumar, Kushwaha, Saksham Singh, Zhang, Baoming +4
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2506.00358

Abstract

While recent audio-visual models have demonstrated impressive performance, their robustness to distributional shifts at test-time remains not fully understood. Existing robustness benchmarks mainly focus on single modalities, making them insufficient for thoroughly assessing the robustness of audio-visual models. Motivated by real-world scenarios where shifts can occur simultaneously in both audio and visual modalities, we introduce AVROBUSTBENCH, a comprehensive benchmark designed to evaluate the test-time robustness of audio-visual recognition models. AVROBUSTBENCH comprises four audio-visual benchmark datasets, AUDIOSET-2C, VGGSOUND-2C, KINETICS-2C, and EPICKITCHENS-2C, each incorporating 75 bimodal audio-visual corruptions that are co-occurring and correlated. Through extensive evaluations, we observe that state-of-the-art supervised and self-supervised audio-visual models exhibit declining robustness as corruption severity increases. Furthermore, online test-time adaptation (TTA) methods, on VGGSOUND-2C and KINETICS-2C, offer minimal improvements in performance under bimodal corruptions. We further propose AV2C, a simple TTA approach enabling on-the-fly cross-modal fusion by penalizing high-entropy samples, which achieves improvements on VGGSOUND-2C. We hope that AVROBUSTBENCH will steer the development of more effective and robust audio-visual TTA approaches. Our code is available \hrefhttps://github.com/sarthaxxxxx/AV-C-Robustness-Benchmarkhere.

Citations

• From Waveforms to Pixels: A Survey on Audio-Visual Segmentation
• Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises
• WATT: Weight Average Test-Time Adaptation of CLIP
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
• A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models
• Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving
• CLIPArTT: Adaptation of CLIP to New Domains at Test Time
• TIM: A Time Interval Machine for Audio-Visual Action Recognition
• Efficient Test-Time Adaptation of Vision-Language Models
• EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning
• Speech Robust Bench: A Robustness Benchmark For Speech Recognition
• MM-LLMs: Recent Advances in MultiModal Large Language Models
• High-Quality Visually-Guided Sound Separation from Diverse Categories
• Robustness Analysis on Foundational Segmentation Models
• Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
• PandaGPT: One Model To Instruction-Follow Them All
• Annotation-free Audio-Visual Segmentation
• ImageBind: One Embedding Space To Bind Them All
• Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring
• Towards Stable Test-Time Adaptation in Dynamic Wild World
• MAViL: Masked Audio-Video Learners
• Audiovisual Masked Autoencoders
• Contrastive Audio-Visual Masked Autoencoder
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
• Improving Test-Time Adaptation via Shift-agnostic Weight Regularization and Nearest Source Prototypes
• Audio-Visual Segmentation
• Large-scale Robustness Analysis of Video Action Recognition Models
• An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation
• MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation
• Contrastive Test-Time Adaptation
• Efficient Test-Time Model Adaptation without Forgetting
• Continual Test-Time Domain Adaptation
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
• DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Omnivore: A Single Model for Many Visual Modalities
• A ConvNet for the 2020s
• A ConvNet for the 2020s
• How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
• Measure and Improve Robustness in NLP Models: A Survey
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
• Wav2CLIP: Learning Robust Audio Representations From CLIP
• MEMO: Test Time Robustness via Adaptation and Augmentation
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Attention Bottlenecks for Multimodal Fusion
• AudioCLIP: Extending CLIP to Image, Text and Audio
• Defending Multimodal Fusion Models against Single-Source Adversaries
• When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks
• If your data distribution shifts, use self-learning
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Can audio-visual integration strengthen robustness under multimodal attacks?
• Slow-Fast Auditory Streams For Audio Recognition
• Perceiver: General Perception with Iterative Attention
• Learning Transferable Visual Models From Natural Language Supervision
• Online Continual Learning in Image Classification: An Empirical Survey
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• RobustBench: a standardized adversarial robustness benchmark
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Improving robustness against common corruptions by covariate shift adaptation
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• VGGSound: A Large-scale Audio-Visual Dataset
• EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action\n Recognition
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• WHAM!: Extending Speech Separation to Noisy Environments
• Co-Separating Sounds of Visual Objects
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Do ImageNet Classifiers Generalize to ImageNet?
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Generalisation in humans and deep neural networks
• Representation Learning with Contrastive Predictive Coding
• The Sound of Pixels
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• Fast Scene Understanding for Autonomous Driving
• Comparing deep neural networks against humans: object recognition when\n the signal gets weaker
• Attention Is All You Need
• Look, Listen and Learn
• The Kinetics Human Action Video Dataset
• Layer Normalization
• Revisiting Batch Normalization For Practical Domain Adaptation
• Distilling the Knowledge in a Neural Network
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• On discrete cosine transform
• Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift
• The Development of Embodied Cognition: Six Lessons from Babies
• Divergence measures based on the Shannon entropy

Related

• Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring
• Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation
• Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation
• AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
• AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch