Benchmarking Large Multimodal Models against Common Corruptions

2024/01/22 by J. W. Zhang, Tianyu Pang, Zhang, Jiawei +9 · 4 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2401.11943

openalex publication_date 2024/01/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This technical report aims to fill a deficiency in the assessment of large multimodal models (LMMs) by specifically examining the self-consistency of their outputs when subjected to common corruptions. We investigate the cross-modal interactions between text, image, and speech, encompassing four essential generation tasks: text-to-image, image-to-text, text-to-speech, and speech-to-text. We create a comprehensive benchmark, named MMCBench, that covers more than 100 popular LMMs (totally over 150 model checkpoints). A thorough evaluation under common corruptions is critical for practical deployment and facilitates a better understanding of the reliability of cutting-edge LMMs. The benchmarking code is available at https://github.com/sail-sg/MMCBench

Cited by

• LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
• BaseReward: A Strong Baseline for Multimodal Reward Model
• From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship
• R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning

Related

• R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?
• Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness
• Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM
• SEED-Bench-2: Benchmarking Multimodal Large Language Models
• LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models