Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing

2025/09/18 by Zichen Wu, Wu, Zichen, Hsiu-Yuan Huang +3 · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Multimedia (cs.MM) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.15361

openalex publication_date 2025/09/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multimodal Large Language Models (MLLMs) have shown substantial capabilities in integrating visual and textual information, yet frequently rely on spurious correlations, undermining their robustness and generalization in complex multimodal reasoning tasks. This paper addresses the critical challenge of superficial correlation bias in MLLMs through a novel causal mediation-based debiasing framework. Specially, we distinguishing core semantics from spurious textual and visual contexts via counterfactual examples to activate training-stage debiasing and employ a Mixture-of-Experts (MoE) architecture with dynamic routing to selectively engages modality-specific debiasing experts. Empirical evaluation on multimodal sarcasm detection and sentiment analysis tasks demonstrates that our framework significantly surpasses unimodal debiasing strategies and existing state-of-the-art models.

Citations

• Aligning Multimodal LLM with Human Preference: A Survey
• SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs
• Multimodal Sentiment Analysis Based on Causal Reasoning
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
• Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective
• MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs
• Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling
• Hallucination of Multimodal Large Language Models: A Survey
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Robust Emotion Recognition in Context Debiasing
• Towards Multimodal Sentiment Analysis Debiasing via Bias Purification
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
• Debiasing Multimodal Models via Causal Information Minimization
• COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs
• Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP
• MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System
• Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement
• CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
• Counterfactual Adversarial Learning with Representation Interpolation
• Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
• Learning Transferable Visual Models From Natural Language Supervision
• Counterfactual VQA: A Cause-Effect Look at Language Bias
• Counterfactual Samples Synthesizing for Robust Visual Question Answering
• Towards Causal VQA: Revealing and Reducing Spurious Correlations by\n Invariant and Covariant Semantic Editing
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
• Don't Just Assume; Look and Answer: Overcoming Priors for Visual\n Question Answering
• Practical Bayesian Optimization of Machine Learning Algorithms

Cited by

• OmniMER: Auxiliary-Enhanced LLM Adaptation for Indonesian Multimodal Emotion Recognition

Related

• Debiasing Multimodal Sarcasm Detection with Contrastive Learning
• Towards Minimal Causal Representations for Human Multimodal Language Understanding
• When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning
• Debiasing Multimodal Large Language Models via Penalization of Language Priors
• A Multi-LLM Debiasing Framework