MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

2022/10/11 by Yatai Ji, Junjie Wang, Ji, Yatai +15 · 18 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2210.05335

openalex publication_date 2022/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets. Such uncertainty is problematic for our interpretation, including inter- and intra-modal uncertainty. Little effort has studied the modeling of this uncertainty, particularly in pre-training on unlabeled datasets and fine-tuning in task-specific downstream datasets. In this paper, we project the representations of all modalities as probabilistic distributions via a Probability Distribution Encoder (PDE) by utilizing sequence-level interactions. Compared to the existing deterministic methods, such uncertainty modeling can convey richer multimodal semantic information and more complex relationships. Furthermore, we integrate uncertainty modeling with popular pre-training frameworks and propose suitable pre-training tasks: Distribution-based Vision-Language Contrastive learning (D-VLC), Distribution-based Masked Language Modeling (D-MLM), and Distribution-based Image-Text Matching (D-ITM). The fine-tuned models are applied to challenging downstream tasks, including image-text retrieval, visual question answering, visual reasoning, and visual entailment, and achieve state-of-the-art results.

Cited by

• Calibrating Uncertainty for Zero-Shot Adversarial CLIP
• Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models
• Robust and Generalizable GNN Fine-Tuning via Uncertainty-aware Adapter Learning
• Cross Modal Fine-Grained Alignment via Granularity-Aware and Region-Uncertain Modeling
• ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology
• Hallucination Filtering in Radiology Vision-Language Models Using Discrete Semantic Entropy
• Xi+: Uncertainty Supervision for Robust Speaker Embedding
• Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Images
• Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoring
• TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
• Uncertainty-driven Embedding Convolution
• UGG-ReID: Uncertainty-Guided Graph Model for Multi-Modal Object Re-Identification
• Text-guided multi-stage cross-perception network for medical image segmentation
• Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models
• PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning
• Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
• Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysis
• UG-UMRE: Uncertainty-Guided Modality Augmentation and Distributional Calibration for Unified Multimodal Relation Extraction

Related

• Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
• Post-hoc Probabilistic Vision-Language Models
• HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Models
• Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models
• Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs