Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment

2025/06/08 by Zhao, Pengfei, Luan, Rongbo, Zhang, Wei +2 · 2 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.06970

Abstract

Despite Contrastive Language-Image Pretraining (CLIP)'s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its feature space. Intriguingly, we discover that off-the-shelf MLLMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality alignment mechanisms fundamentally limits their potential. In this work, We introduce MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that leverages the fine grained alignment priors inherent in MLLM to guide cross modal representation learning. MAPLE formulates the learning process as reinforcement learning with two key components: (1) Automatic preference data construction using off-the-shelf MLLM, and (2) a new Relative Preference Alignment (RPA) loss, which adapts Direct Preference Optimization (DPO) to the embedding learning setting. Experimental results show that our preference-guided alignment achieves substantial gains in fine-grained cross-modal retrieval, underscoring its effectiveness in handling nuanced semantic distinctions.

Citations

• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Qwen2.5-VL Technical Report
• Scalable Vision Language Model Training via High Quality Data Curation
• MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
• Bridging the Modality Gap: Dimension Information Alignment and Sparse Spatial Constraint for Image-Text Matching
• NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• E5-V: Universal Embeddings with Multimodal Large Language Models
• Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
• EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
• Preference Ranking Optimization for Human Alignment
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection
• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
• DINOv2: Learning Robust Visual Features without Supervision
• Sigmoid Loss for Language Image Pre-Training
• SemDeDup: Data-efficient learning at web-scale through semantic deduplication
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning
• LAION-5B: An open large-scale dataset for training next generation image-text models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
• Training language models to follow instructions with human feedback
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Learning Transferable Visual Models From Natural Language Supervision
• Document Ranking with a Pretrained Sequence-to-Sequence Model
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Microsoft COCO: Common Objects in Context
• Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
• Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data

Cited by

• Adapting MLLMs for Nuanced Video Retrieval
• FreeRet: MLLMs as Training-Free Retrievers

Related

• Modality-Fair Preference Optimization for Trustworthy MLLM Alignment
• OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference
• MLLMs are Deeply Affected by Modality Bias
• MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
• Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs