Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs
2025/05/28 by Xudong Li, Li, Xudong, Mengdan Zhang +19
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences
paper · pdf · doi:10.48550/arxiv.2505.22396
openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Multi-modal Large Language Models (MLLMs) excel at single-image tasks but struggle with multi-image understanding due to cross-modal misalignment, leading to hallucinations (context omission, conflation, and misinterpretation). Existing methods using Direct Preference Optimization (DPO) constrain optimization to a solitary image reference within the input sequence, neglecting holistic context modeling. We propose Context-to-Cue Direct Preference Optimization (CcDPO), a multi-level preference optimization framework that enhances per-image perception in multi-image settings by zooming into visual clues -- from sequential context to local details. It features: (i) Context-Level Optimization : Re-evaluates cognitive biases underlying MLLMs' multi-image context comprehension and integrates a spectrum of low-cost global sequence preferences for bias mitigation. (ii) Needle-Level Optimization : Directs attention to fine-grained visual details through region-targeted visual prompts and multimodal preference supervision. To support scalable optimization, we also construct MultiScope-42k, an automatically generated dataset with high-quality multi-level preference pairs. Experiments show that CcDPO significantly reduces hallucinations and yields consistent performance gains across general single- and multi-image tasks.
Citations
- Seed1.5-VL Technical Report
- VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models
- Identifying and Mitigating Position Bias of Multi-image Vision-Language Models
- CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation
- Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs
- Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding
- Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
- Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
- V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization
- MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
- Modality-Fair Preference Optimization for Trustworthy MLLM Alignment
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
- FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Few-Shot Image Quality Assessment via Adaptation of Vision-Language Models
- CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
- mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
- LLaVA-OneVision: Easy Visual Task Transfer
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
- SAM 2: Segment Anything in Images and Videos
- ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
- MIBench: Evaluating Multimodal Large Language Models over Multiple Images
- LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
- LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference
- Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
- MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
- ReMI: A Dataset for Reasoning with Multiple Images
- Needle In A Multimodal Haystack
- Automated Multi-level Preference for MLLMs
- What matters when building vision-language models?
- BLINK: Multimodal Large Language Models Can See but Not Perceive
- Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
- Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples
- Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- Generative Multimodal Models are In-Context Learners
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
- HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- Large Language Models are Visual Reasoning Coordinators
- Improved Baselines with Visual Instruction Tuning
- Aligning Large Multimodal Models with Factually Augmented RLHF
- OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
- MMBench: Is Your Multi-modal Model an All-around Player?
- OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Evaluating Object Hallucination in Large Vision-Language Models
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
- Visual Instruction Tuning
- Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
- What does CLIP know about a red circle? Visual prompt engineering for VLMs
- Micrograph segmentations for DDEVD
- Segment Anything
- GPT-4 Technical Report
- ViperGPT: Visual Inference via Python Execution for Reasoning
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- Flamingo: a Visual Language Model for Few-Shot Learning
- Training language models to follow instructions with human feedback
- DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
- CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
- OK-VQA: A Visual Question Answering Benchmark Requiring External\n Knowledge
- A Corpus for Reasoning About Natural Language Grounded in Photographs
- Object Hallucination in Image Captioning
- Making the V in VQA Matter: Elevating the Role of Image Understanding in\n Visual Question Answering
- A Diagram Is Worth A Dozen Images
- VQA: Visual Question Answering
- Microsoft COCO: Common Objects in Context
- Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Related