mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

2023/11/07 by Qinghao Ye, Ye, Qinghao, Haiyang Xu +16 · 86 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2311.04257

Abstract

Multi-modal Large Language Models (MLLMs) have demonstrated impressive instruction abilities across various open-ended tasks. However, previous methods primarily focus on enhancing multi-modal capabilities. In this work, we introduce a versatile multi-modal large language model, mPLUG-Owl2, which effectively leverages modality collaboration to improve performance in both text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design, with the language decoder acting as a universal interface for managing different modalities. Specifically, mPLUG-Owl2 incorporates shared functional modules to facilitate modality collaboration and introduces a modality-adaptive module that preserves modality-specific features. Extensive experiments reveal that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal tasks and achieving state-of-the-art performances with a single generic model. Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality collaboration phenomenon in both pure-text and multi-modal scenarios, setting a pioneering path in the development of future multi-modal foundation models.

Cited by

• CoFi-Dec: Hallucination-Resistant Decoding via Coarse-to-Fine Generative Feedback in Large Vision-Language Models
• VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs
• Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
• Guiding Perception-Reasoning Closer to Human in Blind Image Quality Assessment
• JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction
• PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction
• Investigate the Low-level Visual Perception in Vision-Language based Image Quality Assessment
• Building Reasonable Inference for Vision-Language Models in Blind Image Quality Assessment
• Hierarchical Process Reward Models are Symbolic Vision Learners
• ChartAnchor: Chart Grounding with Structural-Semantic Fidelity
• ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning
• SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
• VICoT-Agent: A Vision-Interleaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysis
• CLASH: A Benchmark for Cross-Modal Contradiction Detection
• Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
• ChartEditor: A Reinforcement Learning Framework for Robust Chart Editing
• Can MLLMs Detect Phishing? A Comprehensive Security Benchmark Suite Focusing on Dynamic Threats and Multimodal Evaluation in Academic Environments
• TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing
• AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoning
• Draft and Refine with Visual Experts
• Revisiting MLLM Based Image Quality Assessment: Errors and Remedy
• EMODIS: A Benchmark for Context-Dependent Emoji Disambiguation in Large Language Models
• S²LM: Towards Semantic Steganography via Large Language Models
• Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
• SmoothGuard: Defending Multimodal Large Language Models with Noise Perturbation and Clustering Aggregation
• MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models
• Composition-Grounded Instruction Synthesis for Visual Reasoning
• MorphoBench: A Benchmark with Difficulty Adaptive to Model Reasoning
• Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graph
• UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation
• DREAM: A Benchmark Study for Deepfake REalism AssessMent
• Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning
• HandEval: Taking the First Step Towards Hand Quality Evaluation in Generated Images
• Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localization
• MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
• Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
• Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
• AgenticIQA: An Agentic Framework for Adaptive and Interpretable Image Quality Assessment
• Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline
• Training-Free Multimodal Deepfake Detection via Graph Reasoning
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
• VCE: A Zero-Cost Hallucination Mitigation Method of LVLMs via Visual Contrastive Editing
• The Photographer Eye: Teaching Multimodal Large Language Models to Understand Image Aesthetics like Photographers
• Can GRPO Boost Complex Multimodal Table Understanding?
• Generalizable Geometric Image Caption Synthesis
• TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding
• AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
• Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMs
• Explain Before You Answer: A Survey on Compositional Visual Reasoning
• AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
• Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis
• TIDE: Achieving Balanced Subject-Driven Image Generation via Target-Instructed Diffusion Enhancement
• Text4Seg++: Advancing Image Segmentation via Generative Language Modeling
• Parameter-Efficient Adaptation of mPLUG-Owl2 via Pixel-Level Visual Prompts for NR-IQA
• Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection
• Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens
• MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
• CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification
• Improving Alignment in LVLMs with Debiased Self-Judgment
• GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
• Aligning Moments in Time using Video Queries
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
• Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
• EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understanding
• ViDA-UGC: Detailed Image Quality Analysis via Visual Distortion Assessment for UGC Images
• RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts
• Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
• JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics
• SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs
• A Survey on Training-free Alignment of Large Language Models
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding
• SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
• Refine-IQA: Multi-Stage Reinforcement Finetuning for Perceptual Image Quality Assessment
• MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processing
• SGCap: Decoding Semantic Group for Zero-shot Video Captioning
• MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
• UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries
• Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment
• Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
• METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
• RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning
• LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Tasks
• Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
• LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences

Related

• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
• The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
• M4SC: An MLLM-based Multi-modal, Multi-task and Multi-user Semantic Communication System
• Model Composition for Multimodal Large Language Models
• Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs