A survey on multimodal large language models

2023/06/23 by Shukang Yin, Chaoyou Fu, Sirui Zhao +4 · 2 voices · 13 citations
Computer Science · #Computer science #Linguistics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Philosophy #Topic Modeling #cs.AI #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.1093/nsr/nwae403

arxiv published 2023/06/23 · openalex publication_date 2024/11/12 · arxiv updated 2024/11/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Recently, the multimodal large language model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful large language models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of the MLLM, such as writing stories based on images and optical character recognition-free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that can compete with or even outperform GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First, we present the basic formulation of the MLLM and delineate its related concepts, including architecture, training strategy and data, as well as evaluation. Then, we introduce research topics about how MLLMs can be extended to support more granularity, modalities, languages and scenarios. We continue with multimodal hallucination and extended techniques, including multimodal in-context learning, multimodal chain of thought and LLM-aided visual reasoning. To conclude the paper, we discuss existing challenges and point out promising research directions.

Citations

• Fine-Tuning Language Models from Human Preferences
• PathVQA: 30000+ Questions for Medical Visual Question Answering
• TALM: Tool Augmented Language Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Scaling Instruction-Finetuned Language Models
• Visual Programming: Compositional visual reasoning without training
• LLaMA: Open and Efficient Foundation Language Models
• MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• Instruction Tuning with GPT-4
• Chain of Thought Prompt Tuning in Vision Language Models
• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
• Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings
• Caption Anything: Interactive Image Description with Diverse Multimodal Controls
• MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
• VideoChat: Chat-Centric Video Understanding
• ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
• Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
• MIMIC-IT: Multi-Modal In-Context Instruction Tuning
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• Evaluation and Analysis of Hallucination in Large Vision-Language Models
• An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
• What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
• On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving
• To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
• SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
• Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models
• Silkie: Preference Distillation for Large Visual Language Models
• A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
• AppAgent: Multimodal Agents as Smartphone Users
• Segment Anything
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
• ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
• Efficient Multimodal Learning from Data-centric Perspective
• TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

Cited by

• Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models
• BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning
• Simple Visual Artifact Detection in Sora-Generated Videos
• TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training
• MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
• Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer
• Geometric Cross-Modal Token Selection for Latency-Constrained Multimodal Token Communication
• SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
• UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space
• Assessing the Utility of Audio Foundation Models for Heart and Respiratory Sound Analysis
• Building Sustainable and Trustworthy Indigenous Knowledge Preservation Ecosystem
• Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
• Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes

Discussions

• A Survey on Multimodal Large Language Models [hn, 1 points, 0 comments]
• arxiv.org/abs/2306.13549 [bsky, 0 points, 0 comments]

Related

• A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
• Hallucination of Multimodal Large Language Models: A Survey
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
• The Revolution of Multimodal Large Language Models: A Survey
• Empowering Multimodal LLMs with External Tools: A Comprehensive Survey