Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey
2025/03/16 by Yaoting Wang, Wang, Yaoting, Shengqiong Wu +11 · 1 voice · 91 citations
Computer Science · #cs.CV
paper · pdf · doi:10.48550/arxiv.2503.12605
Abstract
By extending the advantage of chain-of-thought (CoT) reasoning in human-like step-by-step processes to multimodal contexts, multimodal CoT (MCoT) reasoning has recently garnered significant research attention, especially in the integration with multimodal large language models (MLLMs). Existing MCoT studies design various methodologies and innovative reasoning paradigms to address the unique challenges of image, video, speech, audio, 3D, and structured data across different modalities, achieving extensive success in applications such as robotics, healthcare, autonomous driving, and multimodal generation. However, MCoT still presents distinct challenges and opportunities that require further focus to ensure consistent thriving in this field, where, unfortunately, an up-to-date review of this domain is lacking. To bridge this gap, we present the first systematic survey of MCoT reasoning, elucidating the relevant foundational concepts and definitions. We offer a comprehensive taxonomy and an in-depth analysis of current methodologies from diverse perspectives across various application scenarios. Furthermore, we provide insights into existing challenges and future research directions, aiming to foster innovation toward multimodal AGI.
Cited by
- See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoning
- Visual-Aware CoT: Achieving High-Fidelity Visual Consistency in Unified Models
- V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions
- CogDoc: Towards Unified thinking in Documents
- Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs
- MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
- COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligence
- JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization
- AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture
- Analyzing Image Beyond Visual Aspect: Image Emotion Classification via Multiple-Affective Captioning
- From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
- Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models
- Hindsight Distillation Reasoning with Knowledge Encouragement Preference for Knowledge-based Visual Question Answering
- MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
- LaRe: Latent Refocusing for Multimodal Reasoning
- Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
- Perception, Understanding and Reasoning, A Multimodal Benchmark for Video Fake News Detection
- On the Faithfulness of Visual Thinking: Measurement and Enhancement
- CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning
- Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoning
- CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation
- A Survey on Agentic Multimodal Large Language Models
- ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
- RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
- A Trustworthy Industrial Fault Diagnosis Architecture Integrating Probabilistic Models and Large Language Models
- MuSLR: Multimodal Symbolic Logical Reasoning
- From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
- PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Images
- Fast Thinking for Large Language Models
- Decoupling Reasoning and Perception: An LLM-LMM Framework for Faithful Visual Reasoning
- GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformation
- From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs
- MOSS-ChatV: Reinforcement Learning with Process Reasoning Reward for Video Temporal Reasoning
- Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models
- Fine-Tuning LLMs to Analyze Multiple Dimensions of Code Review: A Maximum Entropy Regulated Long Chain-of-Thought Approach
- RadAgents: Multimodal Agentic Reasoning for Chest X-ray Interpretation with Radiologist-like Workflows
- The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
- LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detection
- Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generation
- Mitigating Strategy-Selection Bias in Reasoning for More Effective Test-Time Scaling
- Explain Before You Answer: A Survey on Compositional Visual Reasoning
- A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
- Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
- Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
- Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
- MolmoAct: Action Reasoning Models that can Reason in Space
- Coherent Multimodal Reasoning with Iterative Self-Evaluation for Vision-Language Models
- CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos
- CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning
- VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning
- A2R2: Advancing Img2LaTeX Conversion via Visual Reasoning with Attention-Guided Refinement
- Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
- Position: Reasoning After Perception Means Reasoning Without Vision
- Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasoner
- ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models
- Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
- BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
- Look-Back: Implicit Visual Re-focusing in MLLM Reasoning
- MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
- ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing
- Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
- Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search
- What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding
- Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience
- Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challenges
- Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks
- Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
- MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book
- Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzles
- Reinforced Reasoning for Embodied Planning
- Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning
- Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
- VisCRA: A Visual Chain Reasoning Attack for Jailbreaking Multimodal Large Language Models
- SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoring
- Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models
- Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning
- MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning
- RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs
- From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualization
- ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representations
- VISTA: Mitigating Semantic Inertia in Video-LLMs via Training-Free Dynamic Chain-of-Thought Routing
- CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts
- Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation
- 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models
- Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
- TIR-Flow: Active Video Search and Reasoning with Frozen VLMs
- Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
- Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?
- SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models
Discussions
Related