LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
2024/11/15 by Guowei Xu, Peng Jin, Xu, Guowei +11 · 12 voices · 149 citations
Computer Science · #Natural Language Processing Techniques #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2411.10440
Abstract
Large language models have demonstrated substantial advancements in reasoning capabilities. However, current Vision-Language Models (VLMs) often struggle to perform systematic and structured reasoning, especially when handling complex visual question-answering tasks. In this work, we introduce LLaVA-CoT, a large VLM designed to conduct autonomous multistage reasoning. Unlike chain-of-thought prompting, LLaVA-CoT independently engages in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation. This structured approach enables LLaVA-CoT to achieve marked improvements on reasoning-intensive tasks. To accomplish this, we construct the LLaVA-CoT-100k dataset, integrating samples from various visual question answering sources and providing structured reasoning annotations. Besides, we propose a test-time stage-wise retracing search method (SWIRES), which enables effective and efficient test-time scaling. Remarkably, with only 100k training samples and test-time scaling, LLaVA-CoT not only outperforms its base model by 9.4% on a wide range of multimodal reasoning benchmarks, but also surpasses the performance of larger and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct. The code, dataset, and pre-trained weights are publicly available at https://github.com/PKU-YuanGroup/LLaVA-CoT.
Cited by
- ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding
- MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
- Visual Access Boundaries in Vision-Language Model Reasoning
- K2-Think: A Parameter-Efficient Reasoning System
- Efficient Reasoning with Hidden Thinking
- Same or Not? Enhancing Visual Perception in Vision-Language Models
- Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
- Offline-Online Curriculum RL for Multimodal Reasoning
- StAR: Segment Anything Reasoner
- Latent Implicit Visual Reasoning
- Visual-Aware CoT: Achieving High-Fidelity Visual Consistency in Unified Models
- CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning
- V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions
- OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
- MMhops-R1: Multimodal Multi-hop Reasoning
- Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
- More Than the Final Answer: Improving Visual Extraction and Logical Consistency in Vision-Language Models
- Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking
- Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
- Grounding Everything in Tokens for Multimodal Large Language Models
- Rethinking Chain-of-Thought Reasoning for Videos
- No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers
- What Happens When: Learning Temporal Orders of Events in Videos
- See, Think, Learn: A Self-Taught Multimodal Reasoner
- ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning
- Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
- Video-CoM: Interactive Video Reasoning via Chain of Manipulations
- OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning
- From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning
- MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents
- DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA
- A Reason-then-Describe Instruction Interpreter for Controllable Video Generation
- LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
- Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs
- RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoning
- SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization
- ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better
- Learning to Think Fast and Slow for Visual Language Models
- When to Think and When to Look: Uncertainty-Guided Lookback
- RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification
- From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
- Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning
- EcoAlign: An Economically Rational Framework for Efficient LVLM Alignment
- Hindsight Distillation Reasoning with Knowledge Encouragement Preference for Knowledge-based Visual Question Answering
- Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation
- Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives
- Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View
- Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale
- QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models
- DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding
- SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning
- VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
- Understanding the Implicit User Intention via Reasoning with Large Language Model for Image Editing
- A Multi-Modal Neuro-Symbolic Approach for Spatial Reasoning-Based Visual Grounding in Robotics
- CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environments
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
- Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start
- RAVEN: Robust Advertisement Video Violation Temporal Grounding via Reinforcement Reasoning
- Explainable Detection of AI-Generated Images with Artifact Localization Using Faster-Than-Lies and Vision-Language Models for Edge Devices
- MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection
- CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning
- 3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models
- Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculation
- Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
- Activating Visual Context and Commonsense Reasoning through Masked Prediction in VLMs
- VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
- Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
- Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
- LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoning
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models
- Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
- HoneyBee: Data Recipes for Vision-Language Reasoners
- Reallocating Attention Across Layers to Reduce Multimodal Hallucination
- Unleashing Perception-Time Scaling to Multimodal Reasoning Models
- ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping
- CIR-CoT: Towards Interpretable Composed Image Retrieval via End-to-End Chain-of-Thought Reasoning
- ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models
- StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answering
- Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI
- RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
- Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights
- Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
- OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models
- From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
- VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Grounding
- Latent Visual Reasoning
- Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
- Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
- VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
- ThinkFake: Reasoning in Multimodal Large Language Models for AI-Generated Image Detection
- Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
- Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning
- Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
- Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generation
- OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
- SAIL-VL2 Technical Report
- MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook
- Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
- Explain Before You Answer: A Survey on Compositional Visual Reasoning
- LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA
- Measuring Epistemic Humility in Multimodal Large Language Models
- Interleaving Reasoning for Better Text-to-Image Generation
- DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training
- A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
- Implicit Reasoning in Large Language Models: A Comprehensive Survey
- VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samples
- Reinforced Visual Perception with Tools
- Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning
- Self-Rewarding Vision-Language Model via Reasoning Decomposition
- HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
- Creative4U: MLLMs-based Advertising Creative Image Selector with Comparative Reasoning
- Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation
- Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models
- Simple o3: Towards Interleaved Vision-Language Reasoning
- Chart-CoCa: Self-Improving Chart Understanding of Vision LMs via Code-Driven Synthesis and Candidate-Conditioned Answering
- Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understanding
- Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
- Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
- Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
- Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision
- Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise
- ViFP: A Framework for Visual False Positive Detection to Enhance Reasoning Reliability in VLMs
- VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
- CoRGI: Verified Chain-of-Thought Reasoning with Post-hoc Visual Grounding
- VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning
- FairReason: Balancing Reasoning and Social Bias in MLLMs
- MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
- Enhancing Spatial Reasoning through Visual and Textual Thinking
- LMM-Det: Make Large Multimodal Models Excel in Object Detection
- Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
- Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
- Agentar-DeepFinance-100K: A Large-Scale Financial Dataset via Systematic Chain-of-Thought Synthesis Optimization
- A Survey of Deep Learning for Geometry Problem Solving
- A Survey on Interpretability in Visual Recognition
- EduFlow: Advancing MLLMs' Problem-Solving Proficiency through Multi-Stage, Multi-Perspective Critique
- Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoning
- The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs
- Learning Deliberately, Acting Intuitively: Unlocking Test-Time Reasoning in Multimodal LLMs
- MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning
- High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning
- Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning
- Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
- ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning
- Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
- Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning
- MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
- Empowering Small VLMs to Think with Dynamic Memorization and Exploration
- Listener-Rewarded Thinking in VLMs for Image Preferences
- APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization
Discussions
- LLaVA-O1: Let Vision Language Models Reason Step-by-Step [hn, 177 points, 32 comments]
- It appears China has a more critical mass of AI researchers and engineers in open source LLM innovation. A collaboration between Chinese University researchers resulted in this. arxiv.org/abs/2411.104 [bsky, 9 points, 2 comments]
- New model (LLaVA-o1) that seems to improve on multimodal reasoning tasks, for VLM enthusiasts 👀. (If you don’t mind the slight anthropomorphism they use in the verbs 😅). #visionlanguagemodels #llms [bsky, 5 points, 0 comments]
- LLaVA-o1: Let Vision Language Models Reason Step-by-Step arxiv.org/pdf/2411.10440 [bsky, 1 points, 0 comments]
- It's a good paper though! link: arxiv.org/abs/2411.10440 [bsky, 1 points, 0 comments]
- LLaVA-o1: Let Vision Language Models Reason Step-by-Step arxiv: arxiv.org/abs/2411.104... No code. Modifies Llama-3.2-vision to use inference time scaling. Calls their technique stage-level beam sear [bsky, 0 points, 1 comments]
- The recent LLaVA-o1 paper is a great glimpse into where these models are going. arxiv.org/abs/2411.10440 [bsky, 0 points, 1 comments]
- arxiv.org/abs/2411.10440 [bsky, 0 points, 0 comments]
- LLaVA-o1 (based on neither LLaVA nor o1, but rather Llama-3.2-Vision) introduces multistage reasoning for vision-language models and claims major gains in visual tasks. It is yet another CoT-tuned mod [bsky, 0 points, 0 comments]
- arxiv.org/abs/2411.10440 [bsky, 0 points, 0 comments]
- LLaVA-O1: Let Vision Language Models Reason Step-by-Step [bsky, 0 points, 0 comments]
- LLaVA-O1: Let Vision Language Models Reason Step-by-Step https://arxiv.org/abs/2411.10440 https://news.ycombinator.com/item?id=42171043 [bsky, 0 points, 0 comments]
Related