Vision-Language Models Can Self-Improve Reasoning via Reflection

2024/10/30 by Kanzhi Cheng, Yantao Li, Cheng, Kanzhi +9 · 14 citations
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2411.00855

openalex publication_date 2024/10/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, R3V, which iteratively enhances the model's Vision-language Reasoning by Reflecting on CoT Rationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that R3V consistently improves multimodal LLM reasoning, achieving a relative improvement of 23 to 60 percent over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation.

Cited by

• Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking
• Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models
• See, Think, Learn: A Self-Taught Multimodal Reasoner
• RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification
• From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
• OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows
• A Survey on Agentic Multimodal Large Language Models
• Self-Improvement in Multimodal Large Language Models: A Survey
• Can GRPO Boost Complex Multimodal Table Understanding?
• DF-LLaVA: Unlocking MLLMs for Synthetic Image Detection via Knowledge Injection and Conflict-Driven Self-Reflection
• Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
• Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models
• Long-Horizon Visual Imitation Learning via Plan and Code Reflection
• Large Language Models Facilitate Vision Reflection in Image Classification

Related

• Improve Vision Language Model Chain-of-thought Reasoning
• See, Think, Learn: A Self-Taught Multimodal Reasoner
• SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Traces
• EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoning
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models