Clarification as Supervision: Reinforcement Learning for Vision-Language Interfaces

2025/09/30 by John Gkountouras, Ivan Titov, Gkountouras, John +1
Computer Science · Engineering · #68T05 (Primary) 68T45 #68T50 (Secondary) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #I.2.10 #I.2.6 #I.2.7 #Machine Learning (cs.LG) #Robotics and Automated Systems #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2509.26594

openalex publication_date 2025/09/30 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Recent text-only models demonstrate remarkable mathematical reasoning capabilities. Extending these to visual domains requires vision-language models to translate images into text descriptions. However, current models, trained to produce captions for human readers, often omit the precise details that reasoning systems require. This creates an interface mismatch: reasoners often fail not due to reasoning limitations but because they lack access to critical visual information. We propose Adaptive-Clarification Reinforcement Learning (AC-RL), which teaches vision models what information reasoners need through interaction. Our key insight is that clarification requests during training reveal information gaps; by penalizing success that requires clarification, we create pressure for comprehensive initial captions that enable the reasoner to solve the problem in a single pass. AC-RL improves average accuracy by 4.4 points over pretrained baselines across seven visual mathematical reasoning benchmarks, and analysis shows it would cut clarification requests by up to 39% if those were allowed. By treating clarification as a form of implicit supervision, AC-RL demonstrates that vision-language interfaces can be effectively learned through interaction alone, without requiring explicit annotations.

Citations

• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
• Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving
• Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
• R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search
• Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
• DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models
• Beyond Captioning: Task-Specific Prompting for Improved VLM Performance in Mathematical Reasoning
• VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
• LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
• We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
• Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models
• VLRM: Vision-Language Models act as Reward Models for Image Captioning
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
• Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• Large Language Models are Visual Reasoning Coordinators
• ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
• Zero-shot Visual Question Answering with Language Model Feedback
• GPT-4 Technical Report
• Visual Programming: Compositional visual reasoning without training
• OpenAI o1 System Card

Related

• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models
• Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models
• Self-Rewarding Vision-Language Model via Reasoning Decomposition
• See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning