Dual-Uncertainty Guided Policy Learning for Multimodal Reasoning

2025/10/01 by Rui Liu, Dian Yu, Liu, Rui +19 · 2 citations
Computer Science · #AI-based Problem Solving and Planning #Ambiguity #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Focus (optics) #Machine Learning (cs.LG) #Multimodal learning #Perception #Policy learning #Prioritization #Reinforcement learning #Semantic Web and Ontologies #Speech and dialogue systems #Verifiable secret sharing

paper · pdf · doi:10.48550/arxiv.2510.01444

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Reinforcement learning with verifiable rewards (RLVR) has advanced reasoning capabilities in multimodal large language models. However, existing methods typically treat visual inputs as deterministic, overlooking the perceptual ambiguity inherent to the visual modality. Consequently, they fail to distinguish whether a model's uncertainty stems from complex reasoning or ambiguous perception, preventing the targeted allocation of exploration or learning signals. To address this gap, we introduce DUPL, a dual-uncertainty guided policy learning approach for multimodal RLVR that quantifies and leverages both perceptual uncertainty (via symmetric KL divergence) and output uncertainty (via policy entropy) to guide policy updates. By establishing an uncertainty-driven feedback loop and employing a dynamic branch prioritization mechanism, DUPL recalibrates the policy advantage to focus learning on states with high perceptual or decisional ambiguity, enabling effective targeted exploration beyond passive data augmentation. Evaluated on diverse multimodal reasoning benchmarks spanning mathematical and general domains, DUPL achieves solid gains. It improves Qwen2.5-VL accuracy by up to 12.3% (3B) and 7.9% (7B), and Qwen3-VL-Instruct by up to 10.7% (4B) and 12.4% (8B), consistently outperforming GRPO, while seamlessly generalizing to alternative algorithms (DAPO, +6.5% avg) and architectures (LLaVA-OneVision-1.5, +4.7% avg). These results demonstrate that DUPL is an effective and generalizable approach for multimodal RLVR.

Citations

• Stable and Efficient Single-Rollout RL for Multimodal Reasoning
• Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• Outcome-based Exploration for LLM Reasoning
• Jointly Reinforcing Diversity and Quality in Language Model Generations
• Self-Rewarding Vision-Language Model via Reasoning Decomposition
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
• Perception-Aware Policy Optimization for Multimodal Reasoning
• MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
• Reasoning with Exploration: An Entropy Perspective
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
• Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration
• Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems
• Improving RL Exploration for LLM Reasoning through Retrospective Replay
• SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
• Entropy-based Exploration Conduction for Multi-step Reasoning
• LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
• LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
• We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
• ReFT: Reasoning with Reinforced Fine-Tuning
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• Reinforcement Learning with Augmented Data
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models

Cited by

• Stable and Efficient Single-Rollout RL for Multimodal Reasoning
• VisPlay: Self-Evolving Vision-Language Models from Images

Related

• Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning
• First Return, Entropy-Eliciting Explore
• Perception-Aware Policy Optimization for Multimodal Reasoning
• When to Think and When to Look: Uncertainty-Guided Lookback
• Spotlight on Token Perception for Multimodal Reinforcement Learning