Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs
2025/11/24 by Meng‐Yao Lu, Ran Xu, Lu, Meng +31 · 1 citation
Computer Science · #Multimodal Machine Learning Applications #Explainable Artificial Intelligence (XAI) #Domain Adaptation and Few-Shot Learning
paper · pdf · doi:10.48550/arxiv.2511.19773
Abstract
While recent vision-language models (VLMs) demonstrate strong image understanding, their ability to "think with images", i.e., to reason through multi-step visual interactions, remains limited. We introduce VISTA-Gym, a scalable training environment for incentivizing tool-integrated visual reasoning capabilities in VLMs. VISTA-Gym unifies diverse real-world multimodal reasoning tasks (7 tasks from 13 datasets in total) with a standardized interface for visual tools (e.g., grounding, parsing), executable interaction loops, verifiable feedback signals, and efficient trajectory logging, enabling visual agentic reinforcement learning at scale. While recent VLMs exhibit strong text-only reasoning, both proprietary and open-source models still struggle with tool selection, invocation, and coordination. With VISTA-Gym, we train VISTA-R1 to interleave tool-use with agentic reasoning via multi-turn trajectory sampling and end-to-end reinforcement learning. Extensive experiments across 11 public reasoning-intensive VQA benchmarks show that VISTA-R1-8B outperforms state-of-the-art baselines with similar sizes by 9.51%-18.72%, demonstrating VISTA-Gym as an effective training ground to unlock the tool-integrated reasoning capabilities for VLMs.
Citations
- TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
- Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
- VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding
- Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
- DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
- ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Coding
- TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics
- MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline
- ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models
- GEM: A Gym for Agentic LLMs
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
- LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym
- AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
- VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
- Self-Rewarding Vision-Language Model via Reasoning Decomposition
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers
- MMSearch-R1: Incentivizing LMMs to Search
- MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
- REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
- VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
- Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving
- Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
- GRIT: Teaching MLLMs to Think with Images
- G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
- OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
- MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
- SWE-smith: Scaling Data for Software Engineering Agents
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
- Perception-R1: Pioneering Perception Policy with Reinforcement Learning
- VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
- OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization
- R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization
- Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
- Visual-RFT: Visual Reinforcement Fine-Tuning
- MLGym: A New Framework and Benchmark for Advancing AI Research Agents
- Qwen2.5-VL Technical Report
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
- LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
- Training Software Engineering Agents and Verifiers with SWE-Gym
- Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration
- The BrowserGym Ecosystem for Web Agent Research
- ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration
- LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
- ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding
- MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models
- Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models
- CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning
- BioXP-0.5B: Explainable Medical-AI via RL-GRPO
- G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Micrograph segmentations for DDEVD
- Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
- UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
- MapQA: A Dataset for Question Answering on Choropleth Maps
- ReAct: Synergizing Reasoning and Acting in Language Models
- Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning
- A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- ScienceWorld: Is your Agent Smarter than a 5th Grader?
- BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
- GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning
- Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
- InfographicVQA
- DocVQA: A Dataset for VQA on Document Images
- PlotQA: Reasoning over Scientific Plots
- Proximal Policy Optimization Algorithms
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary\n Visual Reasoning
- A Diagram Is Worth A Dozen Images
- MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science
- VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning
- Supervising the search process produces reliable and generalizable information-seeking agents
Cited by
Related