MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
2025/07/29 by Gao, Tianhong, Fu, Yannian, Wu, Weiqun +3
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2507.21924
Abstract
Large Language Models (LLMs), enhanced through agent tuning, have demonstrated remarkable capabilities in Chain-of-Thought (CoT) and tool utilization, significantly surpassing the performance of standalone models. However, the multimodal domain still lacks a large-scale, high-quality agent tuning dataset to unlock the full potential of multimodal large language models. To bridge this gap, we introduce MMAT-1M, the first million-scale multimodal agent tuning dataset designed to support CoT, reflection, and dynamic tool usage. Our dataset is constructed through a novel four-stage data engine: 1) We first curate publicly available multimodal datasets containing question-answer pairs; 2) Then, leveraging GPT-4o, we generate rationales for the original question-answer pairs and dynamically integrate API calls and Retrieval Augmented Generation (RAG) information through a multi-turn paradigm; 3) Furthermore, we refine the rationales through reflection to ensure logical consistency and accuracy, creating a multi-turn dialogue dataset with both Rationale and Reflection (RR); 4) Finally, to enhance efficiency, we optionally compress multi-turn dialogues into a One-turn Rationale and Reflection (ORR) format. By fine-tuning open-source multimodal models on the MMAT-1M, we observe significant performance gains. For instance, the InternVL2.5-8B-RR model achieves an average improvement of 2.7% across eight public benchmarks and 8.8% on the RAG benchmark Dyn-VQA, demonstrating the dataset's effectiveness in enhancing multimodal reasoning and tool-based capabilities. The dataset is publicly available at https://github.com/VIS-MPU-Agent/MMAT-1M.
Citations
- Qwen2.5-VL Technical Report
- Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
- GPT-4o System Card
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning
- MiniCPM-V: A GPT-4V Level MLLM on Your Phone
- VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
- Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
- M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
- What matters when building vision-language models?
- How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
- Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models
- LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
- Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- Compositional Chain-of-Thought Prompting for Large Multimodal Models
- ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
- Agent Lumos: Unified and Modular Training for Open-Source Language Agents
- LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
- HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- AgentTuning: Enabling Generalized Agent Abilities for LLMs
- FireAct: Toward Language Agent Fine-tuning
- Improved Baselines with Visual Instruction Tuning
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- VisText: A Benchmark for Semantically Rich Chart Captioning
- AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
- GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
- Document Understanding Dataset and Evaluation (DUDE)
- Visual Instruction Tuning
- Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
- Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning
- Visual Spatial Reasoning
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
- Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
- Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
- InfographicVQA
- PP-OCR: A Practical Ultra Lightweight OCR System
- DocVQA: A Dataset for VQA on Document Images
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- TextCaps: a Dataset for Image Captioning with Reading Comprehension
- Scene Text Visual Question Answering
- Towards VQA Models That Can Read
- RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- A Diagram Is Worth A Dozen Images
- Visual7W: Grounded Question Answering in Images
- Compositional Semantic Parsing on Semi-Structured Tables
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Related