InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning

2024/09/19 by Xiaotian Han, Han, Xiaotian, Yiren Jian +19 · 16 citations
Computer Science · Social Sciences · #Computer Vision and Pattern Recognition (cs.CV) #Educational Tools and Methods #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Multimedia (cs.MM) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2409.12568

openalex publication_date 2024/09/19 · openalex created_date 2024/10/25 · openalex updated_date 2026/07/28

Abstract

Pre-training on large-scale, high-quality datasets is crucial for enhancing the reasoning capabilities of Large Language Models (LLMs), especially in specialized domains such as mathematics. Despite the recognized importance, the Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source pre-training dataset specifically designed for mathematical reasoning. To address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of interleaved image-text documents. It comprises 24 million web pages, 85 million associated image URLs, and 40 billion text tokens, all meticulously extracted and filtered from CommonCrawl. We provide a detailed overview of our data collection and processing pipeline. To demonstrate the robustness of InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal settings. Our evaluations on text-only benchmarks show that, despite utilizing only 40 billion tokens, our dataset significantly enhances the performance of our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses 120 billion tokens for the same model size. Nevertheless, with the introduction of our multi-modal math pre-training dataset, our models set a new state-of-the-art among open-source models on multi-modal math benchmarks such as MathVerse and We-Math. We release our data at https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.

Cited by

• Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?
• Mid-Training of Large Language Models: A Survey
• We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning
• A Survey of Deep Learning for Geometry Problem Solving
• Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
• OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling
• SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
• Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation
• Growing Through Experience: Scaling Episodic Grounding in Language Models
• Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start
• Music's Multimodal Complexity in AVQA: Why We Need More than General Multimodal LLMs
• Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
• Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving
• Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning
• MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning
• Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training

Related

• Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models
• MegaMath: Pushing the Limits of Open Math Corpora
• MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts
• CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models
• OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text