OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning

2025/11/28 by Ossowski, Timothy, Zhang, Sheng, Liu, Qianchu +5 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning in Healthcare #Multimodal Machine Learning Applications #Topic Modeling

paper · doi:10.48550/arxiv.2511.23269

openalex publication_date 2025/11/28 · openalex created_date 2025/12/03 · openalex updated_date 2026/07/28

Abstract

High-quality and carefully curated data is a cornerstone of training medical large language models, as it directly impacts both generalization and robustness to unseen clinical tasks. We investigate strategies for training and data curation to develop a robust multimodal reasoning model in the medical domain. Our work focuses on supervised fine-tuning (SFT) and explores data recipes that leverage structured reasoning traces. Using our proposed data recipe, we scale experiments to a dataset of over 8 million examples and 6.8 billion response tokens, achieving state-of-the-art performance among open-source models across diverse out-of-distribution medical benchmark tasks. Our results further indicate that curating a high-quality, diverse training dataset with varying structured reasoning trace lengths enables the fine-tuned model to self-calibrate its reasoning trajectory lengths based on the downstream task, without explicit supervision. We present key insights, describe the data curation strategy, and outline next steps toward developing robust medical vision-language reasoning system.

Citations

• FineVision: Open Data Is All You Need
• HoneyBee: Data Recipes for Vision-Language Reasoners
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
• Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
• MedGemma 1.5 Technical Report
• GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search
• Spurious Rewards: Rethinking Training Signals in RLVR
• ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning
• Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
• ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understanding
• OpenThoughts: Data Recipes for Reasoning Models
• QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
• Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner
• X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains
• Learning and Generalization with Mixture Data
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Skywork R1V: Pioneering Multimodal Reasoning with Chain-of-Thought
• MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
• R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
• CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
• Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learning
• MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning
• Qwen2.5-VL Technical Report
• s1: Simple test-time scaling
• MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale
• LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
• GPT-4o System Card
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
• RegMix: Data Mixture as Regression for Language Model Pre-training
• HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale
• MAIRA-2: Grounded Radiology Report Generation
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• OLIVE: Object Level In-Context Visual Embeddings
• EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• A Foundational Multimodal Vision Language AI Assistant for Human Pathology
• Hierarchical Pretraining on Multimodal Electronic Health Records
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• PRAG: Paninian Retrieval-Augmented Generation for Safety-Critical Medical Question Answering
• Measuring Massive Multitask Language Understanding
• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• HEAD-QA: A Healthcare Dataset for Complex Reasoning
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning

Cited by

• PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis

Related

• MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
• Elicit and Enhance: Advancing Multimodal Reasoning in Medical Scenarios
• InfiMed: Low-Resource Medical MLLMs with Advancing Understanding and Reasoning
• FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Training
• Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications