ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning

2025/06/11 by Yu Sun, Xingyu Qian, Sun, Yu +19 · 14 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Multiagent Systems (cs.MA) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.09513

openalex publication_date 2025/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reasoning-based large language models have excelled in mathematics and programming, yet their potential in knowledge-intensive medical question answering remains underexplored and insufficiently validated in clinical contexts. To bridge this gap, we introduce ReasonMed, the largest medical reasoning dataset to date, comprising 370k high-quality examples distilled from 1.75 million initial reasoning paths generated by complementary LLMs and curated through a cost-efficient easy-medium-difficult (EMD) pipeline. ReasonMed is built through a multi-agent generation, verification, and refinement process, in which an Error Refiner improves reasoning paths by correcting error-prone steps identified by a verifier. Using ReasonMed, we investigate effective strategies for training medical reasoning models and find that integrating detailed CoT reasoning with concise answer summaries yields the most robust fine-tuning results. Models trained on ReasonMed set a new benchmark: ReasonMed-7B surpasses the prior best sub-10B models by 4.17% and even exceeds LLaMA3.1-70B on PubMedQA by 4.60%. When scaled to ReasonMed-14B, it remains highly competitive, underscoring consistent scaling potential. The codes and datasets are available at https://github.com/YuSun-Work/ReasonMed.

Citations

• AMAZe: A Multi-Agent Zero-shot Index Advisor for Relational Databases
• Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
• STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework
• DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue
• RAPID: Efficient Retrieval-Augmented Long Text Generation with Writing Planning and Information Discovery
• FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
• Logical Reasoning in Large Language Models: A Survey
• Can Large Language Models Be Query Optimizer for Relational Databases?
• BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation
• s1: Simple test-time scaling
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
• KG4Diagnosis: A Hierarchical Multi-Agent LLM Framework with Knowledge Graph Enhancement for Medical Diagnosis
• A Survey on LLM-as-a-Judge
• Long²RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall
• Adaptive Reasoning and Acting in Medical Language Agents
• O1 Replication Journey: A Strategic Progress Report -- Part 1
• Towards Leveraging Large Language Models for Automated Medical Q&A Evaluation
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs
• CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Control
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• Large Language Models
• Can large language models reason about medical questions?
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
• Measuring Massive Multitask Language Understanding
• PubMedQA: A Dataset for Biomedical Research Question Answering
• AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation

Cited by

• Multilingual Medical Reasoning for Question Answering with Large Language Models
• OralGPT-Omni: A Versatile Dental Multimodal Large Language Model
• OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning
• Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives
• Knowledge-Augmented Long-CoT Generation for Complex Biomolecular Reasoning
• S-Chain: Structured Visual Chain-of-Thought For Medicine
• Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
• InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
• ToolLibGen: Scalable Automatic Tool Creation and Aggregation for LLM Reasoning
• Explore-Execute Chain: Towards an Efficient Structured Reasoning Paradigm
• LegalΔ: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gain
• Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
• Towards Efficient Medical Reasoning with Minimal Fine-Tuning Data
• Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications

Related

• Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications
• Building a Human-Verified Clinical Reasoning Dataset via a Human LLM Hybrid Pipeline for Trustworthy Medical AI
• MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
• MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
• ControlMed: Adding Reasoning Control to Medical Language Model