AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering

2025/09/26 by Ziqing Wang, Chengsheng Mao, Wang, Ziqing +7
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multiagent Systems (cs.MA) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.02328

openalex publication_date 2025/09/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Medical Multimodal Large Language Models (Med-MLLMs) have shown great promise in medical visual question answering (Med-VQA). However, when deployed in low-resource settings where abundant labeled data are unavailable, existing Med-MLLMs commonly fail due to their medical reasoning capability bottlenecks: (i) the intrinsic reasoning bottleneck that ignores the details from the medical image; (ii) the extrinsic reasoning bottleneck that fails to incorporate specialized medical knowledge. To address those limitations, we propose AMANDA, a training-free agentic framework that performs medical knowledge augmentation via LLM agents. Specifically, our intrinsic medical knowledge augmentation focuses on coarse-to-fine question decomposition for comprehensive diagnosis, while extrinsic medical knowledge augmentation grounds the reasoning process via biomedical knowledge graph retrieval. Extensive experiments across eight Med-VQA benchmarks demonstrate substantial improvements in both zero-shot and few-shot Med-VQA settings. The code is available at https://github.com/REAL-Lab-NU/AMANDA.

Citations

• MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration
• MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
• Biomedical Visual Instruction Tuning with Clinician Preference Alignment
• Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA
• FairCLIP: Harnessing Fairness in Vision-Language Learning
• Benchmarking Retrieval-Augmented Generation for Medicine
• MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
• BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
• Q2ATransformer: Improving Medical VQA via an Answer Querying Decoder
• ViperGPT: Visual Inference via Python Execution for Reasoning
• Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training
• Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge
• SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering
• Generating Radiology Reports via Memory-driven Transformer
• Attention to Lesion: Lesion-Aware Convolutional Neural Network for Retinal Optical Coherence Tomography Image Classification
• Towards Rationality in Language and Multimodal Agents: A Survey
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

Related

• Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge
• Inquire, Interact, and Integrate: A Proactive Agent Collaborative Framework for Zero-Shot Multimodal Medical Reasoning
• STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering
• LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models
• MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering