LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
2023/06/01 by Chunyuan Li, Cliff Wong, Li, Chunyuan +15 · 257 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2306.00890
openalex publication_date 2023/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
Cited by
- FETAL-GAUGE: A Benchmark for Assessing Vision-Language Models in Fetal Ultrasound
- PathSelect: Sequential Token Selection for Whole Slide Pathology
- PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis
- Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
- OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis
- Enhancing Pathological VLMs with Cross-scale Reasoning
- Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
- Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts
- MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence
- A Medical Multimodal Diagnostic Framework Integrating Vision-Language Models and Logic Tree Reasoning
- A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding
- MarineEval: Assessing the Marine Intelligence of Vision-Language Models
- A DeepSeek-Powered AI System for Automated Chest Radiograph Interpretation in Clinical Practice
- Anatomy-R1: Enhancing Anatomy Reasoning in Multimodal Large Language Models via Anatomical Similarity Curriculum and Group Diversity Augmentation
- R-GenIMA: Integrating Neuroimaging and Genetics with Interpretable Multimodal AI for Alzheimer's Disease Progression
- SafeMed-R1: Adversarial Reinforcement Learning for Generalizable and Robust Medical Reasoning in Vision-Language Models
- Enhancing Medical Large Vision-Language Models via Alignment Distillation
- Investigating Spatial Attention Bias in Vision-Language Models
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
- RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
- CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency
- Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMs
- Exploration of Augmentation Strategies in Multi-modal Retrieval-Augmented Generation for the Biomedical Domain: A Case Study Evaluating Question Answering in Glycobiology
- Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation
- MRG-R1: Reinforcement Learning for Clinically Aligned Medical Report Generation
- Foundation Models in Biomedical Imaging: Turning Hype into Reality
- Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models
- Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning
- Native Intelligence Emerges from Large-Scale Clinical Practice: A Retinal Foundation Model with Deployment Efficiency
- Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making
- Insight Miner: A Time Series Analysis Dataset for Cross-Domain Alignment with Natural Language
- DentalGPT: Incentivizing Multimodal Complex Reasoning in Dentistry
- Synthetic Vasculature and Pathology Enhance Vision-Language Model Reasoning
- LDP: Parameter-Efficient Fine-Tuning of Multimodal LLM for Medical Report Generation
- Enhancing Radiology Report Generation and Visual Grounding using Reinforcement Learning
- A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
- MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding
- Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning
- MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis
- I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models
- Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoning
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models
- Fairness-Aware Fine-Tuning of Vision-Language Models for Medical Glaucoma Diagnosis
- Beyond N-grams: A Hierarchical Reward Learning Framework for Clinically-Aware Medical Report Generation
- Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning
- Concept-Guided Backdoor Attack on Vision Language Models
- OralGPT-Omni: A Versatile Dental Multimodal Large Language Model
- Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reporting
- Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
- OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning
- UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretation
- From Compound Figures to Composite Understanding: Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmarking and Validation
- SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
- LungNoduleAgent: A Collaborative Multi-Agent System for Precision Diagnosis of Lung Nodules
- Generating Querying Code from Text for Multi-Modal Electronic Health Record
- Large Language Model Aided Birt-Hogg-Dube Syndrome Diagnosis with Multimodal Retrieval-Augmented Generation
- Online-PVLM: Advancing Personalized VLMs with Online Concept Learning
- fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding
- Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generation
- MedSAM3: Delving into Segment Anything with Medical Concepts
- Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
- MedVision: Benchmarking Quantitative Medical Image Analysis
- Can a Second-View Image Be a Language? Geometric and Semantic Cross-Modal Reasoning for X-ray Prohibited Item Detection
- PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning
- Boosting Medical Visual Understanding From Multi-Granular Language Learning
- Fairness in Multi-modal Medical Diagnosis with Demonstration Selection
- SkinGPT-R1: Adapter-Only Dual Distillation for Efficient Dermatology Reasoning
- Learning Skill-Attributes for Transferable Assessment in Video
- MedGEN-Bench: Contextually entangled benchmark for open-ended multimodal medical generation
- Multimodal Large Language Models as Image Classifiers
- HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
- R2Seg: Training-Free OOD Medical Tumor Segmentation via Anatomical Reasoning and Statistical Rejection
- CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training
- MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Images
- Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models
- Prompt Triage: Structured Optimization Enhances Vision-Language Model Performance on Medical Imaging Benchmarks
- CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging
- Reinforcing Trustworthiness in Multimodal Emotional Support Systems
- Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models
- Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives
- Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Masks
- Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
- Fine-Tuning Vision-Language Models for Multimodal Polymer Property Prediction
- SCALE-VLP: Soft-Weighted Contrastive Volumetric Vision-Language Pre-training with Spatial-Knowledge Semantics
- RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning
- Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
- OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
- VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learning
- PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting
- CATCH: A Modular Cross-domain Adaptive Template with Hook
- MedVLSynther: Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs
- Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography
- SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation
- Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge
- Generative AI for Healthcare: Fundamentals, Challenges, and Perspectives
- MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering
- S-Chain: Structured Visual Chain-of-Thought For Medicine
- Mitigating Coordinate Prediction Bias from Positional Encoding Failures
- Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation
- 3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models
- UniMedVL: Unifying Medical Multimodal Understanding And Generation Through Observation-Knowledge-Analysis
- BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
- MedReason-R1: Learning to Reason for CT Diagnosis with Reinforcement Learning and Local Zoom
- Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents
- Fine-Tuning MedGemma for Clinical Captioning to Enhance Multimodal RAG over Malaysia CPGs
- The Impact of Image Resolution on Biomedical Multimodal Large Language Models
- RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology
- Voice EHR: introducing multimodal audio data for health
- OralGPT: A Two-Stage Vision-Language Model for Oral Mucosal Disease Diagnosis and Description
- VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models
- SpineBench: Benchmarking Multimodal LLMs for Spinal Pathology Analysis
- Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
- Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation
- A Survey on Agentic Multimodal Large Language Models
- Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
- Think Twice to See More: Iterative Visual Reasoning in Medical VLMs
- MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
- From Generic to Specialized: A Subspecialty Diagnostic System Powered by Self-Supervised Learning for Cervical Histopathology
- Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localization
- M3Retrieve: Benchmarking Multimodal Retrieval for Medicine
- RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model
- SpineBench: A Clinically Salient, Level-Aware Benchmark Powered by the SpineMed-450k Corpus
- Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights
- Automated Structured Radiology Report Generation with Rich Clinical Context
- Evaluating New AI Cell Foundation Models on Challenging Kidney Pathology Cases Unaddressed by Previous Foundation Models
- Glaucoma Detection and Structured OCT Report Generation via a Fine-tuned Multimodal Large Language Model
- ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment
- SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Models
- A Multimodal LLM Approach for Visual Question Answering on Multiparametric 3D Brain MRI
- LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
- From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
- MedMMV: A Controllable Multimodal Multi-Agent Framework for Reliable and Verifiable Clinical Reasoning
- TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models
- Q-FSRU: Quantum-Augmented Frequency-Spectral For Medical Visual Question Answering
- MedLA: A Logic-Driven Multi-Agent Framework for Complex Medical Reasoning with Large Language Models
- CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding
- DentVLM: A Multimodal Vision-Language Model for Comprehensive Dental Diagnosis and Enhanced Clinical Practice
- Patient-specific Biomolecular Instruction Tuning
- EditGRPO: Reinforcement Learning with Post-Rollout Edits for Clinically Accurate Chest X-Ray Report Generation
- RAU: Reference-based Anatomical Understanding with Vision Language Models
- InfiMed-Foundation: Pioneering Advanced Multimodal Medical Models with Compute-Efficient Pre-Training and Multi-Stage Fine-Tuning
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks
- ArchGPT: Understanding the World's Architectures with Large Multimodal Models
- EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
- Objective-Aligned Direct Answer SFT for Robust Multi-Frame Medical VQA
- LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA
- Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning
- DS@GT ARC at ImageCLEFmedical 2026: Architectural Diversity for Concept Detection and Foundation-Model Scaling for Caption Prediction in Medical Image Analysis
- Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
- Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning
- OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generation
- Advances in Large Language Models for Medicine
- Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration
- RadEval: A framework for radiology text evaluation
- AgriDoctor: A Multimodal Intelligent Assistant for Agriculture
- MCTS-EP: Empowering Embodied Planning with Online Preference Optimization
- EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery
- MedFact-R1: Towards Factual Medical Reasoning via Pseudo-Label Augmentation
- EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence
- Data Leakage in Visual Datasets
- Lightweight Joint Optimization of General-Purpose Vision-Language Models and Retrievers for RAG-Based Medical Diagnosis
- WHU-STree: A Multi-modal Benchmark Dataset for Street Tree Inventory
- UTI-LLM: A Personalized Articulatory-Speech Therapy Assistance System Based on Multimodal Large Language Model
- SitLLM: Large Language Models for Sitting Posture Health Understanding via Pressure Sensor Data
- How to make Medical AI Systems safer? Simulating Vulnerabilities, and Threats in Multimodal Medical RAG System
- Adapting and Evaluating Multimodal Large Language Models for Adolescent Idiopathic Scoliosis Self-Management: A Divide and Conquer Framework
- Towards Understanding Visual Grounding in Visual Language Models
- Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis
- Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
- An Iterative LLM Framework for SIBT utilizing RAG-based Adaptive Weight Optimization
- MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance
- Transparency of medical artificial intelligence systems
- Teaching AI Stepwise Diagnostic Reasoning with Report-Guided Chain-of-Thought Learning
- A Foundation Model for Chest X-ray Interpretation with Grounded Reasoning via Online Reinforcement Learning
- MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting
- PediatricsMQA: a Multi-modal Pediatrics Question Answering Benchmark
- Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning
- Can General-Purpose Omnimodels Compete with Specialists? A Case Study in Medical Image Segmentation
- LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression
- Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety
- Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
- DentalBench: Benchmarking and Advancing LLMs Capability for Bilingual Dentistry Understanding
- PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis
- MedGR2: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learning
- CataractSurg-80K: Knowledge-Driven Benchmarking for Structured Reasoning in Ophthalmic Surgery Planning
- GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity
- Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs
- AT-CXR: Uncertainty-Aware Agentic Triage for Chest X-rays
- Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning
- SurgWound-Bench: A Benchmark for Surgical Wound Diagnosis
- LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
- Glo-VLMs: Leveraging Vision-Language Models for Fine-Grained Diseased Glomerulus Classification
- S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Planner
- Organ-Agents: Virtual Human Physiology Simulator via LLMs
- ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
- CardAIc-Agents: A Multimodal Framework with Hierarchical Adaptation for Cardiac Care Support
- HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks
- ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving
- Adaptive Reinforcement for Open-ended Medical Reasoning via Semantic-Guided Reward Collapse Mitigation
- Exploring Efficiency Frontiers of Thinking Budget in Medical Reasoning: Scaling Laws between Computational Resources and Reasoning Quality
- Q-FSRU: Quantum-Augmented Frequency-Spectral Fusion for Medical Visual Question Answering
- A Dataset for Distilling Knowledge Priors from Literature for Therapeutic Design
- Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Dataset
- PASS: Probabilistic Agentic Supernet Sampling for Interpretable and Adaptive Chest X-Ray Reasoning
- A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generation
- SurgPub-Video: A Comprehensive Surgical Video Dataset for Enhanced Surgical Intelligence in Vision-Language Model
- Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoring
- AMRG: Extend Vision Language Models for Automatic Mammography Report Generation
- MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision
- Grasp-HGN: Grasping the Unexpected
- EndoCogniAgent: Closed-Loop Agentic Reasoning with Self-Consistency Validation for Endoscopic Diagnosis
- Mediator-Guided Multi-Agent Collaboration among Open-Source Models for Medical Decision-Making
- Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning
- MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling
- SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Images
- MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models
- Knowledge to Sight: Reasoning over Visual Attributes via Knowledge Decomposition for Abnormality Grounding
- From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models
- GRIT: Graph-Regularized Logit Refinement for Zero-shot Cell Type Annotation
- PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomography
- MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
- A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering
- Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
- Accurate and Interpretable Postmenstrual Age Prediction via Multimodal Large Language Model
- MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
- What Makes "Good" Distractors for Object Hallucination Evaluation in Large Vision-Language Models?
- ForenX: Towards Explainable AI-Generated Image Detection with Multimodal Large Language Models
- MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
- On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
- Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI
- OW-CLIP: Data-Efficient Visual Supervision for Open-World Object Detection via Human-AI Collaboration
- A Survey of Multimodal Hallucination Evaluation and Detection
- EH-Benchmark Ophthalmic Hallucination Benchmark and Agent-Driven Top-Down Traceable Reasoning Workflow
- Resource Consumption Red-Teaming for Large Vision-Language Models
- Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG
- Efficient Whole Slide Pathology VQA via Token Compression
- Towards accurate differential diagnosis with large language models
- A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model
- Automatic Fine-grained Segmentation-assisted Report Generation
- Analysis of Image-and-Text Uncertainty Propagation in Multimodal Large Language Models with Cardiac MR-Based Applications
- Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMs
- Semantically Informed Salient Regions Guided Radiology Report Generation
- Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models
- RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze
- MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models
- Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models
- MIRA: A Novel Framework for Fusing Modalities in Medical RAG
- CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale
- Barriers in Integrating Medical Visual Question Answering into Radiology Workflows: A Scoping Review and Clinicians' Insights
- Computed Tomography Visual Question Answering with Cross-modal Feature Graphing
- A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities
- Causal-SAM-LLM: Large Language Models as Causal Reasoners for Robust Medical Segmentation
Related