OrthoInsight: Rib Fracture Diagnosis and Report Generation Based on Multi-Modal Large Models

2025/07/18 by Ning Wu, Wang, Jinzhi, Zhang, Jiangbo +9 · 1 citation
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Multimodal Machine Learning Applications #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.13993

openalex publication_date 2025/07/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

The growing volume of medical imaging data has increased the need for automated diagnostic tools, especially for musculoskeletal injuries like rib fractures, commonly detected via CT scans. Manual interpretation is time-consuming and error-prone. We propose OrthoInsight, a multi-modal deep learning framework for rib fracture diagnosis and report generation. It integrates a YOLOv9 model for fracture detection, a medical knowledge graph for retrieving clinical context, and a fine-tuned LLaVA language model for generating diagnostic reports. OrthoInsight combines visual features from CT images with expert textual data to deliver clinically useful outputs. Evaluated on 28,675 annotated CT images and expert reports, it achieves high performance across Diagnostic Accuracy, Content Completeness, Logical Coherence, and Clinical Guidance Value, with an average score of 4.28, outperforming models like GPT-4 and Claude-3. This study demonstrates the potential of multi-modal learning in transforming medical image analysis and providing effective support for radiologists.

Citations

• KG4Diagnosis: A Hierarchical Multi-Agent LLM Framework with Knowledge Graph Enhancement for Medical Diagnosis
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Advancing High Resolution Vision-Language Models in Biomedicine
• Power Optimization for Integrated Active and Passive Sensing in DFRC Systems
• Improved Baselines with Visual Instruction Tuning
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• GPT-4 Technical Report
• Large Language Models Encode Clinical Knowledge
• Large language models encode clinical knowledge
• CogView: Mastering Text-to-Image Generation via Transformers
• GLM: General Language Model Pretraining with Autoregressive Blank Infilling
• You Only Look Once: Unified, Real-Time Object Detection
• The Measurement of Observer Agreement for Categorical Data
• Gradient-based learning applied to document recognition

Cited by

• Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents

Related

• Deep Rib Fracture Instance Segmentation and Classification from CT on the RibFrac Challenge
• Deep Omni-supervised Learning for Rib Fracture Detection from Chest Radiology Images
• ORF-Net: Deep Omni-supervised Rib Fracture Detection from Chest CT Scans
• Knowledge Distillation with Adaptive Asymmetric Label Sharpening for Semi-supervised Fracture Detection in Chest X-rays
• Fine-Grained Rib Fracture Diagnosis with Hyperbolic Embeddings: A Detailed Annotation Framework and Multi-Label Classification Model