Affective-ROPTester: Capability and Bias Analysis of LLMs in Predicting Retinopathy of Prematurity

2025/07/08 by Shuai Zhao, Yulin Zhang, Zhao, Shuai +15 · 3 citations
Medicine · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computational Engineering #FOS: Computer and information sciences #Finance #Infant Development and Preterm Care #Neonatal and fetal brain pathology #Retinopathy of Prematurity Studies #and Science (cs.CE)

paper · pdf · doi:10.48550/arxiv.2507.05816

openalex publication_date 2025/07/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the remarkable progress of large language models (LLMs) across various domains, their capacity to predict retinopathy of prematurity (ROP) risk remains largely unexplored. To address this gap, we introduce a novel Chinese benchmark dataset, termed CROP, comprising 993 admission records annotated with low, medium, and high-risk labels. To systematically examine the predictive capabilities and affective biases of LLMs in ROP risk stratification, we propose Affective-ROPTester, an automated evaluation framework incorporating three prompting strategies: Instruction-based, Chain-of-Thought (CoT), and In-Context Learning (ICL). The Instruction scheme assesses LLMs' intrinsic knowledge and associated biases, whereas the CoT and ICL schemes leverage external medical knowledge to enhance predictive accuracy. Crucially, we integrate emotional elements at the prompt level to investigate how different affective framings influence the model's ability to predict ROP and its bias patterns. Empirical results derived from the CROP dataset yield two principal observations. First, LLMs demonstrate limited efficacy in ROP risk prediction when operating solely on intrinsic knowledge, yet exhibit marked performance gains when augmented with structured external inputs. Second, affective biases are evident in the model outputs, with a consistent inclination toward overestimating medium- and high-risk cases. Third, compared to negative emotions, positive emotional framing contributes to mitigating predictive bias in model outputs. These findings highlight the critical role of affect-sensitive prompt engineering in enhancing diagnostic reliability and emphasize the utility of Affective-ROPTester as a framework for evaluating and mitigating affective bias in clinical language modeling systems.

Citations

• From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEM
• Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
• Towards Robust Evaluation of STEM Education: Leveraging MLLMs in Project-Based Learning
• Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis
• Uni-Retrieval: A Multi-Style Retrieval Framework for STEM's Education
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
• GPT-4o System Card
• Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Survey
• Gemma 2: Improving Open Language Models at a Practical Size
• Time Matters: Examine Temporal Effects on Biomedical Language Models
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contexts
• Yi: Open Foundation Models by 01.AI
• Fine-tuning Large Language Model (LLM) Artificial Intelligence Chatbots in Ophthalmology and LLM-based evaluation using GPT-4
• Mixtral of Experts
• ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences
• Evaluating multiple large language models in pediatric ophthalmology
• A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics
• BloombergGPT: A Large Language Model for Finance
• GPT-4 Technical Report
• Training language models to follow instructions with human feedback
• LoRA: Low-Rank Adaptation of Large Language Models

Cited by

• P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs
• Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systems
• HRIPBench: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

Related

• Efficient Automated Diagnosis of Retinopathy of Prematurity by Customize CNN Models
• REACT-LLM: A Benchmark for Evaluating LLM Integration with Causal Features in Clinical Prognostic Tasks
• Retinopathy of Prematurity: A Global Perspective and Recent Developments
• Novel Fundus Image Preprocessing for Retcam Images to Improve Deep Learning Classification of Retinopathy of Prematurity
• Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening