Plain language adaptations of biomedical text using LLMs: Comparision of evaluation metrics

2025/12/18 by Primož Kocbek, Kocbek, Primoz, Leon Kopitar +3
Computer Science · Health Professions · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Health Literacy and Information Accessibility #I.2.7 #Text Readability and Simplification

paper · doi:10.48550/arxiv.2512.16530

openalex publication_date 2025/12/18 · openalex created_date 2025/12/21 · openalex updated_date 2026/07/28

Abstract

This study investigated the application of Large Language Models (LLMs) for simplifying biomedical texts to enhance health literacy. Using a public dataset, which included plain language adaptations of biomedical abstracts, we developed and evaluated several approaches, specifically a baseline approach using a prompt template, a two AI agent approach, and a fine-tuning approach. We selected OpenAI gpt-4o and gpt-4o mini models as baselines for further research. We evaluated our approaches with quantitative metrics, such as Flesch-Kincaid grade level, SMOG Index, SARI, and BERTScore, G-Eval, as well as with qualitative metric, more precisely 5-point Likert scales for simplicity, accuracy, completeness, brevity. Results showed a superior performance of gpt-4o-mini and an underperformance of FT approaches. G-Eval, a LLM based quantitative metric, showed promising results, ranking the approaches similarly as the qualitative metric.

Citations

• Contamination Report for Multilingual Benchmarks
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• A Dataset for Plain Language Adaptation of Biomedical Abstracts
• Language Models are Few-Shot Learners

Cited by

• An Architectural Advantage of The Instruction-Tuned LLM in Containing The Readability-Accuracy Tension in Text Simplification

Related

• Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstracts
• Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data
• Adapting Biomedical Abstracts into Plain language using Large Language Models
• A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
• UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text