Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models

2025/09/18 by Stephanie G. Yi, Yi, Seungjun, Nguyen, Joakim +16 · 1 citation
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.14597

openalex publication_date 2025/09/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This position paper examines how large language models (LLMs) can support thematic analysis of unstructured clinical transcripts, a widely used but resource-intensive method for uncovering patterns in patient and provider narratives. We conducted a systematic review of recent studies applying LLMs to thematic analysis, complemented by an interview with a practicing clinician. Our findings reveal that current approaches remain fragmented across multiple dimensions including types of thematic analysis, datasets, prompting strategies and models used, most notably in evaluation. Existing evaluation methods vary widely (from qualitative expert review to automatic similarity metrics), hindering progress and preventing meaningful benchmarking across studies. We argue that establishing standardized evaluation practices is critical for advancing the field. To this end, we propose an evaluation framework centered on three dimensions: validity, reliability, and interpretability.

Citations

• Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis
• Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case
• Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning
• Cost-of-Pass: An Economic Framework for Evaluating Language Models
• BEYONDWORDS is All You Need: Agentic Generative AI based Social Media Themes Extractor
• Potential and Perils of Large Language Models as Judges of Unstructured Textual Data
• Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks
• How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment
• Program Synthesis with Large Language Models
• Language Models are Few-Shot Learners
• BERTScore: Evaluating Text Generation with BERT
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Reliability in Content Analysis.
• A Coefficient of Agreement for Nominal Scales

Cited by

• SFT-TA: Supervised Fine-Tuned Agents in Multi-Agent LLMs for Automated Inductive Thematic Analysis

Related

• Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages
• Large Language Models in Qualitative Analysis: Comparing Traditional and Researcher-Interpreted Approaches
• Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review
• Human vs. LLM-Based Thematic Analysis for Digital Mental Health Research: Proof-of-Concept Comparative Study
• Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning