LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations

2025/11/18 by Tatiane Ornelas, Allysson Allex Araújo, Ornelas, Tatiane +8
Medicine · Social Sciences · #Artificial Intelligence in Healthcare and Education #Computational and Text Analysis Methods #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2511.14528

openalex publication_date 2025/11/18 · openalex created_date 2025/11/20 · openalex updated_date 2026/07/28

Abstract

[Context] Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE), yet the methodological implications of this usage remain underexplored. Their integration into interpretive processes such as thematic analysis raises fundamental questions about rigor, transparency, and researcher agency. [Objective] This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis. [Method] A reflective workshop with 25 ISERN researchers guided participants through structured discussions of LLM-assisted open coding, theme generation, and theme reviewing, using color-coded canvases to document perceived opportunities, limitations, and recommendations. [Results] Participants recognized potential efficiency and scalability gains, but highlighted risks related to bias, contextual loss, reproducibility, and the rapid evolution of LLMs. They also emphasized the need for prompting literacy and continuous human oversight. [Conclusion] Findings portray LLMs as tools that can support, but not substitute, interpretive analysis. The study contributes to ongoing community reflections on how LLMs can responsibly enhance qualitative research in SE.

Citations

• Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research
• Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering
• Towards Evaluation Guidelines for Empirical Studies involving LLMs
• Large Language Models in Qualitative Research: Uses, Tensions, and Intentions
• Performing an Inductive Thematic Analysis of Semi-Structured Interviews With a Large Language Model: An Exploration and Provocation on the Limits of the Approach
• CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models

Related

• Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Research
• Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering
• Guidelines for Empirical Studies in Software Engineering involving Large Language Models
• Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research
• LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis