2024/09/11 by Chu, SeongYeub, Jong‐Woo Kim, Kim, JongWoo +2 · 5 citations
Computer Science · Social Sciences · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #Education Practices and Evaluation #Educational Research and Analysis #FOS: Computer and information sciences
paper · pdf · doi:10.48550/arxiv.2409.07355
openalex publication_date 2024/09/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This study introduces InteractEval, a framework that integrates human expertise and Large Language Models (LLMs) using the Think-Aloud (TA) method to generate attributes for checklist-based text evaluation. By combining human flexibility and reasoning with LLM consistency, InteractEval outperforms traditional non-LLM-based and LLM-based baselines across four distinct dimensions, consisting of Coherence, Fluency, Consistency, and Relevance. The experiment also investigates the effectiveness of the TA method, showing that it promotes divergent thinking in both humans and LLMs, leading to the generation of a wider range of relevant attributes and enhance text evaluation performance. Comparative analysis reveals that humans excel at identifying attributes related to internal quality (Coherence and Fluency), but LLMs perform better at those attributes related to external alignment (Consistency and Relevance). Consequently, leveraging both humans and LLMs together produces the best evaluation outcomes. In other words, this study emphasizes the necessity of effectively combining humans and LLMs in an automated checklist-based text evaluation framework. The code is available at \textbf\urlhttps://github.com/BBeeChu/InteractEval.git.