Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
2024/12/12 by Ruiqi Deng, Maoli Jiang, Xinlu Yu +2 · 2 voices · 31 citations
Computer Science · Medicine · #Online Learning and Analytics #Artificial Intelligence in Healthcare and Education #Intelligent Tutoring Systems and Adaptive Learning
paper · doi:10.1016/j.compedu.2024.105224
Abstract
Chat Generative Pre-Trained Transformer (ChatGPT) has generated excitement and concern in education. While cross-sectional studies have highlighted correlations between ChatGPT use and learning performance, they fall short of establishing causality. This review examines experimental studies on ChatGPT’s impact on student learning to address this gap. A comprehensive search across five databases identified 69 articles published between 2022 and 2024 for analysis. The findings reveal that ChatGPT interventions are predominantly implemented at the university level, cover various subject areas focusing on language education, are integrated into classroom environments as part of regular educational practices, and primarily involve direct student use of ChatGPT. Overall, ChatGPT improves academic performance, affective-motivational states, and higher-order thinking propensities; it reduces mental effort and has no significant effect on self-efficacy. However, methodological limitations, such as the lack of power analysis and concerns regarding post-intervention assessments, warrant cautious interpretation of results. This review presents four propositions from the findings: (1) distinguish between the quality of ChatGPT outputs and the positive effects of interventions on academic performance by shifting from well-defined problems in post-intervention assessments to more complex, project-based assessments that require skill demonstration, adopting proctored assessments, or incorporating metrics such as originality alongside quality; (2) evaluate long-term impacts to determine whether the positive effects on affective-motivational states are sustained or merely owing to novelty effect; (3) prioritise objective measures to complement subjective assessments of higher-order thinking; and (4) use power analysis to determine adequate sample sizes to avoid Type II errors and provide reliable effect size estimates. This review provides valuable insights for researchers, instructors, and policymakers evaluating the effectiveness of generative AI integration in educational practice. • ChatGPT enhances academic performance. • ChatGPT boosts affective-motivational states. • ChatGPT improves higher-order thinking propensities. • ChatGPT reduces mental effort. • ChatGPT does not influence self-efficacy.
Citations
Cited by
- Little Impact of ChatGPT Availability on High School Student Test Score Performance
- The Digital Divide in Generative AI: Evidence from Large Language Model Use in College Admissions Essays
- Effects of LLM use and note-taking on reading comprehension and memory: A randomised experiment in secondary schools
- Experimental evidence of the effects of large language models versus web search on depth of learning
- A systematic review of generative AI in education: Empirical insights from a human– AI interaction perspective
- Looking Beyond the Hype: Understanding the Effects of AI on Learning
- Think First, ChatGPT Later: Guiding Human–AI Collaboration for Learning Gains in Independent Human Creativity
- Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course
- ChatGPT-5 in Secondary Education: A Mixed-Methods Analysis of Student Attitudes, AI Anxiety, and Hallucination-Aware Use
- When Machines Join the Moral Circle: The Persona Effect of Generative AI Agents in Collaborative Reasoning
- ChatGPT in Education: An Effect in Search of a Cause
- Exploring students’ (mis)conceptions about ChatGPT-generated text: A qualitative study
- How to Use Generative AI in Educational Research
- Building AI Companions that Prioritise Learning over Performance
- Autoeficacia docente para el uso de la inteligencia artificial generativa: un estudio en el profesorado universitario andaluz
- Enhancing pre-service teachers' reflective thinking skills through generative AI-assisted digital storytelling creation: A three-dimensional framework analysis
- The seven roles of generative AI: Potential & pitfalls in combatting misinformation
- Investigating Students' Preferences for AI Roles in Mathematical Modelling: Evidence from a Randomized Controlled Trial
- Small Language Models for Curriculum-based Guidance
- A Meta-Analysis of LLM Effects on Students across Qualification, Socialisation, and Subjectification
- Generative AI in Self-Directed Learning: a thematic scoping review
- Open-Source Large Language Models in Education: A Narrative Review of Evidence, Pedagogical Roles, and Learning Outcomes
- Motivational Factors Influencing Ethiopian Student Teachers’ Self-Efficacy in Adopting AI in Education
- Mind Over Model? Students' Evaluation and Use of ChatGPT ‐Generated Versus Human‐Generated Texts
- RETRACTED ARTICLE: The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis
- Designing and Evaluating AI Margin Notes in Document Reader Software
- Aligning LLMs for the Classroom with Knowledge-Based Retrieval -- A Comparative RAG Study
- Meta-analysis on the influence of AI agents on K-12 student cognitive performance
- Structured AI Demonstrations and Student LLM Use in Engineering Mechanics: Study Design and Preliminary Results
- Structured Prompts, Better Outcomes? Exploring the Effects of a Structured Interface with ChatGPT in a Graduate Robotics Course
- The Revolution Has Arrived: What the Current State of Large Language Models in Education Implies for the Future
Discussions
Related