AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course

2026/02/18 by Xinyi Lu, Kexin Phyllis Ju, Mitchell Dudley +2 · 2 voices
Computer Science · Psychology · Social Sciences · #Control (management) #Course (navigation) #Educational Strategies and Epistemologies #Feedback control #Innovations in Educational Methods #Innovative Teaching Methodologies in Social Sciences #Randomized controlled trial #Randomized experiment #Spotting #cs.AI #cs.HC

paper · pdf · doi:10.1145/3772318.3791121

arxiv published 2026/02/18 · arxiv updated 2026/02/24 · openalex publication_date 2026/04/13 · openalex created_date 2026/04/14 · openalex updated_date 2026/08/06

Abstract

Despite growing interest in using LLMs to generate feedback on students’ writing, little is known about how students respond to AI-mediated versus human-provided feedback. We address this gap through a randomized controlled trial in a large introductory economics course (N=354), where we introduce and deploy FeedbackWriter—a system that generates AI suggestions to teaching assistants (TAs) while they provide feedback on students’ knowledge-intensive essays. TAs have the full capacity to adopt, edit, or dismiss the suggestions. Students were randomly assigned to receive either handwritten feedback from TAs (baseline) or AI-mediated feedback where TAs received suggestions from FeedbackWriter. Students revise their drafts based on the feedback, which is further graded. In total, 1,366 essays were graded using the system. We found that students receiving AI-mediated feedback produced significantly higher-quality revisions, with gains increasing as TAs adopted more AI suggestions. TAs found the AI suggestions useful for spotting gaps and clarifying rubrics.

Citations

• Generative AI without guardrails can harm learning: Evidence from high school mathematics
• LLM-Generated Feedback Supports Learning If Learners Choose to Use It
• Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use
• From First Draft to Final Insight: A Multi-Agent Approach for Feedback Generation
• TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students
• Generative AI vs. instructor vs. peer assessments: a comparison of grading and feedback in higher education
• Assessing Large Language Models for Automated Feedback Generation in Learning Programming Problem Solving
• Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns
• Towards Adaptive Feedback with AI: Comparing the Feedback Quality of LLMs and Teachers on Experimentation Protocols
• Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
• Comparing the quality of human and ChatGPT feedback of students’ writing
• LLM Evaluators Recognize and Favor Their Own Generations
• Can ChatGPT effectively complement teacher assessment of undergraduate students’ academic writing?
• Using Large Language Models to Assess Tutors' Performance in Reacting to Students Making Math Errors
• Fine-tuning ChatGPT for Automatic Scoring
• A large language model-assisted education tool to provide feedback on open-ended responses
• Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources
• MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems
• Learning gain differences between ChatGPT and human tutor generated algebra hints
• Large Language Models are Zero-Shot Reasoners
• Language Models are Few-Shot Learners
• The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research
• The challenges of feedback in higher education
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Manipulating and Measuring Model Interpretability
• The nature of feedback: How peer feedback features affect students’ implementation rate and quality of revisions.
• Fitting Linear Mixed-Effects Models using lme4
• The Knowledge‐Learning‐Instruction Framework: Bridging the Science‐Practice Chasm to Enhance Robust Student Learning
• The Power of Feedback
• The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory.
• The Out-of-the-Loop Performance Problem and Level of Control in Automation
• Formative assessment and the design of instructional systems
• Genre as social action
• Writing as a Mode of Learning
• A Coefficient of Agreement for Nominal Scales
• Fitting Linear Mixed-Effects Models Using lme4

Discussions

• When we assume students will either largely use AI for all written work or we believe we should never allow or encourage them to do so, we miss out on human-AI combined writing support studies like th [bsky, 1 points, 0 comments]
• "TAs found the AI suggestions useful for spotting gaps and clarifying rubrics" www.arxiv.org/abs/2602.16820 [bsky, 0 points, 0 comments]

Related

• Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use
• AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher Education
• AI feedback literacy in higher education: understanding, measuring, and predicting student feedback uptake
• Summarise, elaborate, try again: exploring the effect of feedback literacy on AI-enhanced essay writing
• Student Evaluation of Repeated AI Feedback Across a Semester of Writing