ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback

2021/07/23 by Mike Wu, Noah Goodman, Noah D. Goodman +6
Computer Science · #Computers and Society (cs.CY) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Online Learning and Analytics #Software Testing and Debugging Techniques #cs.CY #cs.LG

paper · pdf · doi:10.48550/arxiv.2107.14035

9 pages content; 6 pages supplement

openalex publication_date 2021/07/23 · arxiv created 2021/10/04 · arxiv updated 2021/10/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

High-quality computer science education is limited by the difficulty of providing instructor feedback to students at scale. While this feedback could in principle be automated, supervised approaches to predicting the correct feedback are bottlenecked by the intractability of annotating large quantities of student code. In this paper, we instead frame the problem of providing feedback as few-shot classification, where a meta-learner adapts to give feedback to student code on a new programming question from just a few examples annotated by instructors. Because data for meta-training is limited, we propose a number of amendments to the typical few-shot learning framework, including task augmentation to create synthetic tasks, and additional side information to build stronger priors about each task. These additions are combined with a transformer architecture to embed discrete sequences (e.g. code) to a prototypical representation of a feedback class label. On a suite of few-shot natural language processing tasks, we match or outperform state-of-the-art performance. Then, on a collection of student solutions to exam questions from an introductory university course, we show that our approach reaches an average precision of 88% on unseen questions, surpassing the 82% precision of teaching assistants. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university. This is, to the best of our knowledge, the first successful deployment of a machine learning based feedback to open-ended student code.

Citations

• Deep Knowledge Tracing
• Human-level concept learning through probabilistic program induction
• Prototypical Networks for Few-shot Learning
• Few-Shot Learning with Graph Neural Networks
• Recent Advances in Neural Program Synthesis
• Meta-Learning for Semi-Supervised Few-Shot Classification
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Meta-Learning: A Survey
• Automated Essay Scoring based on Two-Stage Learning
• Parameter-Efficient Transfer Learning for NLP
• Assume, Augment and Learn: Unsupervised Few-Shot Meta-Learning via Random Labels and Data Augmentation
• A Convolutional Attention Network for Extreme Summarization of Source Code
• Unsupervised Meta-Learning For Few-Shot Image Classification
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification
• Task Augmentation by Rotating for Meta-Learning
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints
• Self-supervised Knowledge Distillation for Few-shot Learning
• Self-Supervised Prototypical Transfer Learning for Few-Shot Classification
• Generative Grading: Near Human-level Accuracy for Automated Feedback on Richly Structured Problems

Related

• Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference
• You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?
• Data-Driven Feedback Generation for Introductory Programming Exercises
• Giving Feedback on Interactive Student Programs with Meta-Exploration
• Humanizing Automated Programming Feedback: Fine-Tuning Generative Models with Student-Written Feedback