Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools

2025/07/07 by Lorenzo Lee Solano, Charles Koutcheme, Solano, Lorenzo Lee +7 · 3 citations
Computer Science · Materials Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Machine Learning in Materials Science #Software Engineering (cs.SE) #Teaching and Learning Programming

paper · pdf · doi:10.48550/arxiv.2507.05305

openalex publication_date 2025/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Frontier Large language models (LLMs) like ChatGPT and Gemini can decipher cryptic compiler errors for novice programmers, but their computational scale, cost, and tendency to over-assist make them problematic for widespread pedagogical adoption. This work demonstrates that smaller, specialised language models, enhanced via Supervised Fine-Tuning (SFT), present a more viable alternative for educational tools. We utilise a new dataset of 40,000 C compiler error explanations, derived from real introductory programming (CS1/2) student-generated programming errors, which we used to fine-tune three open-source models: Qwen3-4B, Llama-3.1-8B, and Qwen3-32B. We performed a dual evaluation, combining expert human reviews with a large-scale automated analysis of 8,000 responses using a validated LLM-as-judge ensemble. Our results show that SFT significantly boosts the pedagogical quality of smaller models, achieving performance comparable to much larger models. We analyse the trade-offs between model size and quality, confirming that fine-tuning compact, efficient models on high-quality, domain-specific data is a potent strategy for creating specialised models to drive educational tools. We provide a replicable methodology to foster broader access to generative AI capabilities in educational contexts.

Citations

• Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education
• Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness
• Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools
• The Llama 3 Herd of Models
• Evaluating Language Models for Generating and Judging Programming Feedback
• Evaluating Contextually Personalized Programming Exercises Created with Generative AI
• Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation
• Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge
• Exploring the Potential of Large Language Models to Generate Formative Programming Feedback
• Large Language Models (GPT) for automating feedback on programming assignments
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• QLoRA: Efficient Finetuning of Quantized LLMs
• Computing inter‐rater reliability and its variance in the presence of high agreement

Cited by

• Performance Trade-offs of Optimizing Small Language Models for E-Commerce
• Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study
• Perspectives and potential issues in using artificial intelligence for computer science education

Related

• Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education
• Towards Pedagogical LLMs with Supervised Fine Tuning for Computing Education
• Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation
• Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge
• Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code