2025/09/18 by Kadin Matotek, Matotek, Kadin, Heather Cassel +5
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #F.2.2 #FOS: Computer and information sciences #I.2.2 #I.2.7 #Machine Learning (cs.LG) #Machine Learning and Data Classification #Metaheuristic Optimization Algorithms Research #Programming Languages (cs.PL) #Software Engineering (cs.SE)
paper · pdf · doi:10.48550/arxiv.2509.15283
openalex publication_date 2025/09/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This study examines the performance of today's open-source, locally hosted large-language models (LLMs) in handling complex competitive programming tasks with extended problem descriptions and contexts. Building on the original Framework for AI-driven Code Generation Evaluation (FACE), the authors retrofit the pipeline to work entirely offline through the Ollama runtime, collapsing FACE's sprawling per-problem directory tree into a handful of consolidated JSON files, and adding robust checkpointing so multi-day runs can resume after failures. The enhanced framework generates, submits, and records solutions for the full Kattis corpus of 3,589 problems across eight code-oriented models ranging from 6.7-9 billion parameters. The submission results show that the overall pass@1 accuracy is modest for the local models, with the best models performing at approximately half the acceptance rate of the proprietary models, Gemini 1.5 and ChatGPT-4. These findings expose a persistent gap between private, cost-controlled LLM deployments and state-of-the-art proprietary services, yet also highlight the rapid progress of open models and the practical benefits of an evaluation workflow that organizations can replicate on in-house hardware.