LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming

2025/05/21 by Sicheol Sung, Sung, Sicheol, Aditi Aditi +6 · 3 citations
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Logic, Reasoning, and Knowledge #Semantic Web and Ontologies #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2505.15039

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Automated Test Case Generation (ATCG) is crucial for evaluating software reliability, particularly in competitive programming where robust algorithm assessments depend on diverse and accurate test cases. However, existing ATCG methods often fail to meet complex specifications or generate effective corner cases, limiting their utility. In this work, we introduce Context-Free Grammars with Counters (CCFGs), a formalism that captures both syntactic and semantic structures in input specifications. Using a fine-tuned CodeT5 model, we translate natural language input specifications into CCFGs, enabling the systematic generation of high-quality test cases. Experiments on the CodeContests dataset demonstrate that CCFG-based test cases outperform baseline methods in identifying incorrect algorithms, achieving significant gains in validity and effectiveness. Our approach provides a scalable and reliable grammar-driven framework for enhancing automated competitive programming evaluations.

Citations

• TESTEVAL: Benchmarking Large Language Models for Test Case Generation
• Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing
• Code Llama: Open Foundation Models for Code
• Fuzz4All: Universal Fuzzing with Large Language Models
• Grammar Prompting for Domain-Specific Language Generation with Large Language Models
• No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
• Predicting Patch Correctness Based on the Similarity of Failing Test Cases
• Evaluating Large Language Models Trained on Code
• CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Context-Free Commutative Grammars with Integer Counters and Resets
• An orchestrated survey of methodologies for automated software test case generation

Cited by

• AutoCode: LLMs as Problem Setters for Competitive Programming
• SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs
• Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

Related

• Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
• SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs
• CodeContests+: High-Quality Test Case Generation for Competitive Programming
• Codehacks: A Dataset of Adversarial Tests for Competitive Programming Problems Obtained from Codeforces
• Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models