TESTEVAL: Benchmarking Large Language Models for Test Case Generation

2024/06/06 by Wenhan Wang, Chenyuan Yang, Wang, Wenhan +15 · 9 citations
Computer Science · #FOS: Computer and information sciences #Software Engineering (cs.SE) #Software Engineering Research #Software System Performance and Reliability #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2406.04531

openalex publication_date 2024/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Testing plays a crucial role in the software development cycle, enabling the detection of bugs, vulnerabilities, and other undesirable behaviors. To perform software testing, testers need to write code snippets that execute the program under test. Recently, researchers have recognized the potential of large language models (LLMs) in software testing. However, there remains a lack of fair comparisons between different LLMs in terms of test case generation capabilities. In this paper, we propose TESTEVAL, a novel benchmark for test case generation with LLMs. We collect 210 Python programs from an online programming platform, LeetCode, and design three different tasks: overall coverage, targeted line/branch coverage, and targeted path coverage. We further evaluate sixteen popular LLMs, including both commercial and open-source ones, on TESTEVAL. We find that generating test cases to cover specific program lines/branches/paths is still challenging for current LLMs, indicating a lack of ability to comprehend program logic and execution paths. We have open-sourced our dataset and benchmark pipelines at https://github.com/LLM4SoftwareTesting/TestEval.

Cited by

• SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
• Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python
• Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis
• Do Large Language Models Respect Contracts? Evaluating and Enforcing Contract-Adherence in Code Generation
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
• Comparative Evaluation of Large Language Models for Test-Skeleton Generation
• Learning to Generate Unit Test via Adversarial Reinforcement Learning
• HyClone: Bridging LLM Understanding and Dynamic Execution for Semantic Code Clone Detection
• Benchmarking LLMs for Unit Test Generation from Real-World Functions

Related

• Large Language Models as Test Case Generators: Performance Evaluation and Enhancement
• TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark
• Advancing Code Coverage: Incorporating Program Analysis with Large Language Models
• The Program Testing Ability of Large Language Models for Code
• TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language Models