Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
2026/02/08 by Zhi Chen, Zhensu Sun, Yuling Shi +4 · 1 voice
Computer Science · #cs.SE #cs.AI
paper · pdf
arxiv published 2026/02/08 · arxiv updated 2026/04/09
Abstract
Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.
Citations
- Towards a Science of Scaling Agent Systems
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
- Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories
- An Empirical Study on Failures in Automated Issue Solving
- Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
- Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
- EXPEREPAIR: Dual-Memory Enhanced LLM-based Repository-Level Program Repair
- Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
- Requirements-Based Test Generation: A Comprehensive Survey
- Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios
- CodeMonkeys: Scaling Test-Time Compute for Software Engineering
- Mutation-Guided LLM-based Test Generation at Meta
- Self-Evolving Multi-Agent Collaboration Networks for Software Development
- Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
- Promise and Peril of Collaborative Code Generation Models: Balancing Effectiveness and Memorization
- MarsCode Agent: AI-native Automated Bug Fixing
- A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites
- SpecRover: Code Intent Extraction via LLMs
- Agentless: Demystifying LLM-based Software Engineering Agents
- On the Evaluation of Large Language Models in Unit Test Generation
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
- Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM
- Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools
- Persistent memory for AI coding agents: a pre-registered SWE-bench Verified benchmark
- LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
- ChatDev: Communicative Agents for Software Development
- Software Testing with Large Language Models: Survey, Landscape, and Vision
- Software Testing With Large Language Models: Survey, Landscape, and Vision
- An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
- An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
- A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence
Discussions