LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation

2024/04/15 by Sarah Fakhoury, Aaditya Naik, Georgios Sakkas +3 · 1 voice · 94 citations
Computer Science · #Code (set theory) #Code generation #Computer science #Empirical research #Human–computer interaction #Model-Driven Software Engineering Techniques #Operating system #Programming language #Software Engineering Research #Software Testing and Debugging Techniques #Software engineering #Test (biology) #cs.SE

paper · pdf · doi:10.1109/tse.2024.3428972

published in IEEE Transactions on Software Engineering 50(9), 2254-2268 (IEEE Computer Society)

arxiv published 2024/04/15 · openalex publication_date 2024/07/22 · openalex created_date 2024/07/23 · arxiv updated 2024/10/02 · openalex updated_date 2026/08/01

Abstract

Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflowTiCoderfor guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.

Cited by

• Holistic Evaluation of State-of-the-Art LLMs for Code Generation
• WildCode: An Empirical Analysis of Code Generated by ChatGPT
• SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprinting
• Beyond Code Similarity: Benchmarking the Plausibility, Efficiency, and Complexity of LLM-Generated Smart Contracts
• Performance and interpretability analysis of code generation large language models
• LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation
• Understanding the Characteristics of LLM-Generated Property-Based Tests in Exploring Edge Cases
• CircuitGuard: Mitigating LLM Memorization in RTL Code Generation Against IP Leakage
• Knowledge-Guided Multi-Agent Framework for Application-Level Software Code Generation
• Software Testing with Large Language Models: An Interview Study with Practitioners
• Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization
• TENET: Leveraging Tests Beyond Validation for Code Generation
• When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning
• What You Code Is What We Prove: Translating BLE App Logic into Formal Models with LLMs for Vulnerability Detection
• app.build: A Production Framework for Scaling Agentic Prompt-to-App Generation with Environment Scaffolding
• AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
• CodeGrad: Integrating Multi-Step Verification with Gradient-Based LLM Refinement
• Mockingbird: How does LLM perform in general machine learning tasks?
• Tool-integrated Reinforcement Learning for Repo Deep Search
• Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions
• SIMCODE: A Benchmark for Natural Language to ns-3 Network Simulation Code Generation
• CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance
• Is Quantization a Deal-breaker? Empirical Insights from Large Code Models
• Assessing Small Language Models for Code Generation: An Empirical Study with Benchmarks
• AI4BayesCode: From Natural Language Descriptions to Validated Modular Stateful Bayesian Samplers
• Capability-Driven Skill Generation with LLMs: A RAG-Based Approach for Reusing Existing Libraries and Interfaces
• System-driven Cloud Architecture Design Support with Structured State Management and Guided Decision Assistance
• Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
• MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning
• JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation
• A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards
• Test Design and Review Argumentation in AI-Assisted Test Generation
• EffiPair: Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback
• Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications
• Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillation

Discussions

• This is an example. It is not TDD (and the paper does not say so) arxiv.org/abs/2404.10100 [bsky, 1 points, 0 comments]

Related

• Interactive Code Generation via Test-Driven User-Intent Formalization
• Testing LLMs on Code Generation with Varying Levels of Prompt Specificity
• Test-Driven Development and LLM-based Code Generation
• StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code
• Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs