LEVER: Learning to Verify Language-to-Code Generation with Execution
2023/02/16 by Ansong Ni, Ni, Ansong, Srini Iyer +11 · 1 voice · 47 citations
Computer Science · #Natural Language Processing Techniques #Software Engineering Research #Topic Modeling #cs.CL #cs.LG #cs.PL #cs.SE
paper · pdf · doi:10.48550/arxiv.2302.08468
arxiv published 2023/02/16 · arxiv updated 2023/09/01
Abstract
The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine LLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the LLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.
Cited by
- Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation
- Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
- Table as a Modality for Large Language Models
- InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution
- Test-time Diverse Reasoning by Riemannian Activation Steering
- SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction
- Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat
- MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQL
- Do Code Language Models Use Tests? A Behavioral and Representational Study of Test-Driven Code Generation
- Pie: A Programmable Serving System for Emerging LLM Applications
- Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation
- When Names Disappear: Revealing What LLMs Actually Understand About Code
- Interactive Learning for LLM Reasoning
- Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generation
- RepLLM: Toward Automatically Reproducing Network Research Results
- LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions
- FVDebug: An LLM-Driven Debugging Assistant for Automated Root Cause Analysis of Formal Verification Failures
- Improving Table Understanding with LLMs and Entity-Oriented Search
- Planning for Success: Exploring LLM Long-term Planning Capabilities in Table Understanding
- TableZoomer: A Collaborative Agent Framework for Large-scale Table Question Answering
- Clean Code, Better Models: Enhancing LLM Performance with Smell-Cleaned Dataset
- Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data
- Reinforce LLM Reasoning through Multi-Agent Reflection
- Invariant-based Robust Weights Watermark for Large Language Models
- Rethinking Verification for LLM Code Generation: From Generation to Testing
- Use Property-Based Testing to Bridge LLM Code Generation and Validation
- Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
- Pareto Optimal Code Generation
- A Reasoning-Based Approach to Cryptic Crossword Clue Solving
- Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities
- BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
- SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
- A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law
- Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
- An End-to-End Model for Logits-Based Large Language Models Watermarking
- From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark
- JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation
- Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking
- ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution
- When Reasoning Beats Scale: A 1.5B Reasoning Model Outranks 13B LLMs as Discriminator
- Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges
- PAMT: Process-Aligned Reinforcement Learning for Multi-Domain Machine Translation
- When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs
- The Order Is the Guarantee: Verifier-Budgeted Code Deletion with Static-First Learned Proposals
- Inducing Vulnerable Code Generation in LLM Coding Assistants
- CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
- Themisto: Jupyter-Based Runtime Benchmark
Discussions
Related