CodeJudge: Evaluating Code Generation with Large Language Models

2024/10/03 by Weixi Tong, Tianyi Zhang, Tong, Weixi +1 · 1 voice · 27 citations
Computer Science · #Code (set theory) #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model-Driven Software Engineering Techniques #Natural Language Processing Techniques #Programming language #Software Engineering (cs.SE) #Software Engineering Research #cs.CL #cs.LG #cs.SE

paper · pdf · doi:10.48550/arxiv.2410.02184

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/10/03 · arxiv published 2024/10/03 · arxiv updated 2024/10/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have shown promising performance in code generation. However, how to reliably evaluate code generated by LLMs remains an unresolved problem. This paper presents CodeJudge, a code evaluation framework that leverages LLMs to evaluate the semantic correctness of generated code without the need for test cases. We investigate different ways to guide the LLM in performing "slow thinking" to arrive at an in-depth and reliable evaluation. We experimented with four LLMs as evaluators on four code generation datasets and five programming languages. The results show that CodeJudge significantly outperformed existing methods in most settings. Furthermore, compared with a SOTA GPT-3.5-based code evaluation method, CodeJudge achieved better results even when using a much smaller model, Llama-3-8B-Instruct. Our code and datasets are available on GitHub https://github.com/VichyTong/CodeJudge.

Cited by

• Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming
• Less Deliberate in Teams: Student LLM Use Across Individual and Collaborative Work
• FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
• Multicalibration for LLM-based Code Generation
• Towards a Human-in-the-Loop Framework for Reliable Patch Evaluation Using an LLM-as-a-Judge
• Learning Bug Context for PyTorch-to-JAX Translation with LLMs
• Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair
• Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
• Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications
• Embedding Alignment in Code Generation for Audio
• On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization
• CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
• Evaluating Efficiency and Novelty of LLM-Generated Code for Graph Analysis
• Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy
• The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMs
• LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
• Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
• KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes
• ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code
• CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation
• Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator
• Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
• FLOW-BENCH: Towards Conversational Generation of Enterprise Workflows
• Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge
• Human-In-The-Loop Software Development Agents: Challenges and Future Directions
• Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
• CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation

Discussions

• I’m now thinking about what good design would be for a metric where you want to understand whether the model has understood the diff in two passes, 1. Summarizing the code with abstractive summariza [bsky, 2 points, 1 comments]

Related

• CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?
• A Survey on Evaluating Large Language Models in Code Generation Tasks
• CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
• Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review
• Improving Code Generation via Small Language Model-as-a-judge