2022/11/29 by Tianyi Zhang, Changyuan Yu, Zhang, Tianyi +10 · 10 citations
Computer Science · #Advanced Malware Detection Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Programming Languages (cs.PL) #Software Engineering (cs.SE) #Software Engineering Research #Software Reliability and Analysis Research
paper · pdf · doi:10.48550/arxiv.2211.16490
openalex publication_date 2022/11/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions. Inspired by collaborative programming, we propose Coder-Reviewer reranking. We augment Coder language models from past work, which generate programs given language instructions, with Reviewer models, which evaluate the likelihood of the instruction given the generated programs. We perform an extensive study across six datasets with eight models from three model families. Experimental results show that Coder-Reviewer reranking leads to consistent and significant improvement (up to 17% absolute accuracy gain) over reranking with the Coder model only. When combined with executability filtering, Coder-Reviewer reranking can often outperform the minimum Bayes risk method. Coder-Reviewer reranking is easy to implement by prompting, can generalize to different programming languages, and works well with off-the-shelf hyperparameters.