Large Language Models Meet NL2Code: A Survey

2022/12/19 by Daoguang Zan, Bei Chen, Zan, Daoguang +13 · 21 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Data Classification #Programming Languages (cs.PL) #Software Engineering (cs.SE) #Software Engineering Research #Software System Performance and Reliability

paper · pdf · doi:10.48550/arxiv.2212.09420

openalex publication_date 2022/12/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The task of generating code from a natural language description, or NL2Code, is considered a pressing and significant challenge in code intelligence. Thanks to the rapid development of pre-training techniques, surging large language models are being proposed for code, sparking the advances in NL2Code. To facilitate further research and applications in this field, in this paper, we present a comprehensive survey of 27 existing large language models for NL2Code, and also review benchmarks and metrics. We provide an intuitive comparison of all existing models on the HumanEval benchmark. Through in-depth observation and analysis, we provide some insights and conclude that the key factors contributing to the success of large language models for NL2Code are "Large Size, Premium Data, Expert Tuning". In addition, we discuss challenges and opportunities regarding the gap between models and humans. We also create a website https://nl2code.github.io to track the latest progress through crowd-sourcing. To the best of our knowledge, this is the first survey of large language models for NL2Code, and we believe it will contribute to the ongoing development of the field.

Cited by

• Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testing
• Attention Distance: A Novel Metric for Directed Fuzzing with Large Language Models
• IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection
• Better Call Grep: Evaluating and Improving Grep-Like Lexical Retrieval for Repository-Level Code Completion
• Do LLMs Recognize Your Latent Preferences? A Benchmark for Latent Information Discovery in Personalized Interaction
• A Matter of Representation: Towards Graph-Based Abstract Code Generation
• Repairing Regex Vulnerabilities via Localization-Guided Instructions
• Bamboo: LLM-Driven Discovery of API-Permission Mappings in the Android Framework
• ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluation
• SolContractEval: A Benchmark for Evaluating Contract-Level Solidity Code Generation
• MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models
• STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases
• ST-Raptor: LLM-Powered Semi-Structured Table Question Answering
• Large Language Models Meet Legal Artificial Intelligence: A Survey
• EnvX: Agentize Everything with Agentic AI
• GRACE: Graph-Guided Repository-Aware Code Completion through Hierarchical Code Fusion
• The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completion
• Can Multi-turn Self-refined Single Agent LMs with Retrieval Solve Hard Coding Problems?
• Tapas Are Free! Training-Free Adaptation of Programmatic Agents via LLM-Guided Program Synthesis in Dynamic Environments
• Exploring the Potential of Large Language Models in Fine-Grained Review Comment Classification
• ProCut: LLM Prompt Compression via Attribution Estimation

Related

• A Survey on Large Language Models for Code Generation
• Improving Natural Language Capability of Code Large Language Model
• A Systematic Evaluation of Large Language Models of Code
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
• Large Language Models for Code Generation: The Practitioners Perspective