Few-shot training LLMs for project-specific code-summarization

2022/07/09 by Toufique Ahmed, Ahmed, Toufique, Prémkumar Dévanbu +1 · 21 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Software Engineering (cs.SE) #Software Engineering Research #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2207.04237

openalex publication_date 2022/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Few-shotting has particular synergies in software engineering, where there are a lot of phenomena (identifier names, APIs, terminology, coding patterns) that are known to be highly project-specific. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs might be very relevant. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-of-the-art models for code-summarization, leveraging project-specific training.

Cited by

• Toward Explaining Large Language Models in Software Engineering Tasks
• Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap
• A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code
• Smart but Costly? Benchmarking LLMs on Functional Accuracy and Energy Efficiency
• Evaluating Language Model Applications for Identifying Solution-Related Content in Issue Report Discussions
• Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat
• Issue-Oriented Agent-Based Framework for Automated Review Comment Generation
• ALMAS: an Autonomous LLM-based Multi-Agent Software Engineering Framework
• CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebases
• Trace: Securing Smart Contract Repository Against Access Control Vulnerability
• Case Study of GAI for Generating Novel Images for Real-World Embroidery
• RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
• Multi Language Models for On-the-Fly Syntax Highlighting
• REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinement
• Towards Reliable Generation of Executable Workflows by Foundation Models
• DocFetch - Towards Generating Software Documentation from Multiple Software Artifacts
• CLMTracing: Black-box User-level Watermarking for Code Language Model Tracing
• UserTrace: User-Level Requirements Generation and Traceability Recovery from Software Project Repositories
• TraceRAG: A LLM-Based Framework for Explainable Android Malware Detection and Behavior Analysis
• CS-Agent: LLM-based Community Search via Dual-agent Collaboration
• Meta-RAG on Large Codebases Using Code Summarization

Related

• Does Few-Shot Learning Help LLM Performance in Code Synthesis?
• Exploring Large Language Models for Code Explanation
• Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code
• The Few-shot Dilemma: Over-prompting Large Language Models
• Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation