Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
2021/08/20 by Hammond Pearce, Pearce, Hammond, Baleegh Ahmad +7 · 4 voices · 98 citations
Computer Science · #Advanced Malware Detection Techniques #Software Engineering Research #Software Testing and Debugging Techniques #cs.AI #cs.CR
paper · pdf · doi:10.48550/arxiv.2108.09293
openalex publication_date 2021/08/20 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Abstract
There is burgeoning interest in designing AI-based systems to assist humans in designing computing systems, including tools that automatically generate computer code. The most notable of these comes in the form of the first self-described `AI pair programmer', GitHub Copilot, a language model trained over open-source GitHub code. However, code often contains bugs - and so, given the vast quantity of unvetted code that Copilot has processed, it is certain that the language model will have learned from exploitable, buggy code. This raises concerns on the security of Copilot's code contributions. In this work, we systematically investigate the prevalence and conditions that can cause GitHub Copilot to recommend insecure code. To perform this analysis we prompt Copilot to generate code in scenarios relevant to high-risk CWEs (e.g. those from MITRE's "Top 25" list). We explore Copilot's performance on three distinct code generation axes -- examining how it performs given diversity of weaknesses, diversity of prompts, and diversity of domains. In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable.
Citations
Cited by
- Tool-Guided Retrieval-Augmented Repair for Securing LLM-Generated C Code
- Evaluating Large Language Models for Symbolic Security Protocol Analysis
- Insecure Coding Preferences in Long-Term Memory: Security Risks for LLM-based Code Generation
- Beyond Resolved Rate: A Non-Functional Quality Study
- Specification-Driven Development as the Foundation of AI-Native Enterprise Software Engineering
- When Post-Anchor Metrics Fail: Stabilization Regimes in AI-Evidence Open-Source Projects
- The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs
- Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code
- The End of Code Review: Coding Agents Supersede Human Inspection
- SWE-chat: Coding Agent Interactions From Real Users in the Wild
- Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild
- The 4/δ Bound: Designing Predictable LLM-Verifier Systems for Formal Method Guarantee
- Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects
- Type-Constrained Code Generation with Language Models
- Discrete prompt optimization using genetic algorithm for secure Python code generation
- "An Endless Stream of AI Slop": How Developers Discuss the Burden of AI-Assisted Software Development
- The Hitchhiker's Guide to Monoculture
- Generative Artificial Intelligence for Software Engineering -- A Research Agenda
- The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting
- Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
- SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code
- Design Theater: A Benchmark for Generative UI
- AutoBaxBuilder: Bootstrapping Code Security Benchmarking
- Assessing the Software Security Comprehension of Large Language Models
- Comment Traps: How Defective Commented-out Code Augment Defects in AI-Assisted Code Generation
- SoK: Understanding (New) Security Issues Across AI4Code Use Cases
- Holistic Evaluation of State-of-the-Art LLMs for Code Generation
- From Prompt to Product: A Human-Centered Benchmark of Agentic App Generation Systems
- A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection
- Diverse LLMs vs. Vulnerabilities: Who Detects and Fixes Them Better?
- Mind the Gap! Pathways Towards Unifying AI Safety and Ethics Research
- Multicalibration for LLM-based Code Generation
- CFCEval: Evaluating Security Aspects in Code Generated by Large Language Models
- PrivCode: When Code Generation Meets Differential Privacy
- Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
- Supporting Students in Navigating LLM-Generated Insecure Code
- DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
- LLM-CSEC: Empirical Evaluation of Security in C/C++ Code Generated by Large Language Models
- Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems
- Agentic Program Verification
- The Future of Development Environments with AI Foundation Models: NII Shonan Meeting 222 Report
- A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code
- Reducing Hallucinations in LLM-Generated Code via Semantic Triangulation
- GenSIaC: Toward Security-Aware Infrastructure-as-Code Generation with Large Language Models
- InData: Towards Secure Multi-Step, Tool-Based Data Analysis
- "I need to learn better searching tactics for privacy policy laws." Investigating Software Developers' Behavior When Using Sources on Privacy Issues
- SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction
- From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting
- Secure Code Generation at Scale with Reflexion
- Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories
- Dissect-and-Restore: AI-based Code Verification with Transient Refactoring
- A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositories
- SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents
- Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents
- Impossible to hide secret ...: Uncovering Security and Privacy Issues in LLM-native IDEs
- RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents
- RefleXGen:The unexamined code is not worth using
- Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategies
- RESCUE: Retrieval Augmented Secure Code Generation
- BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI
- When AI Takes the Wheel: Security Analysis of Framework-Constrained Program Generation
- Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models
- RAG-Pull: Imperceptible Attacks on RAG Systems for Code Generation
- TypePilot: Leveraging the Scala Type System for Secure LLM-generated Code
- Benchmarking Correctness and Security in Multi-Turn Code Generation
- Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support
- Fortifying LLM-Based Code Generation with Graph-Based Reasoning on Secure Coding Practices
- Secure-Instruct: An Automated Pipeline for Synthesizing Instruction-Tuning Datasets Using LLMs for Secure Code Generation
- Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review
- Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
- SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios
- "Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors
- Enhancing Python Programming Education with an AI-Powered Code Helper: Design, Implementation, and Impact
- Evaluating and Mitigating Errors in LLM-Generated Web API Integrations
- Investigating Security Implications of Automatically Generated Code on the Software Supply Chain
- SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
- OpenAnt: LLM-Powered Vulnerability Discovery Through Code Decomposition, Adversarial Verification, and Dynamic Testing
- Can You Trust Your Copilot? A Privacy Scorecard for AI Coding Assistants
- Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges
- Agentic AI for Software: thoughts from Software Engineering community
- GitHub's Copilot Code Review: Can AI Spot Security Flaws Before You Commit?
- A Systematic Evaluation of Parameter-Efficient Fine-Tuning Methods for the Security of Code LLMs
- Rethinking Technology Stack Selection with AI Coding Proficiency
- Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs
- Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity
- Multi-Agent Penetration Testing AI for the Web
- A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
- VerilogLAVD: LLM-Aided Rule Generation for Vulnerability Detection in Verilog
- AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
- Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development
- Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach
- Prompt to Pwn: Automated Exploit Generation for Smart Contracts
- SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval
- On LLM-Assisted Generation of Smart Contracts from Business Processes
- A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models
- OpenAI Codex (language model) [wikipedia]
- GitHub Copilot [wikipedia]
- AI alignment [wikipedia]
Discussions
- An empirical cybersecurity evaluation of GitHub Copilot's code contributions [hn, 137 points, 42 comments]
- An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions [lobsters, 7 points, 0 comments]
- Yeah, without that context from the précis that it's when performing a high-risk task, the 40% stat is meaningless. E.g. high risk: generating a database query from an untrusted input which can lead t [bsky, 1 points, 0 comments]
- - Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions (Arxiv) — arxiv.org/abs/2108.09293 [bsky, 0 points, 1 comments]
Related