Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
2021/08/20 by Hammond Pearce, Pearce, Hammond, Baleegh Ahmad +7 · 4 voices · 168 citations
Computer Science · #Advanced Malware Detection Techniques #Code (set theory) #Computer science #Computer security #Diversity (politics) #Programmer #Programming language #Software Engineering Research #Software Testing and Debugging Techniques #Source code #cs.AI #cs.CR
paper · pdf · doi:10.48550/arxiv.2108.09293
published in arXiv (Cornell University) (Cornell University) · Accepted for publication in IEEE Symposium on Security and Privacy 2022
openalex publication_date 2021/08/20 · arxiv created 2021/12/16 · arxiv updated 2021/12/17 · openalex created_date 2022/07/25 · openalex updated_date 2026/08/05
Abstract
There is burgeoning interest in designing AI-based systems to assist humans in designing computing systems, including tools that automatically generate computer code. The most notable of these comes in the form of the first self-described `AI pair programmer', GitHub Copilot, a language model trained over open-source GitHub code. However, code often contains bugs - and so, given the vast quantity of unvetted code that Copilot has processed, it is certain that the language model will have learned from exploitable, buggy code. This raises concerns on the security of Copilot's code contributions. In this work, we systematically investigate the prevalence and conditions that can cause GitHub Copilot to recommend insecure code. To perform this analysis we prompt Copilot to generate code in scenarios relevant to high-risk CWEs (e.g. those from MITRE's "Top 25" list). We explore Copilot's performance on three distinct code generation axes -- examining how it performs given diversity of weaknesses, diversity of prompts, and diversity of domains. In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable.
Citations
Cited by
- Tool-Guided Retrieval-Augmented Repair for Securing LLM-Generated C Code
- Evaluating Large Language Models for Symbolic Security Protocol Analysis
- Insecure Coding Preferences in Long-Term Memory: Security Risks for LLM-based Code Generation
- Beyond Resolved Rate: A Non-Functional Quality Study
- Specification-Driven Development as the Foundation of AI-Native Enterprise Software Engineering
- When Post-Anchor Metrics Fail: Stabilization Regimes in AI-Evidence Open-Source Projects
- The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs
- Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code
- The End of Code Review: Coding Agents Supersede Human Inspection
- SWE-chat: Coding Agent Interactions From Real Users in the Wild
- Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild
- The 4/δ Bound: Designing Predictable LLM-Verifier Systems for Formal Method Guarantee
- Speed at the Cost of Quality
- Type-Constrained Code Generation with Language Models
- Discrete prompt optimization using genetic algorithm for secure Python code generation
- "An Endless Stream of AI Slop": How Developers Discuss the Burden of AI-Assisted Software Development
- The Hitchhiker's Guide to Monoculture
- Generative Artificial Intelligence for Software Engineering -- A Research Agenda
- The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting
- Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
- SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code
- Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools
- AutoBaxBuilder: Bootstrapping Code Security Benchmarking
- Assessing the Software Security Comprehension of Large Language Models
- Comment Traps: How Defective Commented-out Code Augment Defects in AI-Assisted Code Generation
- SoK: Understanding (New) Security Issues Across AI4Code Use Cases
- Holistic Evaluation of State-of-the-Art LLMs for Code Generation
- From Prompt to Product: A Human-Centered Benchmark of Agentic App Generation Systems
- A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection
- Diverse LLMs vs. Vulnerabilities: Who Detects and Fixes Them Better?
- Mind the Gap! Pathways Towards Unifying AI Safety and Ethics Research
- Multicalibration for LLM-based Code Generation
- CFCEval: Evaluating Security Aspects in Code Generated by Large Language Models
- PrivCode: When Code Generation Meets Differential Privacy
- Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
- Supporting Students in Navigating LLM-Generated Insecure Code
- DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
- LLM-CSEC: Empirical Evaluation of Security in C/C++ Code Generated by Large Language Models
- Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems
- Agentic Program Verification
- The Future of Development Environments with AI Foundation Models: NII Shonan Meeting 222 Report
- A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code
- Reducing Hallucinations in LLM-Generated Code via Semantic Triangulation
- GenSIaC: Toward Security-Aware Infrastructure-as-Code Generation with Large Language Models
- InData: Towards Secure Multi-Step, Tool-Based Data Analysis
- "I need to learn better searching tactics for privacy policy laws." Investigating Software Developers' Behavior When Using Sources on Privacy Issues
- SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction
- From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting
- Secure Code Generation at Scale with Reflexion
- Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories
- Dissect-and-Restore: AI-based Code Verification with Transient Refactoring
- A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositories
- SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents
- Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents
- Impossible to hide secret ...: Uncovering Security and Privacy Issues in LLM-native IDEs
- RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents
- RefleXGen:The unexamined code is not worth using
- Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategies
- RESCUE: Retrieval Augmented Secure Code Generation
- BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI
- When AI Takes the Wheel: Security Analysis of Framework-Constrained Program Generation
- Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models
- RAG-Pull: Imperceptible Attacks on RAG Systems for Code Generation
- TypePilot: Leveraging the Scala Type System for Secure LLM-generated Code
- Benchmarking Correctness and Security in Multi-Turn Code Generation
- Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support
- Fortifying LLM-Based Code Generation with Graph-Based Reasoning on Secure Coding Practices
- Secure-Instruct: An Automated Pipeline for Synthesizing Instruction-Tuning Datasets Using LLMs for Secure Code Generation
- Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review
- Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
- SecureAgentBench: Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios
- "Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors
- Enhancing Python Programming Education with an AI-Powered Code Helper: Design, Implementation, and Impact
- Evaluating and Mitigating Errors in LLM-Generated Web API Integrations
- Investigating Security Implications of Automatically Generated Code on the Software Supply Chain
- SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
- OpenAnt: LLM-Powered Vulnerability Discovery Through Code Decomposition, Adversarial Verification, and Dynamic Testing
- Can You Trust Your Copilot? A Privacy Scorecard for AI Coding Assistants
- Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges
- Agentic AI for Software: thoughts from Software Engineering community
- GitHub's Copilot Code Review: Can AI Spot Security Flaws Before You Commit?
- A Systematic Evaluation of Parameter-Efficient Fine-Tuning Methods for the Security of Code LLMs
- Rethinking Technology Stack Selection with AI Coding Proficiency
- Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs
- Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity
- Multi-Agent Penetration Testing AI for the Web
- A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
- VerilogLAVD: LLM-Aided Rule Generation for Vulnerability Detection in Verilog
- AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
- Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development
- Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach
- Prompt to Pwn: Automated Exploit Generation for Smart Contracts
- SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval
- On LLM-Assisted Generation of Smart Contracts from Business Processes
- A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models
- User-Centered Design with AI in the Loop: A Case Study of Rapid User Interface Prototyping with "Vibe Coding"
- When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions
- Toward Inclusive AI-Driven Development: Exploring Gender Differences in Code Generation Tool Interactions
- Learn from Your Mistakes: Tree-like Self-Play for Secure Code LLMs
- The Impact of AI-Assisted Development on Software Security: A Study of Gemini and Developer Experience
- Security-by-Design for LLM-Based Code Generation: Leveraging Internal Representations for Concept-Driven Steering Mechanisms
- Securing Agents With Tracked Capabilities
- AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms
- SVAgent: AI Agent for Hardware Security Verification Assertion
- Learning Program Behavioral Models from Synthesized Input–Output Pairs
- Self-Admitted GenAI Usage in Open-Source Software
- When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents
Droid: A Resource Suite for AI-Generated Code Detection- Disappearing Ink: Obfuscation Breaks N-gram Code Watermarks in Theory and Practice
- How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench
- Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation
- Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation
- RedCoder: Automated Multi-Turn Red Teaming for Code LLMs
- A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures
- SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows
- codetransformed: The Influence of Large Language Models on Code
- GeoPandas-AI: A Smart Class Bringing LLM as Stateful AI Code Assistant
- SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
- SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Design
- Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences
- Directed Greybox Fuzzing via Large Language Model
- VERINA: Benchmarking Verifiable Code Generation
- HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding
- Risks of AI-driven product development and strategies for their mitigation
- IRCopilot: Automated Incident Response with Large Language Models
- CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs
- SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
- Eradicating the Unseen: Detecting, Exploiting, and Remediating a Path Traversal Vulnerability across GitHub
- What Needs Attention? Prioritizing Drivers of Developers' Trust and Adoption of Generative AI
- ReCopilot: Reverse Engineering Copilot in Binary Analysis
- On Automated and Explainable Provenance of AI-Generated Code
- SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks
- Security Degradation in Iterative AI Code Generation -- A Systematic Analysis of the Paradox
- EduPluginBench: Executable Assurance for AI-Generated Educational Plugins
- The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code
- Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective
- "I Apologize For Not Understanding Your Policy": Exploring the Specification and Evaluation of User-Managed Access Control Policies by AI Virtual Assistants
- BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models
- Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection
- Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation
- SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forward
- AI Slop and the Software Commons
- Security Concerns in Generative AI Coding Assistants: Insights from Online Discussions on GitHub Copilot
- What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
- The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers
- MagicCraft: Natural Language-Driven Generation of Dynamic and Interactive 3D Objects for Commercial Metaverse Platforms
- SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
- Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges
- Secure Coding with AI -- From Detection to Repair
- CodeBC: A More Secure Large Language Model for Smart Contract Code Generation in Blockchain
- An Empirical Security Evaluation of LLM-Generated Cryptographic Rust Code
- LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda
- Can Large Language Models Implement Agent-Based Models? An ODD-based Replication Study
- Surgical Repair of Insecure Code Generation in LLMs
- Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code
- CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation
- Frontier AI's Impact on the Cybersecurity Landscape
- Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injection
- On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practices
- Inducing Vulnerable Code Generation in LLM Coding Assistants
- On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review
- The Institutional Window: Occupation- and Jurisdiction-Specific Calibration of Liability Signaling for Preserved Human Fallback Capability
- LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
- The Code Barrier: What LLMs Actually Understand?
- Achilles Heel of Distributed Multi-Agent Systems
- OpenAI Codex (language model) [wikipedia]
- GitHub Copilot [wikipedia]
- AI alignment [wikipedia]
Discussions
- An empirical cybersecurity evaluation of GitHub Copilot's code contributions [hn, 137 points, 42 comments]
- An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions [lobsters, 7 points, 0 comments]
- Yeah, without that context from the précis that it's when performing a high-risk task, the 40% stat is meaningless. E.g. high risk: generating a database query from an untrusted input which can lead t [bsky, 1 points, 0 comments]
- - Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions (Arxiv) — arxiv.org/abs/2108.09293 [bsky, 0 points, 1 comments]
Related