Do Users Write More Insecure Code with AI Assistants?

2022/11/07 by Neil Perry, Megha Srivastava, Deepak Kumar +1 · 23 voices · 53 citations
Computer Science · Social Sciences · #Software Engineering Research #Ethics and Social Impacts of AI #Mobile Crowdsensing and Crowdsourcing

paper · pdf · doi:10.1145/3576915.3623157

Abstract

AI code assistants have emerged as powerful tools that can aid in the software development life-cycle and can improve developer productivity. Unfortunately, such assistants have also been found to produce insecure code in lab environments, raising significant concerns about their usage in practice. In this paper, we conduct a user study to examine how users interact with AI code assistants to solve a variety of security related tasks. Overall, we find that participants who had access to an AI assistant wrote significantly less secure code than those without access to an assistant. Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code. To better inform the design of future AI-based code assistants, we release our user-study apparatus to researchers seeking to build on our work.

Cited by

• Rewriting the Response Path: Silent Tampering and Provider-Signed Defense in BYOK LLM Agents
• Tool-Guided Retrieval-Augmented Repair for Securing LLM-Generated C Code
• The Aura in the Machine: Genealogy and the Status of the Work of Art in the Generative Era
• How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice
• Specification-Driven Development as the Foundation of AI-Native Enterprise Software Engineering
• The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs
• Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
• Setup Complete, Now You Are Compromised: Weaponizing Setup Instructions Against AI Coding Agents
• FuzzingBrain V2: A Multi-Agent LLM System for Automated Vulnerability Discovery and Reproduction
• Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
• SWE-chat: Coding Agent Interactions From Real Users in the Wild
• Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild
• Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects
• EditLord: Learning Code Transformation Rules for Code Editing
• Training large language models on narrow tasks can lead to broad misalignment
• From Accuracy to Impact: The Impact-Driven AI Framework (IDAIF) for Aligning Engineering Architecture with Theory of Change
• Discrete prompt optimization using genetic algorithm for secure Python code generation
• CSLib: The Lean Computer Science Library
• Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
• AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software
• SoK: Understanding (New) Security Issues Across AI4Code Use Cases
• A Retrieval-Augmented Generation Approach to Extracting Algorithmic Logic from Neural Networks
• Feedback Loops and Code Perturbations in LLM-based Software Engineering: A Case Study on a C-to-Rust Translation System
• DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
• LLM-CSEC: Empirical Evaluation of Security in C/C++ Code Generated by Large Language Models
• InData: Towards Secure Multi-Step, Tool-Based Data Analysis
• "I need to learn better searching tactics for privacy policy laws." Investigating Software Developers' Behavior When Using Sources on Privacy Issues
• Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories
• SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents
• Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents
• From Today’s Code to Tomorrow’s Symphony: The AI Transformation of Developer’s Routine by 2030
• VeriStruct: AI-assisted Automated Verification of Data-Structure Modules in Verus
• Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategies
• Does In-IDE Calibration of Large Language Models work at Scale?
• When AI Takes the Wheel: Security Analysis of Framework-Constrained Program Generation
• RAG-Pull: Imperceptible Attacks on RAG Systems for Code Generation
• Secure-Instruct: An Automated Pipeline for Synthesizing Instruction-Tuning Datasets Using LLMs for Secure Code Generation
• Selecting Cybersecurity Requirements: Effects of LLM Use and Professional Software Development Experience
• Model-Agnostic Correctness Assessment for LLM-Generated Code via Dynamic Internal Representation Selection
• Takedown: How It's Done in Modern Coding Agent Exploits
• "Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors
• Evaluating and Mitigating Errors in LLM-Generated Web API Integrations
• Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges
• GitHub's Copilot Code Review: Can AI Spot Security Flaws Before You Commit?
• Evolution of Programmers' Trust in Generative AI Programming Assistants
• A Systematic Evaluation of Parameter-Efficient Fine-Tuning Methods for the Security of Code LLMs
• VisDocSketcher: Towards Scalable Visual Documentation with Agentic Systems
• Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications
• Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter
• Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness
• A Moral Agency Framework for Legitimate Integration of AI in Bureaucracies
• AIAP: A No-Code Workflow Builder for Non-Experts with Natural Language and Multi-Agent Collaboration
• SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval

Discussions

• Do Users Write More Insecure Code with AI Assistants? [lemmy, 75 points, 26 comments]
• Apparently yes [bsky, 29 points, 5 comments]
• Do Users Write More Insecure Code with AI Assistants? [lobsters, 20 points, 10 comments]
• - AI generated code is insecure between 39% and 50% of the time (dependent on language) arxiv.org/abs/2211.03622 [bsky, 7 points, 1 comments]
• The article links to this study, showing that code written with an AI assistant is more buggy and less secure. Pichai is bragging that a quarter of all new Google code is of lower quality than normal [bsky, 5 points, 0 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 5 points, 4 comments]
• Study: Do Users Write More Insecure Code with AI Assistants? [hn, 4 points, 2 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 4 points, 1 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 3 points, 0 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 3 points, 0 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 3 points, 1 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 2 points, 1 comments]
• AI-assistenter gör så att folk skriver fler säkerhetsbuggar i sin kod "Overall, we find that participants who had access to an AI assistant based [...] wrote significantly less secure code than those [bsky, 2 points, 1 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 2 points, 2 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 2 points, 0 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 2 points, 0 comments]
• "[P]articipants who had access to an #AI assistant ... wrote significantly less secure code than those without access. Additionally, participants with access to an AI assistant were more likely to bel [bsky, 1 points, 0 comments]
• Do Users Write More Insecure Code with AI Assistants? [hn, 1 points, 0 comments]
• Do Users Write More Insecure Code with AI Assistants? - arxiv.org/abs/2211.0... [bsky, 0 points, 0 comments]
• Ref for the data: www.gitclear.com/ai_assistant... dora.dev/research/202... arxiv.org/abs/2211.03622 [bsky, 0 points, 1 comments]
• Do Users Write More Insecure Code with AI Assistants? [yes] https://arxiv.org/abs/2211.03622 [bsky, 0 points, 0 comments]
• Pretty much: arxiv.org/abs/2211.03622 [bsky, 0 points, 0 comments]
• Surprising: no one, actually arxiv.org/abs/2211.03622 [bsky, 0 points, 0 comments]

Related

• How Are We Doing With Using AI-Based Programming Assistants For Privacy-Related Code Generation? The Developers' Experience
• Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models
• Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity
• Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
• SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment