The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

2023/02/13 by Sida Peng, Peng, Sida, Eirini Kalliamvakou +5 · 5 voices · 69 citations
Computer Science · #Online Learning and Analytics #Open Source Software Innovations #Software Engineering Research #cs.SE

paper · pdf · doi:10.48550/arxiv.2302.06590

openalex publication_date 2023/02/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.

Cited by

• An Empirical Study of Generative AI Adoption in Software Engineering
• AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher Education
• The Hitchhiker's Guide to Monoculture
• Early-Stage Prediction of Review Effort in AI-Generated Pull Requests
• CoTDeceptor:Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agents
• Developers' Experience with Generative AI -- First Insights from an Empirical Mixed-Methods Field Study
• A survey of generative AI adoption and perceived productivity among scientists who program
• AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software
• On Assessing the Relevance of Code Reviews Authored by Generative Models
• DeepCode: Open Agentic Coding
• The AI Attribution Paradox: Transparency as Social Strategy in Open-Source Software Development
• CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
• Automated Design Optimization via Strategic Search with Large Language Models
• Bug Detective and Quality Coach: Developers' Mental Models of AI-Assisted IDE Tools
• NNGPT: Rethinking AutoML with Large Language Models
• Optimizing LLM Code Suggestions: Feedback-Driven Timing with Lightweight State Bounds
• From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI
• Design principles for text-to-image generative artificial intelligence creativity support tools for visual design
• Comprehension-Performance Gap in GenAI-Assisted Brownfield Programming: A Replication and Extension
• AI as Equalizer or Amplifier? Task Complexity as the Moderating Factor for Human Expertise in Hybrid Intelligence Systems
• Faster, Higher, Stronger? The Impact of GenAI on Knowledge Work Productivity - Evidence from the Field
• AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate
• Impossible to hide secret ...: Uncovering Security and Privacy Issues in LLM-native IDEs
• Building AI Companions that Prioritise Learning over Performance
• Machine-Generated, Machine-Checked Proofs for a Verified Compiler (Experience Report)
• The Fast and Spurious: Developer Productivity with GenAI
• An efficient probabilistic hardware architecture for diffusion-like models
• CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation
• Collaborative LLM Agents for C4 Software Architecture Design Automation
• GAPO: Robust Advantage Estimation for Real-World Code LLMs
• RESCUE: Retrieval Augmented Secure Code Generation
• A Systematic Literature Review of the Use of GenAI Assistants for Code Comprehension: Implications for Computing Education Research and Practice
• Agentic Inequality
• Quantum Reinforcement Learning: Recent Advances and Future Directions
• DocReward: A Document Reward Model for Structuring and Stylizing
• Modeling AI-Driven Production and Competitiveness A Multi-Agent Economic Simulation of China and the United States
• AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden
• "AI Slop is DDoSing Open Source": Understanding the Impact of AI-Generated Contributions on Open Source Sustainability
• GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis
• PyMigTool: a tool for end-to-end Python library migration
• Modeling Developer Burnout with GenAI Adoption
• Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review
• Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning
• Socio-Economic Model of AI Agents
• The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution
• Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications in AI-assisted Data Analysis
• Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study
• Intuition to Evidence: Measuring AI's True Impact on Developer Productivity
• From moral panic to pragmatic governance: reframing AI’s societal impacts in employment, education, and ethics
• HICode: Hierarchical Inductive Coding with LLMs
• Understanding the Role of Large Language Models in Competitive Programming
• Evolution of Programmers' Trust in Generative AI Programming Assistants
• Automating Code Generation for Semiconductor Equipment Control from Developer Utterances with LLMs
• Vibe Coding for UX Design: Understanding UX Professionals' Perceptions of AI-Assisted Design and Development
• Proactive AI Adoption can be Threatening: When Help Backfires
• Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter
• Aligning Requirement for Large Language Model's Code Generation
• Making AI Inevitable: Historical Perspective and the Problems of Predicting Long-Term Technological Change
• Collaborating with GenAI: Incentives and Replacements
• On the Future of Software Reuse in the Era of AI Native Software Engineering
• SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion
• "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants
• An Embodied AR Navigation Agent: Integrating BIM with Retrieval-Augmented Generation for Language Guidance
• Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
• How Exposed Are UK Jobs to Generative AI? Developing and Applying a Novel Task-Based Index
• Improving Generative Ad Text on Facebook using Reinforcement Learning
• Navigating the Complexity of Generative AI Adoption in Software Engineering
• MemoCoder: Automated Function Synthesis using LLM-Supported Agents
• uGen: An Agentic Framework for Generating Microarchitectural Attack PoCs

Discussions

• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023) [hn, 6 points, 0 comments]
• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot [hn, 4 points, 1 comments]
• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot [hn, 2 points, 0 comments]
• AI and Microeconomics: AI tools (GitHub Copilot) shown to reduce software developers' processing time by over 50% in controlled experiments. arxiv.org/pdf/2302.06590 [bsky, 1 points, 0 comments]
• Here is one of the first ones (2022) based on co-pilot and GPT3 thats been cited quite a lot, and claims much larger productivity gains arxiv.org/abs/2302.06590 [bsky, 0 points, 1 comments]

Related

• Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects
• Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle
• AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden
• The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot
• Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study