The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
2023/02/13 by Sida Peng, Peng, Sida, Eirini Kalliamvakou +5 · 5 voices · 121 citations
Computer Science · #Online Learning and Analytics #Open Source Software Innovations #Software Engineering Research #cs.SE
paper · pdf · doi:10.48550/arxiv.2302.06590
openalex publication_date 2023/02/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31
Abstract
Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.
Cited by
- An Empirical Study of Generative AI Adoption in Software Engineering
- AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher Education
- The Hitchhiker's Guide to Monoculture
- Early-Stage Prediction of Review Effort in AI-Generated Pull Requests
- CoTDeceptor:Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agents
- Developers' Experience with Generative AI -- First Insights from an Empirical Mixed-Methods Field Study
- A survey of generative AI adoption and perceived productivity among scientists who program
- AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software
- On Assessing the Relevance of Code Reviews Authored by Generative Models
- DeepCode: Open Agentic Coding
- The AI Attribution Paradox: Transparency as Social Strategy in Open-Source Software Development
- CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
- Automated Design Optimization via Strategic Search with Large Language Models
- Bug Detective and Quality Coach: Developers' Mental Models of AI-Assisted IDE Tools
- NNGPT: Rethinking AutoML with Large Language Models
- Optimizing LLM Code Suggestions: Feedback-Driven Timing with Lightweight State Bounds
- From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI
- Design principles for text-to-image generative artificial intelligence creativity support tools for visual design
- Comprehension-Performance Gap in GenAI-Assisted Brownfield Programming: A Replication and Extension
- AI as Equalizer or Amplifier? Task Complexity as the Moderating Factor for Human Expertise in Hybrid Intelligence Systems
- Faster, Higher, Stronger? The Impact of GenAI on Knowledge Work Productivity - Evidence from the Field
- AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate
- Impossible to hide secret ...: Uncovering Security and Privacy Issues in LLM-native IDEs
- Building AI Companions that Prioritise Learning over Performance
- Machine-Generated, Machine-Checked Proofs for a Verified Compiler (Experience Report)
- The Fast and Spurious: Developer Productivity with GenAI
- An efficient probabilistic hardware architecture for diffusion-like models
- CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation
- Collaborative LLM Agents for C4 Software Architecture Design Automation
- GAPO: Robust Advantage Estimation for Real-World Code LLMs
- RESCUE: Retrieval Augmented Secure Code Generation
- A Systematic Literature Review of the Use of GenAI Assistants for Code Comprehension: Implications for Computing Education Research and Practice
- Agentic Inequality
- Quantum Reinforcement Learning: Recent Advances and Future Directions
- DocReward: A Document Reward Model for Structuring and Stylizing
- Modeling AI-Driven Production and Competitiveness A Multi-Agent Economic Simulation of China and the United States
- AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden
- "AI Slop is DDoSing Open Source": Understanding the Impact of AI-Generated Contributions on Open Source Sustainability
- GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis
- MigrateLib: a tool for end-to-end Python library migration
- Modeling Developer Burnout with GenAI Adoption
- Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review
- Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning
- Socio-Economic Model of AI Agents
- The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution
- Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications in AI-assisted Data Analysis
- Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study
- Intuition to Evidence: Measuring AI's True Impact on Developer Productivity
- From moral panic to pragmatic governance: reframing AI’s societal impacts in employment, education, and ethics
- HICode: Hierarchical Inductive Coding with LLMs
- Understanding the Role of Large Language Models in Competitive Programming
- Evolution of Programmers' Trust in Generative AI Programming Assistants
- Automating Code Generation for Semiconductor Equipment Control from Developer Utterances with LLMs
- Vibe Coding for UX Design: Understanding UX Professionals' Perceptions of AI-Assisted Design and Development
- Proactive AI Adoption can be Threatening: When Help Backfires
- Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter
- Aligning Requirement for Large Language Model's Code Generation
- Making AI Inevitable: Historical Perspective and the Problems of Predicting Long-Term Technological Change
- Collaborating with GenAI: Incentives and Replacements
- On the Future of Software Reuse in the Era of AI Native Software Engineering
- SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion
- "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants
- An Embodied AR Navigation Agent: Integrating BIM with Retrieval-Augmented Generation for Language Guidance
- AI-Assisted Fixes to Code Review Comments at Scale
- Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
- Nine Raters, One Index: Carrying LLM Disagreement into Labour-Market Estimates
- Improving Generative Ad Text on Facebook using Reinforcement Learning
- Automation, AI, and the Intergenerational Transmission of Knowledge
- Navigating the Complexity of Generative AI Adoption in Software Engineering
- MemoCoder: Automated Function Synthesis using LLM-Supported Agents
- The Invisible Leash: Why RLVR May or May Not Escape Its Origin
- uGen: An Agentic Framework for Generating Microarchitectural Attack PoCs
- Kodezi Chronos: A Debugging-First Language Model for Repository-Scale Code Understanding
- Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows
- The Checking Problem: What must be true before AI ships in a regulated firm
- ACE: Automated Technical Debt Remediation with Validated Large Language Model Refactorings
- ReservoirChat: Interactive Documentation Enhanced with LLM and Knowledge Graph for ReservoirPy
- Epitome: Pioneering an Experimental Platform for AI-Social Science Integration
- RedCoder: Automated Multi-Turn Red Teaming for Code LLMs
- The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks
- From Developer Pairs to AI Copilots: A Comparative Study on Knowledge Transfer
- From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems
- Bridging Expertise Gaps: The Role of LLMs in Human-AI Collaboration for Cybersecurity
- Structure-Aware Fill-in-the-Middle Pretraining for Code
- HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding
- Statically Contextualizing Large Language Models with Typed Holes
- What Needs Attention? Prioritizing Drivers of Developers' Trust and Adoption of Generative AI
- Unreliable in Practice? A Comprehensive Study of Errors in LLM-Generated Code
- VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification
- The Scaling Paradox in Human-AI Collaboration
- AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
- Joint Optimization of Human Headcount and Stochastic AI Resource Capacity
- AI Cosplaying as Astrophysicists: A Controlled Synthetic-Agent Study of AI-Assisted Astrophysical Research Workflows
- PRWeaver: Evaluating LLM-Based Code Auditors against Long-Horizon Malicious Pull Requests
- Developers' Experience with Generative AI Beyond Productivity Assessment -- Insights from an Empirical Mixed-Methods Field Study
- Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks
- Achieving Productivity Gains with AI-based IDE features: A Journey at Google
- AI in the Enterprise: How People Use M365 Copilot Chat
- Upskilling with Generative AI: Practices and Challenges for Freelance Knowledge Workers
- The software space of science
- The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study
- The Buy-or-Build Decision, Revisited: How Agentic AI Changes the Economics of Enterprise Software
- Agentic Much? Adoption of Coding Agents on GitHub
- Understanding and supporting how developers prompt for LLM-powered code editing in practice
- AI Recommendations and Non-instrumental Image Concerns
- Supporting Effective Goal Setting with LLM-Based Chatbots
- The Augmentation Trap: AI Productivity and the Cost of Cognitive Offloading
- IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks
- AI, Metacognition, and the Verification Bottleneck: A Three-Wave Longitudinal Study of Human Problem-Solving
- Conversational AI for Rapid Scientific Prototyping: A Case Study on ESA's ELOPE Competition
- Making AI Visible, Not Vanished: How AI Policies Reshape Developer Experience on GitHub
- On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practices
- An Empirical Study of Python Library Migration Using Large Language Models
- AI Safety Should Prioritize the Future of Work
- Towards Conversational AI for Human-Machine Collaborative MLOps
- Creating benchmarkable components to measure the quality ofAI-enhanced developer tools
- Estimating time spent on work tasks
- Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency
- Generative AI in Live Operations: Evidence of Productivity Gains in Cybersecurity and Endpoint Management
- How do Copilot Suggestions Impact Developers' Frustration and Productivity?
- From Teacher to Colleague: How Coding Experience Shapes Developer Perceptions of AI Tools
Discussions
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023) [hn, 6 points, 0 comments]
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot [hn, 4 points, 1 comments]
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot [hn, 2 points, 0 comments]
- AI and Microeconomics: AI tools (GitHub Copilot) shown to reduce software developers' processing time by over 50% in controlled experiments.
arxiv.org/pdf/2302.06590 [bsky, 1 points, 0 comments]
- Here is one of the first ones (2022) based on co-pilot and GPT3 thats been cited quite a lot, and claims much larger productivity gains arxiv.org/abs/2302.06590 [bsky, 0 points, 1 comments]
Related