Learning from 53.6K Real-World Developer Edits of AI-Generated Code

2026/07/27 by Jenny T. Liang, Mihika Bairathi, Wayne Chi +3
#cs.SE #cs.AI #cs.HC #cs.LG

paper · pdf

Abstract

Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant. Manual code edits provide more realistic and granular information on editing behavior than Git commits, which only contain final successful code snippets. Yet, due to a lack of high-quality, realistic code editing data, LLMs are mostly trained on publicly available Git data (e.g., commits). To address this gap, we introduce DECODE (Developer Edits of Code Dataset), a dataset of 53.6K real-world in-IDE code edits of AI-generated code in Python, TypeScript, and JavaScript, sourced from 1K+ developers. First, we demonstrate the utility of DECODE for data analysis, obtaining insights on when, why, and how AI-generated code is edited. We find that most edits occur within the first 15 minutes after accepting an AI completion, resulting in the removal of AI completions in 31% of edit trajectories. Second, we use DECODE to benchmark the ability of LLMs to predict code edits. We find that finetuning on DECODE enables open-source 3B models to perform code edit prediction tasks significantly better than frontier LLMs. We then discuss implications of this work, emphasizing the necessity of developer-centric machine learning approaches for future AI programming assistants.

Citations

• SWE-chat: Coding Agent Interactions From Real Users in the Wild
• OpenAI GPT-5 System Card
• EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
• Modeling Student Learning with 3.8 Million Program Traces
• The Era of Real-World Human Interaction: RL from User Conversations
• Generating High-Quality Datasets for Code Editing via Open-Source Language Models
• Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction History
• User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal
• Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild
• Copilot Arena: A Platform for Code LLM Evaluation in the Wild
• Clio: Privacy-Preserving Insights into Real-World AI Use
• Retrospective Learning from Interactions
• Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code
• Qwen2.5-Coder Technical Report
• The Llama 3 Herd of Models
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• The Use of Generative Search Engines for Knowledge Work and Complex Tasks
• StarCoder 2 and The Stack v2: The Next Generation
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
• InstructCoder: Instruction Tuning Large Language Models for Code Editing
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Code Llama: Open Foundation Models for Code
• OctoPack: Instruction Tuning Code Large Language Models
• GrACE: Generation using Associated Code Edits
• StarCoder: may the source be with you!
• A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
• Learning Performance-Improving Code Edits
• CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
• A Qualitative Study on the Implementation Design Decisions of Developers
• CoditT5: Pretraining for Source Code and Natural Language Editing
• Overwatch: Learning Patterns in Code Edit Sequences
• InCoder: A Generative Model for Code Infilling and Synthesis
• Training language models to follow instructions with human feedback
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• Sequence Transduction with Recurrent Neural Networks

Related

• AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans
• EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
• An Efficient and Adaptive Next Edit Suggestion Framework with Zero Human Instructions in IDEs
• AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software
• GrACE: Generation using Associated Code Edits