vix.ing · top · new · best · stats · spec

How AI Impacts Skill Formation

A randomized study found developers who used AI to learn a new Python library scored 17% lower on a skill quiz without coding any faster.

2026/01/28 by Judy Hanwen Shen, Alex Tamkin · 100 voices · 5 citations
#cs.CY #cs.AI #cs.HC

paper · pdf

Abstract

AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.

Summary

Researchers ran a randomized experiment where 52 developers learned an unfamiliar Python library (Trio, for asynchronous programming) either with or without a chat-based AI assistant. People who used the AI assistant scored about 17% lower on a follow-up quiz testing debugging, code reading, and conceptual understanding, even though they did not finish the coding tasks any faster on average. Watching screen recordings of every participant, the researchers found that AI users who mainly asked conceptual questions (rather than having AI generate or debug their code for them) kept most of the learning benefit that the no-AI group got.

machine-generated · claude-sonnet-5

Outline

machine-generated · claude-sonnet-5

Claims

machine-generated · claude-sonnet-5

Key figure

Figure 1 — Left: workers who used AI assistance to complete tasks with an unfamiliar Python library scored notably lower on tests of conceptual understanding, code reading, and debugging than workers who did the same tasks without AI. Right: of six distinct ways participants used the AI assistant, three that kept them actively thinking (asking conceptual questions or requesting explanations) preserved learning, while three built on delegating code writing or debugging to AI did not.

machine-generated · claude-sonnet-5

Glossary

Trio
A Python library for asynchronous, concurrent programming that participants had never used before the study.
Structured concurrency
A programming style, used by Trio, where concurrently running tasks are always nested inside a clear parent scope so none can outlive or escape it.
Nursery
Trio's construct for launching and supervising a group of concurrent tasks together.
Cognitive offloading
Letting a tool (like an AI assistant) do mental work for you instead of working through it yourself, which can weaken learning.
Between-subjects experiment
A study design where each participant is randomly assigned to only one condition (here, AI or no-AI), so the two groups are compared rather than the same person tried both.
Cohen's d
A standardized measure of how big a difference between two groups is, independent of sample size; larger values mean a bigger effect.
Overreliance
Accepting an AI's output as correct without checking it, even when the AI is wrong.
Local item dependence
A flaw in a test where the answer to one question can be figured out from another question, which inflates scores without reflecting real understanding.

machine-generated · claude-sonnet-5

Audience

Engineering managers, technical onboarding/L&D designers, CS educators, and researchers studying human-AI collaboration who need evidence on whether AI coding assistants help or hurt on-the-job skill acquisition.

prerequisites: Basic familiarity with Python and asynchronous programming concepts, Comfort reading experimental results reported as effect sizes and p-values, No machine learning background required

machine-generated · claude-sonnet-5

Open questions

machine-generated · claude-sonnet-5

Supplementary links

machine-generated · claude-sonnet-5

Citations

Cited by

Discussions

Related