AI Coding Assistant Productivity: What Studies Found

If you run a business in Metro Vancouver and you're paying for software work, you've probably heard that AI coding assistants make developers dramatically faster. Some controlled studies back that up. One well-run study found the opposite. The honest answer is that the effect depends heavily on who is coding and what they're working on, and the research says so plainly.

Here's what the studies actually measured, with the numbers and who published them.

The studies that found large gains

The first widely quoted experiment came out of GitHub and Microsoft. In The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (Peng, Kalliamvakou, Cihon and Demirer, 2023), recruited developers were asked to implement an HTTP server in JavaScript as quickly as possible. The group with access to Copilot finished 55.8% faster than the control group. That's the figure you see in most sales decks. It's a real result, but it describes one self-contained greenfield task with no existing codebase to understand first.

The strongest evidence for gains in actual jobs comes from The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers (Cui, Demirer, Jaffe, Musolff, Peng and Salz, February 2025). Three randomized trials at Microsoft, Accenture and an anonymous Fortune 100 manufacturer, pooled across 4,867 developers, found a 26.08% increase in completed tasks among developers using the tool, with a standard error of 10.3%. The same paper reports a 13.55% increase in commits and a 38.38% increase in the number of times code was compiled.

The distribution matters more than the average. That paper found Copilot significantly raised task completion for more recent hires and people in more junior positions, but not for developers with longer tenure and more senior titles.

Google ran its own trial. How much does AI impact development speed? An enterprise-based randomized controlled trial (Paradis and colleagues, 2024) gave 96 full-time Google engineers a complex enterprise-grade task. The best estimate was that AI shortened time on task by about 21%, and the authors are careful to say the confidence interval is large and that they cannot assume the effect carries to other tools or later years.

The study that found the opposite

Then there's the result nobody selling AI tools brings up. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker, Rush, Barnes and Rein, 2025) ran a randomized trial with 16 experienced open-source developers completing 246 real tasks in mature projects they had worked on for an average of five years. Each task was randomly assigned to allow or disallow AI. When AI was allowed, developers mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, the tooling frontier from February to June 2025.

Allowing AI increased completion time by 19%.

The perception gap is the part worth sitting with. Before starting, those developers forecast that AI would cut their completion time by 24%. After finishing, having actually been slowed down, they still estimated AI had made them 20% faster. Economics and machine learning experts had predicted 39% and 38% speedups. The authors collected and evaluated 20 properties of their setup that might explain the slowdown, and reported that the effect held up across their analyses.

Why the results disagree

The studies aren't measuring the same thing. Once you see that, the spread stops looking like a contradiction.

• Task type. A fresh HTTP server has no context to load. A bug in a ten-year-old repository does.
• Familiarity with the codebase. Gains cluster among newer and more junior developers. The slowdown showed up in people with five years inside one specific project.
• What gets counted. Minutes on a single task, weekly tasks completed, and commits are three different outcomes, and AI can push them in different directions.
• Time horizon. Speed at the Cost of Quality (He, Miller, Agarwal, Kastner and Vasilescu, November 2025) compared GitHub projects that adopted Cursor against a matched control group that didn't. It found a large but transient increase in development velocity, along with a substantial and persistent increase in static analysis warnings and code complexity, and identified those quality regressions as a major factor driving long-term velocity slowdown.
• Team health. That's the theme of Google's own survey work.

The 2025 DORA State of AI-assisted Software Development report, announced by Google Cloud on 23 September 2025 and based on responses from nearly 5,000 technology professionals, lands in a similar place from a different angle. Ninety percent of respondents said they use AI at work, and more than 80% believed it increased their productivity, yet 30% reported little or no trust in the code it generates. DORA describes AI adoption as tracking with higher throughput and with less stable software delivery at the same time. Their summary line is that AI doesn't fix a team, it amplifies what's already there.

What this means if you're the one signing the invoice

Treat "AI coding assistant productivity" as a range, not a multiplier. Across the credible controlled work published between 2023 and late 2025, the measured effects run from roughly 20% to 26% better in enterprise settings, up to 55.8% on an isolated greenfield task, down to 19% worse for veterans in code they know cold. Anyone quoting you a flat "3x faster" is quoting a feeling, not a study, and the METR paper is direct evidence that the feeling and the stopwatch disagree.

The practical read for a small company in Burnaby or New Westminster: the gains are most reliable on new, well-scoped, low-context work, and on onboarding people who are new to your stack. They're least reliable on deep maintenance of systems your senior person already carries in their head.

Where this doesn't apply

If your team is two senior people maintaining a codebase they've lived in for years, the METR setup is the closest match to your reality, and you should plan for no speedup in the first few months rather than a windfall. If your bottleneck isn't writing code at all, and for most small businesses it isn't, a coding assistant solves the wrong problem. Deciding what to build, getting the phone answered, and following up on quotes are usually further upstream.

There's also a quality bill to pay. The Cursor study found the complexity and warning count went up and stayed up, which means review capacity has to grow with output. If you don't have someone who can competently review generated code, faster generation just moves the risk somewhere less visible. And none of these studies measured revenue, defect escape rates in production, or customer outcomes. They measured time, tasks and commits. Those aren't the same thing as a better business.

Finally, everything here has a date on it. The tools moved between the 2023 Copilot experiment and the 2025 agentic tooling in the METR and Cursor studies, and they've moved again since. Re-check the evidence before committing budget, and prefer results measured on work that looks like yours.

Sources

• Peng, S., Kalliamvakou, E., Cihon, P., and Demirer, M. (2023). *The Impact of AI on Developer Productivity: Evidence from GitHub Copilot*. arXiv:2302.06590. https://arxiv.org/abs/2302.06590
• Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., and Salz, T. (February 2025). *The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers*. MIT Department of Economics. https://economics.mit.edu/sites/default/files/inline-files/draft_copilot_experiments.pdf
• Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., and Chandra, S. (2024). *How much does AI impact development speed? An enterprise-based randomized controlled trial*. arXiv:2410.12944. https://arxiv.org/abs/2410.12944
• Becker, J., Rush, N., Barnes, E., and Rein, D. (2025). *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*. METR, arXiv:2507.09089. https://arxiv.org/abs/2507.09089
• He, H., Miller, C., Agarwal, S., Kastner, C., and Vasilescu, B. (November 2025). *Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects*. arXiv:2511.04427. https://arxiv.org/abs/2511.04427
• Harvey, N., and DeBellis, D. (23 September 2025). *Announcing the 2025 DORA Report: State of AI-assisted Software Development*. Google Cloud Blog. https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report

If you're weighing whether to put AI into your own operations and you'd rather start from evidence than from a pitch, we're happy to talk it through. Autana Solutions builds AI employees and automations for businesses across Burnaby, New Westminster and the rest of Metro Vancouver, and part of that job is telling you when a process isn't worth automating yet. Book a free call and we'll look at where your actual bottleneck is.